Management interface slow to come up on reboot

[idahoakl]

I have a esxi 3.5 U4 server with the latest firmware from today. When I reboot the server at any time the screen on the console comes back and I can see the logs and all but I'm unable to connection via the client or get to the web interface. I'm connecting directly to ESXi and not through VC. The hostd.log shows that the server appears to be going through all of the files on the datastore multiple times. Has anyone seen anything like this?

Thanks

Matt

[AndreTheGiant]

Do you have a iSCSI storage with software initiator?

Andre

[idahoakl]

I use a 3ware 9650SE card for the storage with a local raid array.

[AndreTheGiant]

How is "slow" for you? How many minutes?

ESX bootstrap is really slow

Andre

[idahoakl]

"slow" for me could mean a lot of things :smileyblush: However in this case from the time the console is up (ability to reboot the machine etc) it is about another 20-45 minutes before I can actually logon via the client and start a vm.

Matt

[AndreTheGiant]

ok... more than 10 min is "slow"

You have to check in your ESX log (/var/log) to see where you waste all that time.

Andre

[idahoakl]

It seems to me that DISKLIB-VMFS likes to open all of the files multiple times. Not sure what to make of it. Can you tell anything from the log excerpt below? This happens multiple times for each vm.

2009-07-30 17:24:38.847 'BaseLibs' 16384 info DISKLIB-VMFS : "/vmfs/volumes/49c01410-a48d3c08-4eca-00188b44fa90/Ziegler-Vista/Ziegler-Vista-000001-delta.vmdk" : open successful (21) size = 21474836480, hd = 0. Type 8

2009-07-30 17:24:38.847 'BaseLibs' 16384 info DISKLIB-VMFS : "/vmfs/volumes/49c01410-a48d3c08-4eca-00188b44fa90/Ziegler-Vista/Ziegler-Vista-000001-delta.vmdk" : closed.

2009-07-30 17:24:38.914 'BaseLibs' 16384 info DISKLIB-VMFS : "/vmfs/volumes/49c01410-a48d3c08-4eca-00188b44fa90/Ziegler-Vista/Ziegler-Vista-000002-delta.vmdk" : open successful (21) size = 21474836480, hd = 0. Type 8

2009-07-30 17:24:38.914 'BaseLibs' 16384 info DISKLIB-VMFS : "/vmfs/volumes/49c01410-a48d3c08-4eca-00188b44fa90/Ziegler-Vista/Ziegler-Vista-000002-delta.vmdk" : closed.

2009-07-30 17:24:38.939 'BaseLibs' 16384 info DISKLIB-VMFS : "/vmfs/volumes/49c01410-a48d3c08-4eca-00188b44fa90/Ziegler-Vista/Ziegler-Vista-000003-delta.vmdk" : open successful (21) size = 21474836480, hd = 0. Type 8

2009-07-30 17:24:38.939 'BaseLibs' 16384 info DISKLIB-VMFS : "/vmfs/volumes/49c01410-a48d3c08-4eca-00188b44fa90/Ziegler-Vista/Ziegler-Vista-000003-delta.vmdk" : closed.

2009-07-30 17:24:40.689 'BaseLibs' 16384 info DISKLIB-VMFS : "/vmfs/volumes/49c01410-a48d3c08-4eca-00188b44fa90/Ziegler-Vista/Ziegler-Vista-000004-delta.vmdk" : open successful (21) size = 21474836480, hd = 0. Type 8

2009-07-30 17:24:40.689 'BaseLibs' 16384 info DISKLIB-VMFS : "/vmfs/volumes/49c01410-a48d3c08-4eca-00188b44fa90/Ziegler-Vista/Ziegler-Vista-000004-delta.vmdk" : closed.

2009-07-30 17:24:40.725 'BaseLibs' 16384 info DISKLIB-VMFS : "/vmfs/volumes/49c01410-a48d3c08-4eca-00188b44fa90/Ziegler-Vista/Ziegler-Vista-flat.vmdk" : open successful (21) size = 21474836480, hd = 0. Type 3

2009-07-30 17:24:40.725 'BaseLibs' 16384 info DISKLIB-VMFS : "/vmfs/volumes/49c01410-a48d3c08-4eca-00188b44fa90/Ziegler-Vista/Ziegler-Vista-flat.vmdk" : closed.

[AndreTheGiant]

You have a VM with an active snapshot.

Remember that snap on ESX are "critical": you must use when you need, but not forever.

Each snap grow on your free space and can slow down also disk access.

Andre

[idahoakl]

So you're saying that as the number of snapshots on a machine increases the time to reboot will take longer and longer? I can understand that as the disk gets full access slows down, but I can't believe that it takes 30 minutes to reboot. Looking through the log files it seems as if the system isn't sure which files are there and is continually checking them, and instead of checking them once it is checking them multiple times.

If having multiple snapshots on a vm kills the entire server then I believe ESXi has a huge problem!

Matt

[AndreTheGiant]

So you're saying that as the number of snapshots on a machine increases the time to reboot will take longer and longer?

I just say that the warning was related to a shapshot and suggest you to do not leave it.

The boot problem is another, and must be somewhere in your boot log.

Andre

[idahoakl]

Sounds good. I'll keep probing around. Thank you for the insight.

Matt

[AndreTheGiant]

I found similar issues on ESX 4.0, but the solution could also works on ESXi 3.5 and maybe it's you case.

See this thread:

Andre