Hi Josh,
This is a known issue with ESXi 3.5. During any shutdown there is a small chance that it will hang. This is a race condition and for some reason the vSphere 4.0 upgrade agents seem to be particularly good at "winning" the race.
In technical terms what's happening is that the CIM daemon (sfcbd) somehow manages to kill the init process. Since the final stages of ESXi shutdown / reboot are fired off by init, if init is dead the system is unable to shut down.
If you flip to the vmkernel log screen on alt-F12 you'll see complaints about init having died. At the moment of death there is a message that ends in "wantCoreDump : init -enabled : 0"; this only appears once and will eventually scroll out of sight. You will also see one or more "UserCartel: CartelCleanup:993 bootstrap/init UW died (0)" messages.
When init dies in this manner, the ESXi host is already sufficiently shut down that it is safe to hit a hardware reset or power switch (manually or through DRAC/iLO/IPMI/RSA remote management).
If it happened during an vSphere 4.0 upgrade, the upgrade should continue from where it left off, unless enough time has passed that the upgrade engine timed out. (I have a feeling this is about 15 minutes but I'm not very sure.)
Because the issue is caused by sfcbd, you can avoid it by shutting down sfcbd before starting an upgrade. However, be aware that this very act of shutting down sfcbd could cause the problem! On the gripping hand, it's much less trouble for your already-evacuated ESXi host to hang while shutting down sfcbd manually, outside the vSphere upgrade process.
How can we ship with this problem? Catch 22. To fix the problem on your ESXi 3.5 host we would have to ship you a 3.5 patch, for which the system would have to be shut down, possibly invoking the problem! And you wouldn't be too thrilled to be told that you have to first upgrade 3.5 to (e.g.) Update 5 before even considering vSphere.
The problem is self-correcting in that when the box hangs, you eventually kick it, it wakes up and goes on with its business.
Several aspects of this issue will be corrected in future patch work for 3.5 and 4.0. It's a pretty complex interaction and only needs to be stopped in at least one place.
Field remedy: ensure that sfcbd is not running before starting vSphere upgrade. You can do this by manually shutting it down (if it hangs, reboot and do it again); or by disabling CIM. has a nice writeup.
In the context of a vSphere upgrade you'll want to:
move or shutdown VMs (probably telling VC to put the host into standby mode);
manually stop sfcbd (optionally by disabling it and rebooting); if the empty host hangs at this point, just power cycle and repeat;
start vSphere upgrade.
We've found this issue very difficult to reproduce in the lab(), so it's unlikely you'll need to power cycle even once. ()Not impossible. I'm by no means claiming this problem doesn't exist, only that it's not very common.
I apologize for the inconvenience. I didn't cause it, but I was on the team trying to resolve it just before the GA cut, and we did not succeed in finding a foolproof way of avoiding it in all cases.
>Bela<
