Here's a new and very troubling isssue. At approx. 6:18 last night, one of our ESX servers in a 6-node cluster (all HP Proliant DL360G5 systems running ESX 3.0.2) suddenly went into a "Host is not responding" state in VirtualCenter. From that moment forward, the following applied:
No VI client connectivity to that host either via VirtualCenter or direct via VI3 client.
No http response.
No SSH response.
No at-console response to console/specialized VM that provides host interaction.
Could ping the host / service console address, however. Only sign of 'life'. Sigh.
iLo (v2) diagnostics on that host show no signs of foul play whatsoever on the hardware front. Fans all running, local disks fine, temps reasonably low, no sign of ASR/reset in logs, etc.
A complete mystery! All other hosts in this same cluster have literally identical hardware, firmware levels, and ESX 3.0.2, and have no problems.
There were 4 VMs running at the time the host decided to virtually drop off the planet---and if there is any good news it's that they all remained running and quite functional as I type this nearly 24 hours after the host 'died' in this sense, but all VMs show up as 'disconnected' in VC (not surprisingly, given the 'not responding' state of their host in VC as well), so there's no way to manage them at all, vmotion them, shut them down via VC, etc.
So, tonight, after advising users of 'maintenance', I must go into each guest, shut each down gracefully, bring the host down hard (have to since it's completely unresponsive in the software sense), bring it back up, confirm its guests are up, subsequently migrate guests to other hosts in the cluster, put host into maint mode, remove host from the cluster cuz I don't want any part of that host until root cause of this quasi-disaster is determined-all so I can extract the vm support files for submission to VMware per the ticket I opened with them this morning and after the engineer came in via WebEx, took a look around, confirmed all of what I noted above, and agreed that this is all that can be done right now. They must, of course, have the support output toHOPEFULLY--find root cause. This type of outage/crash of an ESX host is of course the classic 'unacceptable' (from both management and my own mouth) because it negates all of the HA/DRS/Vmotion benefits and, well, it's unacceptable/unanticipated. Ugh. Not good.
Hopefully this issue will turn out to be a hardware problem and I thusly won't be able to blame VMware, but as I confirmed with the engineer assigned to this case, hardware doesn't appear to be the case at all. Still, after the host is back up and out of the cluster, etc. I'm going to run a 'fsck' command on the ESX host just to see if there's any possible problem/s with the disk/array/filesystem (per the VMware tech's suggestion).
Regardless of outcome, problems such as this are particularly bad because VMs are 'stuck' in a 'disconnected' state and cannot be migrated via Vmotion. I do hope we can get to the root cause of why a seemingly fine ESX host with a very stable history would suddenly/spontaneously lose all communications with the console, SSH, HTTP and VirtualCenter/VI client ports. It basically 'died' in that sense, without any evidence of ever going 'down'. Very disconcerting/frustrating.
Has anyone ever heard of an ESX host just 'dying' like this with these symptoms? Any suggestions/thoughts? Hopefully, the ESX logs will confirm what happened when VirtualCenter initially reported the host system as being down.
