Great resource, thanks!
In our particular case we found a host seemingly disconnected generating a host connection state alarm because the HA agent in the cluster had an error. The host and all VMs showed up as if they were disconnected, but we could remotely (RDP, etc.) access all of the VMs. We couldn't SSH into the console, but we could ping it. We couldn't log in at the ESX terminal itself. For the local terminal, a login prompt would display but no authentication would take place. IIRC, we couldn't even enter a password.
We had a failed disk in the host. My co-worker had set it up as a RAID5 with a hot spare, but something must have really hosed up the config, because when I looked at the RAID BIOS it displayed a RAID 0 config, with a failed disk, and a hot spare that was basically useless since the adapter treated the array as RAID 0. Not sure if this was a mistake in initial setup or some kind of bad corruption.
Since we could remote into the VMs, we were able to shut them down gracefully. After that, we rebooted the host and since it had a RAID 0 array with a failed disk, there was no ESX to load. I guess we had essentially been running ESX off memory when that disk failure occurred. Pretty amazing that those VMs kept chugging along.
After the host rebooted, HA was able to detect a downed host and migrated all VMs to the other online hosts. HA brought them all back online, since its last known state of the VMs was that they were online (almost like zombie VMs).
We had one issue, however, which brought me to this thread. I missed shutting down 1 VM before the failed host reboot. So VC thought this VM was running, disconnected on the failed host and I couldn't migrate it. In reality, it had hard crashed when I rebooted the failed host and was offline.
After removing the host from VC, the VM was also removed from VC and I was able to add it back to the inventory per this thread.
