Hey,
Yesterday we had two hosts fail in our cluster. I believe this was because we over commited resources and it started causing problems with our HA/DRS setup.
On each host we had errors like this:
5/4/2009 11:32:24 AM, HA Agent on <host> in cluster <> in <> has an error
These errors would happen every 3 minutes on both hosts. On a few, there was a related event error that said "Insuficient resources to satisfy HA failover level on cluster <cluster> in <datacenter>."
About 30 minutes into the errors, both hosts showed a disconnected state in VIC.
At this point I went to the console to try and restart the management services so that I could reconnect them to the cluster and adjust my resources. The console though would freeze, unable to move any highlights.
After searching the internet, there were a few suggestions to try going to the terminal and issuing /sbin/services.sh restart. This would hang on stopping vmware-aam and would sit there and do nothing.
Luckily the VM's were still running, so we were able to take a maintenence window and take the VM's down and reboot the blades.
Has anyone run into this situation and if so any solutions so that we don't have to take an outage hit because of this?
matt
