A couple of weeks ago, a guest RHEL VM, which is only under light load, locked up and became unresponsive, requiring a reboot for recovery. SSH, HTTP and other services had stopped, and after connecting to the console for 3-4 minutes, I eventually got a login prompt, but never got to the point of being able to issue commands. Reboot has fixed things, and we have rebalanced memory load across our ESX cluster to prevent re-occurrence.
Looked at the performance graphing in VIC:
- it was using about 135MHz in cpu cycles
- zero or negligible I/O
- there was about 2.5-2.7GB of memory ballooning in action, and some swapping
- there was a memory alarm on the physical ESX host, but it still had about 10GB free in physical RAM
Guest OS has 4GB RAM allocated. It is in a resource pool which receives 4000 memory shares out of a total 30000, and currently shares this pool with 5 other VMs on the same physical host. CPU shares are similar, but CPU was in abundance. Actual resource use is ticked as 'unlimited' for CPU/Memory, and shares are equivalent across VMs in the pool.
I have done some research through community threads (such as http://communities.vmware.com/thread/133290) and various memory allocation documents (http://www.vmware.com/files/pdf/large_pg_performance.pdf, http://www.vmware.com/pdf/VI3.5_Performance.pdf, and the Resource Management Guide in particular.)
What appears to have happened is that the idle memory tax, in combination with the resource pool shares, were causing this low activity VM to experience a higher level of ballooning than the default 75% setting we have for the cluster, to the point where it had swapped some critical OS memory and became unresponsive.
Is it possible that a RHEL / linux VM can be so inactive that it reaches an inoperable equilibrium state for balloning vs swap and services drop out entirely?
