High CPU ready time

[tomaddox]

Hi, everyone,

I have just replaced all of my ESX hosts in a cluster with ESXi 4.1 hosts of a much newer generation, and I am still seeing a significant amount of CPU ready time on many of the the VMs in that cluster. I'll run through the configuration information and see if anything jumps out:

# of hosts: 5

# of CPU cores per host: 24 (2x Opteron 6176 per host)

# of CPU cores per cluster: 120

RAM per host: 256 GB

RAM per cluster 1.25 TB

# of VMs powered on in cluster: 289

The hosts are HP BL465G7 blade servers.

Most of the VMs in this cluster are virtual desktops with the following configuration:

Guest OS: Windows XP

RAM: 1-2 GB

vCPUs: 1

VMware Tools: yes, up to date

VM Hardware version: 7 (mostly)

CPU utilization is pretty low in terms of MHz, under 25% across the board. I am seeing excursions over 20% CPU ready time in some cases, but there's no indication that the hosts are overloaded at those times.

Any thoughts?

TIA,

Tom

[jamesbowling]

Considering you have about 57 VMs per host, the resources may not be overloaded but it is possible that a VM will need to wait for scheduling time, hence your CPU Ready percentage.  A good resource for helping troubleshoot CPU Ready is esxtop.  Check out Duncan's site:  http://www.yellow-bricks.com/esxtop/

Message was edited by: jamesbowling

[FredPeterson]

Just because CPU usage is "low" doesn't mean CPU requests are low.  Its context switching basically.  Even if each guest only uses bursts of 1GHz for a second or two, if most are doing it constantly you are going to see CPU Ready Times.

http://frankdenneman.nl/2011/04/contention-on-lightly-utilized-hosts/

You really need to watch ESXTOP or batch mode output to get a much more granular (albiet 2s interval) view of what VMs are doing.  vCenters "Real Time" isn't exactly Real Time but generally its good enough.  However in unique situations like this it can be difficult to see that indeed you have overloaded your CPUs.

[edgrigson]

It might also be worth checking your power management settings as per this article;

http://myvirtualcloud.net/?p=1212

This applies to underutilised hosts so it's worth a check.

Regards,

Ed.