ESXTOP, PCPU and Multi-Cores

[clikdude]

Using a Quad CPU (Dual-Core) Server (AMD Opteron architecture), ESX v3.0.1 reports (8) PCPUs available.

With several (2) vCPU Windows 2003 Servers, we are finding "minimal" %RDY numbers at 8% or under. However, periodically, we receive 10% - 30% %RDY numbers for a few seconds. When reviewing the PCPU% at the header of the esxtop cpu monitor screen, the first and second CPUs report 15 - 30% utilization, and the others report 1% or less utilization.

Why is ESX possibly not scheduling those machines which periodically report 10 - 30 %RDY, on the other PCPUs? It looks like the workload isn't "cpu intensive", hence the PCPUs don't jump to 70-80%, but I would at least expect more of the PCPUs to look "used", so when I see 10 - 30 %RDY, at the very least, the other PCPUs should look higher than 1% utilization, no?

Just want to understand this dynamic and internal algorithms better. I would also like to avoid having to use CPU affinity, to ensure VMotion compatibility.

Thanks, - clikdude

[FredPeterson]

Multiple vCPU's have to be scheduled at the same time. So that means even if there are 3 "CPUs" available and 3 VM's waiting, but 2 of those three are 2 vCPU, only two of the VM's will get scheduled and one of them is going to be the 1 vCPU, so that 2 vCPU will sit waiting, regardless of the percentage used.

I wouldn't always equate % used to % ready - plus you have to take into account shares.

[mreferre]

There could be a number of reasons. Some of them are known (see Fred's response) some of them are just not.

There are a number of "unknown performance side effects" when virtualizing physical workloads.........

I remember once I moved a certain application on top of a 16-core ESX server and although 1) the application running on physical was consuming relatively little in terms of resources and 2) that vm was the only one running on the 16-core systems .......... I could see a much higher cpu utilization (compared to physical) and an average of %READY in the range of 15/20% (one would imagine 0 given the green field of resources available for a single vm).

I usually say there must be a reason for which the mainframe is still king when it comes to virtualization technologies. There you push a button and you know 100% of the times what's going to happen. Here you change a value and ..... you keep your fingers crossed (and no ... I am not necessarily implying that the mainframe is a better platform for your workloads ...... I have just reported a fact ).

Massimo.