We've been running Vmware ESXi in our environment for a few years already and its been giving alot of benefits. Typically we run our virtualization hosts with 4-way 32 Cores and 40 Core servers and we recently deployed a large VM with 16vCPUs in a production environment.
An issue we suspect we've hit is running the VM with too many vCPUs that it might be actually having some negative affect on the server due to high CPU time measure inside the VM ~3000ms but at the vCentre level when we look at the CPU ready time the figure is quite low less than a few percent. When we run some SQL statements its showing the ~3000ms.
I've read in the past that having very large VM's (high vCPU count) can at times have a negative affect on performance due to the CPU scheduling mechanism of VMware it must wait for 16 free vCPUs when scheduling such large VM's. To isolate the problem we've currently only placed 2 VM's on our physical host of 40 Cores running 2 VM's with 8vCPU and 16vCPU - and it seems were seeing this issue. Keeping all thing equal to prove the issue we also did another test running similar SQL statements on a smaller VM 4vCPU and found the SQL execution time of the script was the same but the CPU time on the VM measured within the OS was also actually lower as well almost 50% less.
Has anyone encountered similar issue or have any comments - what were thinking of trying next is actually isolating the VM host for the single 16vCPU VM itself and running the SQL queries again and measure the execution time and CPU time as well.
How large are your NUMA-nodes?
Increase VM's CPU share to High and check for some time..
What do you mean cpu time? cpu time in windows?
If the CPU RDY is low then contention or cpu resources would most likely not be your issue. I would look at your cpu wait time then minus the cpu idle if the remaining number is high then there is IO operation that is slowing it down disk? memory?
NUMA should be looked at but if your running 16 vCPUs then ESXi will make the NUMA visible to the guest so it can use it more efficiently.