ESXi CPU Scehduler documentation

[COS]

I had it before but now I can't find it. Can someone point me to the documentation that explains how the CPU scheduler works?

I'm making my case on why shutting off Hyperthreading is more advantageous.

Thanks

[Troy_Clavell · Accepted solution]

you may want to look at http://www.vmware.com/files/pdf/perf-vsphere-cpu_scheduler.pdf as well as http://www.vmware.com/files/pdf/techpaper/Whats-New-VMware-vSphere-50-Performance-Technical-Whitepap...

I don't believe turning off hyperthreading is beneficial.  Hyperthreading is supported and recomended.

http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf

[weinstein5]

I would like to see this too because I was always under the impression you want Hyperthreading enabled to provide the vmkernel more opportunity  to schedule a vcpu particularly since the HT teachnology has become stronger -

[COS]

Thanks!!!

I'm a little confused on how it's beneficial.

The scenario always goes....

CPU0 (Real Core) is at 100%

CPU1 (HT Core) is at 0%

CPU2 (Real Core) is at 25%

CPU3 (HT Core) is at 25%

A users thread is launched, which core does it go to? The least used "CPU" right? CPU1? And if the "Real" core is 100% used, the users thread sits there and twittles it thumbs till the "Real" core is more available. Now thing's might have changed and that now the scheduler can shift/move the users thread to another core but last I read, it can't move the thread once it's sent to that "CPU".

If I'm correct, then what use it the HT core if the users thread has to wait? What if some programmer had a infinate loop in code somewhere? Hung process eating up CPU? Bad Java execution (don't get me started on that!)...etc...

Correct me if I'm wrong please.

[Troy_Clavell]

I am no CPU expert, but the scheduler will try and keep all cpu cycles of a VM on the same socket, therefore utilizing all available core, which include hyperthreaded cores.

[joshodgers]

So it depends heavily on the workload of the VMs if HT is benefical or not.

For example, for Virtual desktops, HT is awesome for the increased CPU scheduling capabilities.

For a cluster full of mission critical servers which use high percentages of there assigned vCPUs, then HT is likley not to benefit, or even reduce performance.

Make sure VMs are sized to suit your physical CPUs

eg: For a 4 core Server , 1,2 and 4 vCPUs , For a 6 core , 1,2,3 and 6 vCPUs.

The reason for the above is to increase scheduling efficiency.

Also... ensure your VMs are right sized. (http://joshodgers.com/2012/07/25/vm-right-sizing-an-example-of-the-benefits/)

[COS]

We "Right sized" as much as the VM owners/Admins would agree to....lol :smileysilly:

All in all, it looks pretty good. We have 2 envoronments, Prod & Dev. Prod is all mission critical Servers. 360+ VM's with some 8vCPU many more with 4vCPU and the bulk are 2vCPU a total of 224 LOGICAL cores but 224/2= 112 Real Cores.

So the chances are high that all 112 Real cores (or all cores in a socket) get busy just long enough for the scheduler to send threads to a HT core, leaving only the HT core looking idle just as my example is above.

In such a scenario, I highly believe HT is counter productive. Sending a users thread to a HT core bound to a 100% utlized Real core is just asking users to call the support desk.... "My application is running slow but my co-worker using the same application is just chugging right along quickly...."..lol

Thoughts?

[joshodgers]

You may be correct, and you can prove the contention by checking the CPU ready of your VMs.

This can be done in the Performance Overview tab, the top left graph shows CPU Util% and Ready, anything over 200ms per vCPU should be looked at.

Make sure you set the graph to the last week or month to get a good idea of contention during peak/offpeak.

I guess your application owners are similar to my customers, they dont believe right sizing. Just send them to my blog and hopefully they will get it.

Otherwsie you will be adding more hardware to the cluster :smileyangry:

[COS]

"I guess your application owners are similar to my customers, they dont believe right sizing"

lol, if they were willing to pay for it, we weren't going to complain. :smileysilly:

My primary concern is overselling the vCPU's because of the perception of HT.

[joshodgers]

If you disable HT, your performance would improve assuming you dont run into contention from the lack of cores (Full & HT) available to the CPU schedulure.

Probably something you should schedule out of hours to test.

If you have low CPU ready ie: <200ms per vCPU, you could consider turning off HT, but if you have CPU ready >200ms on your VMs already, disabling HT would increase the CPU Ready and likley result in poorer performance.

Id suggest if you disable HT, you will likley require one or more additional hosts in the cluster, but performance would improve for your CPU intensive workloads as every vCPU would have a physical core.

Please remember to mark responses as correct or helpful if applicable. Thx