Hi Sylvie
Apologies if any of this gos over stuff you've already tried! CPU affinity (and indeed latency sensitivity) should do what you want for VMs, but according to Denneman & Hagoort's deep dive[*]: "Be aware that it never dedicates the physical CPU to the VM as the VMkernel schedules all its processes across all available pCPUs, regardless of any custom setting a VM has."
I assume you did provide 8 cores worth of CPU reservation too alongside the latency sensitivity? How many pCPUs do you have and what type of CPU are they?
Things I discovered when chasing down performance issues around CPU clock frequencies:
1) Turbo Boost - which increases clock speed on cores temporarily (within thermal constraints)
2) AVX Turbo states - if any AVX2 instructions are used, the core speed (or all cores dependent on processor model) is dropped to mainatin thermal limits, and is held at the lower value for 1ms after the klast AVX2 instruction is executed (that shouldn't be a total stall, though, just a core frequency reduction).
See, for example:https://en.wikichip.org/wiki/intel/frequency_behavior
On C-states: D&H again (though they say nothing about AVX2 clocking details):
"If absolute consistency is necessary, we recommended disabling Turbo Boost Technology as well. Again, we recommend enabling this feature in the BIOS, but programmatically adjusting this in vSphere. Select the Custom user-defined power management policy in vSphere and select the Advanced System settings and set the value to 99 of the Power.MaxFreqPCT setting.
Please note that the custom policy defaults to Balanced. If you need to disable TB, but also want to set MinFreqPct to 99, UseCStates to False."
My interest in this is that I too have been seeing short duration scheduling latency spikes (in my case to 10s of milliseconds) - but in my case I need consistency (latency *always* below 10ms) rather than ultra-low latency per se. My other perception is that 6.5 produces higher %RDY values for some of my workloads than 6.0 (I started a thread here a few days ago about it, no replies!) and I have a call open with VMWare about it, so I think there is a perfomance change in the 6.5 scheduler. Are you in a position to run the workload on 6.0 for comparison?
Maybe there are some ideas there, but I have no knowledge about scheduler intricacies beyond what is published (I think it's supposed to "just work" so it's hard to find anything that goes beyond what's in the manual or the Deep Dive white paper). I'd love there to be a "vSphere Internals" course so as not to have to guess/experiment to answer some of these questions.
John
[*] "VMWare vSphere 6.5 Host Resources Deep Dive", Frank Denneman, Niels Hagoort. Available by signing up at Rubrik | eBook: VMware vSphere 6.5 Host Resources Deep Dive
