ESXi 5.5 CPU affinity behaviour

[cianfa72]

Hi,

I've tried to set CPU affinity for a VM (Linux OS 64bit guest) running on ESXi 5.5 on a host equipped with dual socket quadcore HT Intel CPUs (each CPU socket has 4 x 2 = 8 HT logical processors).

Just to do some test, under VM Preferences-> Resource tab -> Advanced CPU I set: 0,1,8,9 in order to assign 4-way VM vcpus to logical processors from either 2 CPU sockets

Then when pressing 'm' 'f' 'g' from ESXTOP (in order to check NUMA assignment) the following sequence is shown under NHN (Numa Home Node) column: 0,1,2,3,4,5,6,7,8,9,10,11,12,13

It seems CPU affinity setting is not effective

Can you help me ?

[Alistar]

Hi there,

first if you would like to have your  vCPUs set to the acutal physical cores, you would use 0,2,8,10 (the "even" numbers including 0 are the true physical cores, while the odd numbers excluding 0 are hyper-threaded cores).

This is an interesting experiment you are proposing, however what real-world usage this would have? ESXi always tries to keep your VMs on one single Physical CPU where possible and starts to span across NUMA when the vCPU count is higher than the Physical CPU count of one node. Perhaps this applies in this example as well. You also could try changing the vNuma to start up at >3 vCPUs instead of the standard >8 vCPUs. Is your vCPU set to 2 sockets x 2 cores?

[MKguy]

You should avoid CPU affinity settings like the plague.

To achieve what you're trying to do, configure the VM with 4 sockets and 1 core per socket, then set the following advanced VM parameters:

numa.vcpu.maxPerMachineNode=2

numa.vcpu.min=4

The first option will ensure that no more than 2 vCPUs are assigned to a single physical NUMA node. (With this you will see NHN 0,1 for the VM in esxtop.)

The second option will ensure that a virtual NUMA topology is replicated in the GuestOS if you have 4 or more vCPUs. (numactl --hardware will show you 2 corresponding NUMA nodes in the Guest.)

I've tested this behavior myself with a Linux VM.

More details on these advanced parameters are described here:

https://pubs.vmware.com/vsphere-55/index.jsp?topic=%2Fcom.vmware.vsphere.resmgmt.doc%2FGUID-89C52376...

[Alistar]

Beware mate,

the numa.vcpu.min=4 dictates greater than, not equal or greater I was confused by this the first time as well, wondering why vNUMA would kick in after 9 vCPUs were assigned and not 8 as the logic would dictate

[MKguy]

No, as the name implies it's the minimum value. 4 is alright with 4vCPUs and I have tested this myself.

I was confused by this the first time as well, wondering why vNUMA would kick in after 9 vCPUs were assigned and not 8 as the logic would dictate

I know a lot of people including me first mistakenly thought vNUMA would kick in with 8 vCPUs already because in the early days many people writing about this topic said "more than 8" without stressing the fact it really has to be at least 9.

But there is no ulterior logic involved here, this is simply because VMware deliberately choose "9" as the default value for this parameter, as is is documented here:

https://pubs.vmware.com/vsphere-55/index.jsp?topic=%2Fcom.vmware.vsphere.resmgmt.doc%2FGUID-89C52376...

[cianfa72]

Alistar wrote:

This is an interesting experiment you are proposing, however what real-world usage this would have? ESXi always tries to keep your VMs on one single Physical CPU where possible and starts to span across NUMA when the vCPU count is higher than the Physical CPU count of one node. Perhaps this applies in this example as well. You also could try changing the vNuma to start up at >3 vCPUs instead of the standard >8 vCPUs. Is your vCPU set to 2 sockets x 2 cores

yes, VM vcpu configuration settings is 2 socket x 2 cores (each).

At this stage my goal is just to better understand how NUMA load balancer placing and CPU scheduling work...as said before i do not understand what esxtop NUMA view displays when trying to pinpoint VM 4 vcpus to specific HT cores (0,1 for numa node 0 and 8,9 for numa node 1): it seems NUMA scheduling is not involved in placing vpcus for that VM

A related question: as in the image attached when setting VM HT sharing to Internal, ESXi enables CPU cores sharing just for vcpus belonging to the same VM. Now, from my understanding, NUMA scheduling maps NUMA clients on NUMA home nodes taking in account just the number of CPU cores per node (socket) not including hyperthreading (2x). That means vpcus belonging to the same 4-way VM are assigned to different cores of the same NUMA node (here we are assuming an host with 2 quad core cpu sockets). If that is true now CPU scheduling can just choose to schedule a VM  vcpu on a full or a HT logical core but it is not the case vpcus of the same VM are scheduled on logical core (HT logical core) of the same CPU core.

The only way to do that is to set the numa advanced option numa.vcpu.preferHT = TRUE in order to force NUMA scheduler assignment to take in account number of cores including hyperthreading (logical core) placing VM vcpus (possibly) on HT threads of the same CPU core

Does it make sense ?