NUMA memory affinity allocation

[cianfa72]

Hi,

I'm working on a emulated networking lab running on an ESXi 5.5 dual quad-core Xeon server (HP DL360). The lab is composed by 14 nodes each one implemented by a 4-way VM instance.

I've set NUMA memory affinity properties for each VM in order to fix and statically distribute NUMA memory allocation for all of them (see picture below for VM named "T1"). Basically 7 VMs have been assigned to NUMA node 0 while the other ones (7) to NUMA node 1.

The weird thing i can see is that, however, continues to exist a small quota of non-local VM allocated memory (N%L column shown by esxtop + 'm' + 'g' for some VM world groups is about 98% or lower)

Is that an expected behaviour ? Thanks.

[cianfa72]

Anyone?

[npadmani]

Associate Memory Allocations with a NUMA Node in the vSphere Client You can specify that all future memory allocations on a virtual machine use pages associated with a single NUMA node (also known as manual memory affinity). When the virtual machine uses local memory, the performance improves on that virtual machine. The following conditions apply to memory optimization with NUMA:

■ The NUMA option is available on the Advanced Memory Resources page only if the host uses NUMA memory architecture.

■ Affinity settings are meaningful only when used to modify the performance of a specific set of virtual machines on one host. This option is not available when the virtual machine resides on a DRS cluster. All affinity values are cleared when you move the virtual machine to a new host.

■ You can specify nodes to use for future memory allocations only if you have also specified CPU affinity. If you make manual changes only to the memory affinity settings, automatic NUMA rebalancing does not work properly.

■ Checking all the boxes is the same as applying no affinity.

Third bullet point is giving you an idea why it might not be working properly.

Ref: https://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.vm_admin.doc_50%2FGUID-D3B4...

Setting up only memory affinity to NUMA node will still allow you to do vMotion and if you move VM from one host to another, those settings are not going to be valid any more. so make sure that you do CPU affinity settings also. See if this helps.

[cianfa72]

Thanks for your reply....but what if there is no more available memory on the target NUMA node (though NUMA memory affinity VM advanced settings) for future allocation ?

[npadmani]

I was going through following documentation article.

VMware vSphere 5.1

it's stating clearly that

Manual NUMA placement might interfere with ESXi resource management algorithms, which distribute processor resources fairly across a system. For example, if you manually place 10 virtual machines with processor-intensive workloads on one node, and manually place only 2 virtual machines on another node, it is impossible for the system to give all 12 virtual machines equal shares of systems resources.

simple meaning is, if you are mentioning explicit settings towards CPU, Memory and Numa Placement affinity settings, ESXi might end up facing trouble doing fair distribution of resources. But it's not system's fault, in that case we should have trusted ESXi host itself to do this for us.

There's one more Note in the same artcile

Note

In most situations, the ESXi host’s automatic NUMA optimizations result in good performance.

[cianfa72]

Even with CPU affinity set for the same CPUs belonging to the NUMA affinity node configured, I can see memory remotely allocated (numa home node - NHN as shown by esxtop- is the right one for each VM however continues to exist non-local memory allocations) :smileyconfused:

My goal is to have NO memory remotely allocated for VMs in order to optimize latency in accessing NUMA node local RAM (remember in my scenario VMs are implementing networking nodes !)

[npadmani]

when it comes to NUMA node affinity settings,

I don't think it's 'Must' kind of assignment, it sounds more like 'Should' kind of assignment.

see this link: VMware vSphere 5.1‌

Last line on that post says

Completing these two tasks ensures that the virtual machine runs only on NUMA node 1 and, when possible, allocates memory from the same node.

so basically there's a chance that you do cpu affinity and memory numa node affinity settings and still you see memory being taken from Remote Node. as you can notice already in your case.