VMs memory allocation on ESXi 6.5

[virt_swimmer]

Hello

I am running “ESXi 6.5 Update 2” on DELL with virtual NUMA technology enabled (known as “cluster on die” on HP). I have several VMs running on the server with latency sensitivity and shares set to  ‘high’ for CPU and Memory.

For some reason, I noticed that the algorithm of VMs RAM allocation is such that memory is allocated from the same virtual NUMA till it is empty. Moreover, when there is not enough memory in a virtual NUMA, ESXi will try to allocate as much memory as possible on the busy NUMA, and only when its completely empty continue the allocation on the consecutive one. I would call that “occupy as less virtual NUMA nodes as possible” at all costs, even when it comes to splitting the memory of a VM to 2 NUMAs.

That is not what I was used to in ESXi 6.0, as in 6.0 the memory allocation algorithm was CPU aware, allocating VMs memory based on the vCPU allocation. Meaning, on a 24 CPU server, if VMs vCPUs were allocated on cores 0-5 the memory was allocated from NUMA 0, if on cores 6-11 the memory was allocated from NUMA 1 and so forth. I must say that neither on 6.0 nor on 6.5 I was not doing any special core or memory affinity what so ever and the memory allocation was all in the hands of ESXi.

My question is what can be done on 6.5 to maintain that awareness from 6.0 and what implications may rise if that is not done ?

Thank you,

Yan

[vmrale]

Hi,

as I know this vNUMA behaviour is set by design.

When a vNUMA topology is calculated, only the compute dimensions are considered. The calculation does not consider the amount of memory configured for the VM, or the amount of memory available within each physical NUMA node. Therefore, you must manually account for memory.

If the VM is memory-intensive and uses more memory the RAM available from one NUMA Node, You must change the numa.vcpu.maxPerMachineNode advanced option for this VM and spread cores and memory allocation to next nodes.

See this:

Virtual NUMA Controls

[virt_swimmer]

Thanks for your link, but I don't really see a setting that I can use to make sure that for all the VMs I will get  N%L = 100 (100% of memory is local) in memory information of esxtop.

That is something that I got easily on ESXi 6.0 (without any special configuration).

-Yan

[MikeStoica]

Check this post http://frankdenneman.nl/2017/10/05/vm-memory-config-exceeds-memory-capacity-physical-numa-node/ . Frank Denneman has a blog series about Numa and also there is a book - Host Resource Deep Dive.

[virt_swimmer]

Hello Mike,

I am familiar with this blog and the book. However, in my case I see problem in a very basic step when I have all the memory free 32GB in each of the 4 virtual NUMAs and for some reason the very first VM I am starting with RAM of 8GB is allocated on a remote node with N%L = 0 (NRMEM = 8192.00 ; NLMEM = 0).

My simple question is why ?

VMware write black on white:

" When memory is allocated to a virtual machine, the ESXi host preferentially allocates it from the home node. The virtual CPUs of the virtual machine are constrained to run on the home node to maximize memory locality. "

So is this a bug ?

Thanks

Yan

[vmrale]

Yan,

what CPU model the server has installed? How many vCPU did You assign to the VM in question?

[virt_swimmer]

vmrale,

I am running on a server with 28 CPUs (2 chips of 14 cores each - no hyper threading). My VM has 3 vCPUs and I am using "latency sensitivity high" option.

Besides I am using shares "high" for memory and CPU of the VM.

Yan

[vmrale]

virt_swimmer,

your case is interesting and I'll try to respond as good as I can, but I need more informations. I think the behaviour in question is related with "Latency sensitivity" setting. I'm looking for a resonable explanation in literature.

I understood You did set:

- latency sensitivity - high

- shares for CPU and memory - high.

Did You set the CPU reservation for this VM too? If You did, what physical CPU server has installed and what amount of CPU reservation did You set?

Did You set CPU affinity for this VM?

Thx

[virt_swimmer]

Hi,

Yes indeed, apparently it is related to latency sensitivity "high" I have set. For some reason this made ESXi to change its default behavior and ignore NUMA locality. Which I think is a bug, as this was a default in ESXi 6.0.

But the good news is that I found a way to maintain locality by doing NUMA affinity in my VMs (CPU affinity was not needed).

Anyway, thanks a lot for the assistance and sharing your thoughts.

Sincerely,

Yan

[vmrale]

I'm glad. Interesting problems are motivating to dig deeper.

Regards