Advanced virtual machine settings for vCPU

[AutoEng]

I have a host with two 16 core processors, 512GB Memory, which shows up as four NUMA nodes in ESXi, with 128GB Memory per NUMA node.

I have a 16 vCPU virtual machine, configured as 2 sockets with 8 cores each. It has 256GB Memory so it's using 2 NUMA nodes.

Do I need to adjust this setting if I want to increase vCPU from 16 to 20?

numa.autosize.vcpu.maxPerVirtualNode = "8"

Thanks

[ZibiM]

Can you explain a bit how come you have four NUMA nodes with just 2 CPUs ?

Are these some kind of Opteron CPUs ?

As I understand this setting you don't need to change it when you increase the amount of vCPUs.

This setting is for defining the minimum number of cores above which vSphere kernel will start to create additional vNUMA nodes

With the default setting = 8, when your VM has 8 cores it will have just 1 vNUMA node and vNUMA autosizer will set your VM on 1 socket with 8 cores.

If your VM will have 10 cores, then it will have 2 vNUMA nodes and vNUMA autosizer will set it on 2 sockets with 5 cores.

Why it does not matter in your case:

1. The default setting seems to be aligned with your physical NUMA size -> 2 CPU * 16 cores / 4 NUMA nodes = 8 cores

2. If you have CPU settings manually defined, the vNUMA autosizer does not intervene, and this setting is not used

3. If you want to increase the vCPU above 16, you may need to look for amount cores divided by 3. It looks like you are going into VM that will utilize 3 physical NUMA nodes.

You may want to check this fling for further results:

Virtual Machine Compute Optimizer | VMware Flings

Please bear in mind that it has somewhat limited support for more exotic NUMA environments

[AutoEng]

Can you explain a bit how come you have four NUMA nodes with just 2 CPUs ?

The hosts make use of sub-clustering, due to two cache controllers on each physical processor, so two NUMA nodes per physical processor show up in ESXi. It's a BIOS setting enabled by default when selecting the "Virtualization - Max Performance" profile in the BIOS.

3. If you want to increase the vCPU above 16, you may need to look for amount cores divided by 3. It looks like you are going into VM that will utilize 3 physical NUMA nodes.

It should still be two NUMA nodes surely, not three? Each host has dual 16 core processors. If I am using 10 per processor would it not be two NUMA nodes?

Hence my question about that default setting of numa.autosize.vcpu.maxPerVirtualNode = "8", as I'm not sure how it calculates this.

[ZibiM]

You know i was not aware about this SNC feature on the Intel CPUs

Looks like I always managed to avoid this by using memory interleaving feature.

I checked over the details of this feature and I'd say this is smthg that is suited for small VMs (small enough to fit subNuma domain) with low latency requirements.

This due to the fact that Intel CPU is big enough to introduce latency just by the fact the data needs to cross the distance in the ring mesh topology.

If you have VMs that are in need of the low latency, then dividing the CPU in halves and forcing the VM to use closest memory controller should provide more benefits than any downside causing by halving memory throughput.

I'd say that when the VM is big enough and needs to use more than 1 NUMA node, then you ought to be better with this feature disabled.

This will be especially true for the VMs with much bigger need for the memory (eg. when it's actually a RAM size and not number of vCPUs that driving the need for more NUMA nodes). In such scenario memory throughput matters a lot.

BTW I found a comment that looks like it will be compatible with your scenario:

Virtual Machine vCPU and vNUMA Rightsizing - Rules of Thumb - VMware VROOM! Blog - VMware Blogs

I also found that vNUMA behaviour is changing between the vSphere versions.

There was big change in the vSphere 6.5 and there was also something in the vSphere 6.7 u3

From my experience with vSphere 6.7 u3 and AMD Rome CPUs, I'd say that the numa.autosize.vcpu.maxPerVirtualNode setting is changing during the startup in order to reflect CPU characteristics.

[AutoEng]

From my experience with vSphere 6.7 u3 and AMD Rome CPUs, I'd say that the numa.autosize.vcpu.maxPerVirtualNode setting is changing during the startup in order to reflect CPU characteristics.

Yes I have noticed this on 6.7u3.