topic ESX 7 NUMA problem in VMware vSphere™ Discussions

ESX 7 NUMA problem

wickyR — Thu, 28 Apr 2022 14:53:17 GMT

Hello,

I have been given to analyse a problem related to a NUMA vCPU placement and memory allocation.

The physical server is based on Intel Xeon E7-8880 V4 having 2 Processor sockets and 22 processor cores per socket with 88 logical processors.

The output of the command sched-stats -t numa-clients shows very imbalanced distribution of vCPUs and significant remote memory access. (Please see attached image). The system has NUMA.PreferHT=1 already set.

My question is, why vCPUs are unevenly distributed and there is significant remote memory access? Where should I look for more information?

Thank you very much for your time and answers.

Roshan

Re: ESX 7 NUMA problem

vbondzio — Fri, 29 Apr 2022 12:14:47 GMT

NUMA clients migrate across physical NUMA nodes due to changes in load, contention and for locality optimization. This doesn't have to be an issue but it might under some circumstances. See https://kb.vmware.com/s/article/2097369

For a better overview, run:

for numaOption in $(sched-stats -h | sed -n 's/.*: $n.*$$/\1/p'); do echo -e "\nsched-stats -t ${numaOption}"; sched-stats -t ${numaOption}; done

and post the text on pastbin.com / link that here.

You might also want to run / pastebin the output of: https://github.com/vbondzio/sowasvonunsupported/blob/master/pci2numa.sh

Re: ESX 7 NUMA problem

wickyR — Fri, 29 Apr 2022 13:02:34 GMT

Thank you very much for your answer!