VMware Cloud Community
DITGUY2012
Enthusiast
Enthusiast
Jump to solution

How to set CPU affinity and memory.

We have explicit recommendations from our application vendor to set CPU affinity for one of our VMs that runs their product. However this process is not entirely clear to me despite available docs. We're on ESXi 5.1 Enterprise Plus.

Looking at the Resources tab for a VM's settings you see under CPU:

Shares (I'd make this high enough so that it's never going to contend with others right?)

Reservation

Limit

The requirement is to set aside X cores for the VM so that there is never waiting or resource contention. They would also potentially like to set up memory affinity tied to the same socket for NUMA node efficiencies.

We have a host with 2 8-core processors with 256GB of memory. I'm not worried about the memory affinity. That should be as simple as checking the "reserve all guest memory" correct?

Is it simpler to just set up a resource pool for that CPU and Memory? The only concern I have there is if the CPU is using the same channels for memory. We don't want a CPU in one socket to use memory from another socket's channels for this VM.

And it's too late to go back to physical for this system.

1 Solution

Accepted Solutions
AndySimmons
Hot Shot
Hot Shot
Jump to solution

Eeghads! Just FYI, CPU affinity is almost never what a vendor actually wants, even though they think they do. I can imagine there are valid use cases, but I've spoken with several vendors who recommended this, and they were all mistaken.

From what I gather in your summary, the application is sensitive to CPU latency. If the physical compute capacity is there, shares and reservations will address latency. Is latency the only reason they want to set affinity, or are there others? If there are others, find out what they are, because there's a good chance those are also invalid reasons. CPU affinity tends to exacerbate any issues you're having with CPU ready times, and can often cause the application MORE CPU latency than it would without it. Yes, you can set CPU reservations to address those issues for that particular VM, but you've just made life more difficult for the CPU scheduler. DRS doesn't factor in CPU ready time either, which means it won't necessarily be moving those other VMs off that host, even if they're co-stopping like crazy. While we're on the subject, this VM would need to be excluded from automatic DRS migrations, since you can't vMotion with CPU affinity configured. This also means the VM needs to be powered off whenever its current host goes into maintenance mode.


Anyway, checking "reserve all guest memory" won't affect memory locality. It'll just back all virtual memory pages with physical memory pages (another interesting side effect is that the VM no longer creates a swap file on boot). If you REALLY have to set CPU/memory affinity, the memory affinity can be set by editing the VM settings, resources tab, select Advanced Memory, and you can configure NUMA there. Also, if you hadn't seen it already, CPU affinity is configured in the resources tab as well, under Advanced CPU.

I'd strongly recommend you challenge the vendor and ask them if CPU reservations are enough. The CPU scheduler already is NUMA aware. How many vCPUs are they asking for, anyway? Unless they want more than 8, it's unlikely to split the VM.

-Andy VCAP5-DCA, VCP-DV 4/5, MCSE, space camp graduate.

View solution in original post

7 Replies
julienvarela
Commander
Commander
Jump to solution

Hi,

Here a good study and explain about CPU affinity on numa architecture. http://frankdenneman.nl/2011/01/11/beating-a-dead-horse-using-cpu-affinity/

Julien.

Regards, J.Varela http://vthink.fr
ScreamingSilenc
Jump to solution

http://www.vmware.com/files/pdf/techpaper/VMware-vSphere-CPU-Sched-Perf.pdf should help you.

Please consider marking this answer "correct" or "helpful" if you found it useful.
AndySimmons
Hot Shot
Hot Shot
Jump to solution

Eeghads! Just FYI, CPU affinity is almost never what a vendor actually wants, even though they think they do. I can imagine there are valid use cases, but I've spoken with several vendors who recommended this, and they were all mistaken.

From what I gather in your summary, the application is sensitive to CPU latency. If the physical compute capacity is there, shares and reservations will address latency. Is latency the only reason they want to set affinity, or are there others? If there are others, find out what they are, because there's a good chance those are also invalid reasons. CPU affinity tends to exacerbate any issues you're having with CPU ready times, and can often cause the application MORE CPU latency than it would without it. Yes, you can set CPU reservations to address those issues for that particular VM, but you've just made life more difficult for the CPU scheduler. DRS doesn't factor in CPU ready time either, which means it won't necessarily be moving those other VMs off that host, even if they're co-stopping like crazy. While we're on the subject, this VM would need to be excluded from automatic DRS migrations, since you can't vMotion with CPU affinity configured. This also means the VM needs to be powered off whenever its current host goes into maintenance mode.


Anyway, checking "reserve all guest memory" won't affect memory locality. It'll just back all virtual memory pages with physical memory pages (another interesting side effect is that the VM no longer creates a swap file on boot). If you REALLY have to set CPU/memory affinity, the memory affinity can be set by editing the VM settings, resources tab, select Advanced Memory, and you can configure NUMA there. Also, if you hadn't seen it already, CPU affinity is configured in the resources tab as well, under Advanced CPU.

I'd strongly recommend you challenge the vendor and ask them if CPU reservations are enough. The CPU scheduler already is NUMA aware. How many vCPUs are they asking for, anyway? Unless they want more than 8, it's unlikely to split the VM.

-Andy VCAP5-DCA, VCP-DV 4/5, MCSE, space camp graduate.
DITGUY2012
Enthusiast
Enthusiast
Jump to solution

Hi Andy. IBM is the vendor. They have a very thorough whitepaper and published presentation on why they want us to use CPU affinity. They saw dramatic performance gains (or rather prevented performance drops) when using it. Right now they have a physical server with 4 quad-core processors with hyperthreading so they see 32 CPUs. They are sure that their jobs are CPU bound and it directly correlates to the number of cores. We have a ESX host with 2 eight-core processors in it. I convinced them to allow me to show only 8 cores to the OS for now but it is very likely they'll ask for more. If they run their jobs and they go faster with more cores they'll ask for more. The consultant they brought in confirmed this requirement for CPU affinity as well. Unfortunately as a jack of trades I'm not in a position to intelligently prove they're wrong or right without significant testing.

That's good to know about DRS migrations. We don't plan on using it but it's good to know.

AndySimmons
Hot Shot
Hot Shot
Jump to solution

Interesting! Did they mention vNUMA at all? It wasn't available until virtual hardware v8, so I'm curious if they've included it in their testing. The jist is that it creates a virtual NUMA topology that aligns with the host's physical NUMA topology. The reason I asked about 8 CPUs is that vNUMA gets enabled by default once you cross that threshold. If all of the hosts in your cluster have a consistent NUMA topology, this can be a pretty effective alternative to CPU/memory affinity in many cases, without the DRS/vMotion restrictions.

Anyway, good luck! I can appreciate the jack of all trades position you're in. It's certainly possible they have a valid use case.

-Andy VCAP5-DCA, VCP-DV 4/5, MCSE, space camp graduate.
Reply
0 Kudos
gregstreuber
Contributor
Contributor
Jump to solution

Great discussion guys. You nailed my question and provided a clear answer. Much appreciated.

Greg Streuber
Reply
0 Kudos
NavalgundRaj
Enthusiast
Enthusiast
Jump to solution

Hi Andy,

As for as this CPU affinity question is concern, we can go ahead and ask IBM,where and how they have deployed this application with CPU affinity so that we can

be very clear about this question.

it is late,however is very interesting to know. Please let me know if you any docs/info  on this CPU affinity related ( not vmware pubs/yellow brick links ).

Note: If you found this correct or answer useful please consider the use of the Correct buttons to award points. Regards Basavaraj.R Navalgund
Reply
0 Kudos