CPU Core Sharing and Affinity

[sel57]

I have an internal customer that would like dedicated CPU for their vm's. We use hyperthreading on all our hosts, but I understand you can set core sharing at the vm level. Could someone explain the difference between core sharing and affinity?

From a resource reservation standpoint, I guess I don't understand the difference between exclusive access to the core (sharing) and assigning a specific core to a specific vm (affinity). It sounds to me like in both cases, the vm gets preferential access to the physical core.

[vLarus]

First of all I would question the reason the customer has for this request. Is it because he has an application that would suffer from getting transferred and losing the CPU cache it had built up?

If not then any of these settings will not help.

If this is an application that might have better performance due to CPU cache locality then you can select either one of these options but do note that each has its pros and cons:

Core Sharing:

A hyperthreading specific setting. It includes Any which is the default and states the vCPU will share the core with other vCPUs. None; which will state the vCPUs can not share the core with any other vCPU, and internal; which states it can share the core only with vCPU on the same VM.

Selecting None or Internal might help if you have an application that doesn't respond well to other VMs accessing the cache or even the core itself (timing). But this does not guarantee that it will be descheduled based on shares and resource entitlements. This will result in the vCPU getting scheduled on a new core thus accessing a new L1 and L2 caches (or even L3 for NUMA migrations).

This will not result in optimal use of CPU caches but will help reduce the noise another vCPU from another VM can cause.

This option also still allows for VM migration using DRS.

CPU Affinity:

Not a hyperthreading specific setting. You basically "tie" down a vCPU to a specific physical core on the ESXi host. This will result in better usage of CPU caches. This implies a core is a logical processor, not a physical processor when using hyperthreaded systems.

I really can't explain this any better here than Frank Denneman does here with a rightly named blog post: http://frankdenneman.nl/2011/01/11/beating-a-dead-horse-using-cpu-affinity/

Setting either option or both of them together will result in huge management overhead with potential negative results on performance both for the VM in question (if not configured correctly) and other VMs running on the same system.

And neither will result in preferential access to the core since its the CPU scheduler that controls that and he only listens to shares, reservations and entitlements when scheduling time on CPU.

[sel57]

Thanks for that write up vLarus. Very helpful.

Yes, I ALWAYS question why someone wants dedicated resources, and to date, I have never had to provide it. We've never had hosts with resource issues, so there has never been a reason to grant these guaranteed resources to vm's. However, this customer wants to buy their own host to put it in our cluster, which we've agreed to as long as they realize that if their host experiences down time, we will only bring back up their environment at a reduced capacity on other hosts in the interim. They were fine with that. (FYI, No DRS, NO HA in this cluster)

So, one would think that I don't need to use either of those things, core sharing or affinity, if the users have their own host, but I was hoping to utilize this host somewhat too, since I know they won't use all of the resources despite what they think. (current environment metrics tell the truth) That's why I was trying to understand these options.

What do you think would be the best setting to enable to ensure their vm's always get host resources they need, even if it comes at the cost of other vm's suffering? It sounds like no matter what solution I choose, I could introduce temporary problems on to other hosts in my cluster during any downtime for their host, but I just want to ensure their vm's always get the resources they need on their host, and any other vm I decide to put there do not if there was ever contention.

[vLarus]

VM reservations will fulfill all these requirements since it will put the VM in front of the line during CPU scheduling and it calculate the memory access differently thus giving priority access to the memory.

It will not impact the other hosts or VMs unless you have HA Admission Control configured for a number of hosts. And since you don't have HA nevermind that

Since you don't have DRS the VMs will stay there unless moved.

Neither of these options (CPU Sharing or Affinity) will give the VM preferred access to resources and should not be used as such. It just for there corner cases where you need extreme tweaking capabilities.

Its just more controlling the location and the noise of VMs could have on the specific vCPUs for those "special" VMs.