Setting CPU affinity for Vmware itself

[virtualesxer]

Quoting Ken Cline (from a now obsolete 3.0.1 thread):

Best practice is to NOT install these types of applications in the service console. You would be much better off creating a VM and running your application there. The service console is, by design, resource constrained.

In ESX 3.5, I found no way to pin the Service Console to a physical core. I checked under "system resource allocation", in the VI client, but it just talks about shares and such.

Basically, I have one very heavy VM, that needs to run on as much vCPUs as possible. I also have 1 FreeBSD machine, that needs 1 full core. I could pin that to cpu1. Then, my Windows 2008 machine, on which I plan to do some heavy rendering, I plan to pin to cpu2 and cpu3, plus cpu0. So, I'd like to limit the Service Console to cpu0, and allow it only 20% or so, while giving the other 80% to the Windows machine.

Thing of it is, I can't find anything on how to set CPU affinity for Vmware itself. Nor how I can split up a core (percent-wise) between two VMs. Perhaps someone could enligten me?

Thanks.

[jyvesd]

Well FYI the esx service console get always schedule on cpu0...

[virtualesxer]

Thanks. No need to set it, then.

And the Vmkernel/scheduler, also on cpu0?

[jyvesd · Accepted solution]

There it is different. By default the VMkernel dynamically load balance the thread accross the cpu/core. It means the vmkernel will try to avoid scheduling cpu intensive on threads in the same core. Now it you are planning to use cpu affinity you will loose the benefit off the CPU load balancing but also things like DRS-VMotion.

[virtualesxer]

Thanks again.

Ok, is it also possible then to set a NEGATIVE affinity? Meaning something like: "Use anything, but cpu0." That would still allow for some load-balancing, but exempt certain cores from being clobbered.

[Ken_Cline]

Another best practice is to NOT use CPU affinity. If you want to assign a specific unit of capacity to a VM, it is much better done using Reservations than via affinity.

If you have a VM that you feel needs to have 100% of a core, then I'd recommend setting a CPU reservation of (100% * Mhz of one core). Actually, I'd probably start out at a 50% reservation and see how much is actually used, then adjust from there.

When you assign CPU affiinity, you are taking the scheduling control out of the hands of the vmkernel (isn't scheduling what you bought it for???) and chances are good (very good) that the vmkernel is better at scheduling than you are.

Also, if you pin a VM to a CPU, you remove it from eligibility for VMotion.

Ken Cline

Technical Director, Virtualization

Wells Landers[/url]

VMware Communities User Moderator

[virtualesxer]

Another best practice is to NOT use CPU affinity. If you want to assign a specific unit of capacity to a VM, it is much better done using Reservations than via affinity.

Thanks. Yes, I just figured that out. At first the Reservation seemed a bit confusing, as it said I could reserve a max of 10,000Mhz for a VM with 1 vCPU. But when I actually tried that (5000Mhz), I got a message saying that limit was out of bounds (which is good, because then things made sense again). So, I reserved 2666Mhz for the 1 VM (= 1 core, to the max, on my Q9450). Meaning (if I understood it correctly now; and I think I do), "This VM can use all of its 100% vCPU when it wants to, but should it need less, others can use its unused resources." Which is exactly what I want: I want my FreeBSD server to run unhindered, but still have the heavy rendering VM take whatever it can when it needs it, and my FreeBSD VM doesn't need it first.

I saw the SC uses Reservation too, which neatly takes care of that as well. I think I'm finally getting a handle on this.

Thanks, again.

[Ken_Cline]

A couple good resources to help solidify things for you:

- ESX Server CPU Scheduling

- Service Level Management with Resource Pools, Reservations, and Limits

You may find that you get better overall resource utilization by setting your Reservation to a lower number (perhaps 50% or 1333Mhz) and assigning the VM a higher allotment of Shares. It really all depends on how heavily loaded the system is. Your current configuration using Reservations is MUCH better than your original idea of CPU affinity. Also, make sure you do not starve the service console. It is a critical component of overall system management and when it needs to do something, it should be able to do it NOW.

Ken Cline

Technical Director, Virtualization

Wells Landers[/url]

VMware Communities User Moderator

[virtualesxer]

And thank you, once again!

Also, make sure you do not starve the service console. It is a critical component of overall system management and when it needs to do something, it should be able to do it NOW.

I haven't changed the console reservation. It's currently set at:

Reservation: 267Mhz

Limit: 2671Mhz

Shares: 1000

Hmm, 267Mhz seems a bit low, but that's how the system configured itself. Is there a reason to up those settings? Other than the HP Managment System Agents, not much is happening at the console.

[virtualesxer]

Got the Reservation thing down now, but I'm still not entirely clear on shares.

It says shares are relative numbers. So, VM 1, having 4000 shares, runs with twice the shares-priority as VM 2, with only 2000 shares. I got that much. But what if I increase VM 1's shares to 8000? Does that mean that, effectively, the 2000 shares of VM 2 have become worth even less? I.c., that VM 2 now only get 1/4th of the available resources?

Also, I have a number of VMs created that are not running (yet), but they still seem to occupy shares, in terms of percentage (in Resource Allocation, in the VI). I hope that's just a number for when they're running, and that they're not actually already occupying share-reservation. Right?

Thanks.

[Ken_Cline]

Got the Reservation thing down now, but I'm still not entirely clear on shares.

It says shares are relative numbers. So, VM 1, having 4000 shares, runs with twice the shares-priority as VM 2, with only 2000 shares. I got that much. But what if I increase VM 1's shares to 8000? Does that mean that, effectively, the 2000 shares of VM 2 have become worth even less? I.c., that VM 2 now only get 1/4th of the available resources?

In your example, (8000 shares and 2000 shares for a total of 10000 shares), VM2 would get only 1/5th of the available resources that are subject to contention.

The shares are a "percentage of the whole" and as you increase the pool of shares (by increasing the shares on a given VM or resource pool or by adding VMs), the value of a single share is diminished. Also, shares only come into play when there is actually contention for a resource...resources that are reserved do not enter the shares lottery.

Also, I have a number of VMs created that are not running (yet), but they still seem to occupy shares, in terms of percentage (in Resource Allocation, in the VI). I hope that's just a number for when they're running, and that they're not actually already occupying share-reservation. Right?

Yep...the shares come into play only when the VM is actually powered on. It doesn't matter whether the guest OS is doing anything...shares are a VM thing. Here is an ancient post I did that helps understand the way shares work. It's not 100% accurate, but it's close enough to give you a good idea of what's happening. http://communities.vmware.com/message/170480#170480

KLC

Ken Cline

Technical Director, Virtualization

Wells Landers[/url]

VMware Communities User Moderator