ESXi 5: Sharing 4 cores with 3 VMs - vCPU settings

[forexverified]

Hey folks,

First, I shamefully recognize that I'm relatively new to ESXi and while I could probably find the answer in the plethora of documentation, I've spent hours and come up with nothing.

Looking for a simple answer to a very simple question:

What's the best way to share 4 CPU cores (4 physical cores on the host node) equally with 3 virtual machines?

I know there's lots of variables that can be introduced to the equation (type of software being used, VM operating system, etc) but I just want to give each VM as equal of a share as possible.

Right now I have 2 VMs, each with 2 vCPUs, and I really like my performance but I'm severely under-utilizing our resources.

I want to add a third VM and split up the 4 cores among the 3 VMs.

My gut says to give 4 vCPUs to each VM and let ESXi balance the load.

Am I right?

-Scott

P.S.  It's a Xeon E3 1230 with hyperthreading disabled, so it shows up with 4 logical cores, not 8.

[mcowger · Accepted solution]

1) Please turn HT back on.  Modern HT on Ivy Bridge helps 99% of application performance...especially with ESXi.

2) If you just give all VMs equal resources (say, 4 vCPUs) and dont change anything else, they will all be treated equally.

[forexverified]

Thanks mcowger!

I've also spent hours googling whether or not to disable hyperthreading and I think I've read every possible opinion out there.      My situation is a little unique, we run 8+ year old software called Metatrader 4 (Forex trading platform) and I'm setting up the VM's with Windows XP.   MT4 is very CPU sensitive - lack of resources can cause crashing or order execution delays.  It's also single-threaded.

It seemed like disabling hyperthreading was best because I could absolutely guarantee each VM half the CPU resources without having to mess with the affinity settings.

But, during the coming week, I will test our usage both with it enabled and disabled to see if I see any difference.  I appreciate the suggestion.

Scott

[Dave_Mishchenko]

It's best not to add vCPUs to VMs that are performing at an acceptable level.  If all 3 VMs have 4 vCPUs each then it is a much harder task for ESXi to schedule CPU resource that if you have  3 VMs with 2 vCPUs each. Generally just give a VM the CPU resources it needs and nothing more.

With a single threaded application you're not going to benefit with a 4 vCPU VM.

[forexverified]

I guess that's my question - if I have 3 VM's each with 2 vCPU's (total of 6 vCPUs) running on a node with 4 cores, will ESXi share the load evenly or will the resources be lopsided?

Scott

[Gortee]

Intresting issue.   Personally I would do the following:

Assuming you have 4 cores and three vm's and your running a single threaded application that is cpu sensitive on Windows XP:

- VM1

No Change

-VM2

No Change

-VM3 - Windows XP

Give it a CPU reservation for max CPU speed

Give is max RAM your OS will support but not as a reservation

Now this eats a lot of resources but it promises your vm's the cpu 100% of the time. 

The other option would be direct path I/O but I would skip it.

I wrote an additional article on it at http://bloodygranola.com/?p=173

Let me know if you have any questions.  I think you will find that your system can handle three vm's just fine.   I have a cluster with 700 virtual machines including tier one applications like exchange running on 132 physical cores.  Vmware just keeps getting better and better with the CPU scheduler.

[Dave_Mishchenko]

The CPU scheduler will try to balance the CPU resources equally between the 3 VMs.  You can tweak performance with reservations, limits and shares.  A reservation guarantees a VM certain resources, but it also means that the resources couldn't be used for other VMs even if the one VM wasn't using them.  Limits will limit host resources even if the VM needs more.  ESXi uses shares to determine how to balance resources when the resource is running short.  If you gave one VM 4000 CPU share and the other two have 2000 shares each, the if the VMs were all running at 100% (within the VM) the one VM with 4000 shares would get 50% of the hosts CPU resources while the other two would split the remaining 50%.

[forexverified]

Ok just to clarify:

We're running a single-threaded application, but we can run 10-20 of them on a VPS.  So they will use multiple cores in that regard.

Our current CPU usage according to the ESXi host (looking at it via vSphere Client) peaks at about 45%.  That's why I'd like to add 1 more VM, because I think we can comfortably run at 45-50% usage with peaks up to 60-70% without it interferring with the performance of our software.

Next week I'll just have to do some experimenting with different vCPU numbers and see if it makes a difference.  I'm not ready to add limits/reservations because I'd like each VM to use whatever resources it needs, I just also wanted to make sure I wasn't unintentionally short-changing any VM's due to the number of vCPUs assigned.

I'll start out with all 3 VMs using 2 vCPU each and see how it goes, and I'll plan on updating this thread mid-to-late next week.

Thanks!

Scott

[BrownUK]

If you really only have 4 Cores do not think about using machines with more than 2 cores.  At the end of the Day Esxi has to run on something (the Hypervisor OS) so if you are using four cores there is nothing left.

If you have hyperthreading, make sure it is on.

Don't use Shares unless there is ocntention, if there is contention get another host

Create a machine with two CPU's, useone CPU if performance is OK leave it, if not increase to two.

Add more machines until you are at 75% overall CPU.

Give them as much memory as you can. leaving 1GB free.

[forexverified]

As promised here's my update.  I did not get any time to experiment with various vCPU settings, but I can confirm that 3 VM's each with 2 vCPU's seem to be sharing the Xeon E3-1230 CPU (4C/8T) just fine.  As far as I can tell the resources are not lopsided, though it's hard to tell.   My average CPU usage is only 35% with usage spikes up to 45-50% during busy market hours, so I do have plenty of room to add another VM before those 'spikes' reach into the edge of my comfort zone.

Once I add a 4th VM I'll have a total of 8 vCPU's running on 4 phyiscal cores with 8 logical processors (the hyperthreading) so I assume that ESXi will have no problem distributing the resources evenly anyway.

I thank everyone for their assistance!

Scott