Add a host or upgrade CPUs?

[Groundbeef79]

We have a 3 host ESXi 4.1 cluster.  Dell PowerEdge R710s.  Each has two CPUs and 64 GB of RAM.  One has two E5530s and the other two have two E5540 CPUs.  These are all 4 core with HT.  We're at the point where we can't add another guest without breaking HA.  I've got things divided pretty well between hosts as far as CPU and RAM goes.  We're only using about 25 GB of RAM on each host.  We have multiple vendors that require their VMs to be 2 core, so that's probably why I'm in this boat, but I just want to be sure.  I took the class on Vsphere installation and implementation almost 2 years ago and I don't touch this stuff every day, so bear with me.  So when I look at the summary, here's what I have:

General:

Total CPU Resources:  59GHz

Total Processors:  24

Virtual Machines and Templates:  26

Total Migrations using vMotion:  265

Vmware HA:

Admission control:  enabled

Current Failover Capacity:  1 host

Configured Failover Capacity:  1 host

Host Monitoring:  Enabled

VM Monitoring:   Disabled

Application Monitoring:  Disabled

HA Advanced Runtime Info:

Slot size:  750 Mhz, 2 virtual CPUs, 4333 MB

Total slots in cluster:  39

Used slots:  25

Available slots:  1

Total powered on vms in cluster:  25

Total hosts in cluster:  3

Total good hosts in cluster:  3

There are 10 guests with 2 vCPUs and 15 with 1 vCPU.  That's a total of (10 * 2) + 15 = 35 vCPUs.  Am I right in thinking that we're CPU limited here?  I'm fairly certain that this is where the hangup is.  Our hardware is about 2 years old and we have two options.  Option 1, add a server to cluster and license vmWare, Windows, etc again at a high cost ($25k roughly) OR we can upgrade the CPUs in these hosts.  Going from E55xx's to E5645 Westmere CPUs.  These newer CPUs have 6 cores vs the 4 we have now.  So with 6 CPUs, we'd be adding 12 additional cores.  This seems to make sense IF we are CPU limited.  The cost is A LOT lower than adding another host.

Oh great experts, what are we to do?    If we go the CPU replacement route, I can just power down the hosts, install the CPUs and be done?  No further config?

[mcowger · Accepted solution]

So you have about 13 slots available per host, you are using 25 slots currently (which is good, it means that all 25 of your VMs fit in 1 slot ), have 1 left over (effectivly unuseable) and 13 'in reserve'.

You are almost certainly limited by # of cores right now, but almost by memory as well.

If you upgrade the CPUs, your number of slots per host should go up a little (counting by memory, you have about 15 slots for memory), but it wont go up a huge amount (maybe by 3x2=6 slots).

I'd recommend the additional host, to be honest.

[JohnADCO]

I like the CPU upgrade idea less the fact it really does look like your going to be running into ram issues just around the corner and you will be right back to adding a host anyways. 

[Groundbeef79]

The CPU upgrade to us is attractive.  The reason is we don’t anticipate adding a whole lot of VMs in the next couple of years.  We’ll most likely replace the entire cluster in a year and a half getting higher end hardware at the same time.  So it’s hard to fit that 4th host in there.  The main thing is verifying that the CPU is the limiting factor here.  As long as we don’t go crazy on the RAM (which I don’t see us doing anyway) we should be ok for the year and a half I’d think right?

It’s easier to pay $6k vs. $25k now you know?

[mcowger]

Well, like I said you will end up with ~6 extra slots, which means around 5-6 more VMs.  Do you really think you will be asked to spinup fewer than 6 VMs over the next 1.5 years? 

The CPU is the limiting factor right now, but the memory is VERY VERY close behind it.

[Dave_Mishchenko]

What sort of numbers do you have on CPU and memory usage in the cluster?  If the slot size is too conservative perhaps you can use the  "percentage of cluster resources" settings for admission control.

[Groundbeef79]

There is no memory ballooning happening at all.  Like I said before, we're using about 25 out of 64 gigs of RAM per host.  Our CPU will spike up to 41 Ghz on one host during some nightly processing, but generally don't go above 30 and generally hover quit a bit lower than that.  Does that follow what you're asking?  Like I said before, I don't touch this stuff every day, so I might be missinig something obvious.

[MKguy]

I too would opt for the percentage instead of number of host failures HA option, unless you have a few huge VMs.

Consult the HA Deedive by Duncan on this matter: http://www.yellow-bricks.com/vmware-high-availability-deepdiv/

Your vCPU count might seem a lot to you, but I'm almost certain you aren't CPU-bound yet. As long as your overall CPU utilization and %RDY times are OK, you should be able to add even more vCPUs without problems. You did enable Hyper-Threading on your 55xx CPUs too, right?

In short, you might imagine things and are worrying too much, so unless the above %CPU and %RDY limits apply to you, have faith in the ESX CPU scheduler and let it do the magic.

Btw., the official support limit is 25 vCPUs per physical core, see http://www.vmware.com/pdf/vsphere4/r41/vsp_41_config_max.pdf.

[mcowger]

Its about slot size.

You may not be USING that memory, but HA still regards 4333 MB as your memory slot size.  So, from HA's perspective, every VM using 1 slot will use 2vCPUs or 4333MB, which ever comes first.

I'm just saying be careful, and that you should pick up Duncan Epping/ Frank Denneman's book on how DRS/HA work: http://www.amazon.com/VMware-vSphere-4-1-Technical-deepdive/dp/1456301446/ref=sr_1_1?ie=UTF8&s=books...

[Groundbeef79]

We were very agressive in P2V-ing everything we had, but have not added any "new" servers in the last year.  We haven't added all that many servers in the 6 years I've been here, so I really think in the next year and a half we might go 2-4 for some other projects in the works, so we'll be cutting it close.  Problem is the hardware will be scheduled to be replaced then anyway.  So it's a tough one.  If upgrading the CPUs will work, that's the route we'll take. RAM upgrades are also cheaper than relicensing Windows, VMware, SQL, etc. too.  I just wanted to make sure I wasn't being a total bonehead on this one.  I appreciate everyone's input and I'll spread the points around.  I'll take a look at the links provided as well.