VMware Cloud Community
jitendrakmr
Hot Shot
Hot Shot

Max Load on one ESX host?

Hello Everyone,

How much load can I put on single ESX host in terms of CPU and Memory? Can I increase load on ESX host to utlilize upto 95% of its CPU and memory or can it be more (upto 100%)? What are VMware's recommendations are here?

Thanks.

Jitendra Kumar

MCSE 2003, VCP, CCNA, ITIL Foundation, Netapp NS0-153

Personal Website -

VCP, MCSE 2003, MCITP Enterprise Admin, CCNA, ITIL Foundation, Netapp NS0-153 (Storage Networking) Personal Website - http://www.virtualizationtrainings.com, http://www.hillsindia.com
Tags (2)
0 Kudos
18 Replies
sbeaver
Leadership
Leadership

That really depends on if you are running hosts in a cluster and if you have HA and such. I have seen an ESX server's CPU spike up to 102% before. You need to leave headroom to handle spikes in over all performance

Steve Beaver

VMware Communities User Moderator

====

Co-Author of "VMware ESX Essentials in the Virtual Data Center"

(ISBN:1420070274) from Auerbach

*Virtualization is a journey, not a project.*

Steve Beaver
VMware Communities User Moderator
VMware vExpert 2009 - 2020
VMware NSX vExpert - 2019 - 2020
====
Co-Author of "VMware ESX Essentials in the Virtual Data Center"
(ISBN:1420070274) from Auerbach
Come check out my blog: [www.virtualizationpractice.com/blog|http://www.virtualizationpractice.com/blog/]
Come follow me on twitter http://www.twitter.com/sbeaver

**The Cloud is a journey, not a project.**
0 Kudos
williambishop
Expert
Expert

I think it is reasonable to assume you can safely use 85% of the box's resources, allowing 15% for spikes and extra demand from the guests....But you should always allow for the cluster's needs before the hosts....If you need to run lower (say 50%) to satisfy failover, then that's what you run, no matter what the host itself is capable of.

--"Non Temetis Messor."
0 Kudos
williambishop
Expert
Expert

I guess we just danced all over each other's toes Steve......

--"Non Temetis Messor."
0 Kudos
sbeaver
Leadership
Leadership

It's all good to hear from different people after all it is all about helping Smiley Happy

Steve Beaver
VMware Communities User Moderator
VMware vExpert 2009 - 2020
VMware NSX vExpert - 2019 - 2020
====
Co-Author of "VMware ESX Essentials in the Virtual Data Center"
(ISBN:1420070274) from Auerbach
Come check out my blog: [www.virtualizationpractice.com/blog|http://www.virtualizationpractice.com/blog/]
Come follow me on twitter http://www.twitter.com/sbeaver

**The Cloud is a journey, not a project.**
0 Kudos
williambishop
Expert
Expert

You know it baby, hows sunny florida today?

--"Non Temetis Messor."
0 Kudos
sbeaver
Leadership
Leadership

I heard it was warm!! I am actually in Portland freezing with Florida but off this week

Steve Beaver
VMware Communities User Moderator
VMware vExpert 2009 - 2020
VMware NSX vExpert - 2019 - 2020
====
Co-Author of "VMware ESX Essentials in the Virtual Data Center"
(ISBN:1420070274) from Auerbach
Come check out my blog: [www.virtualizationpractice.com/blog|http://www.virtualizationpractice.com/blog/]
Come follow me on twitter http://www.twitter.com/sbeaver

**The Cloud is a journey, not a project.**
0 Kudos
williambishop
Expert
Expert

What are you doing in oregon? Shouldn't you be somewhere hot and muggy vs. cold and drizzly?

--"Non Temetis Messor."
0 Kudos
jitendrakmr
Hot Shot
Hot Shot

Thanks for replies. I understand that 85% utilization after cluster failover capacity is being satisfied is a good choice. Can just fulfilling the failover capacity requirement of cluster is enough? In that case if hosts equal to the number of failover capacity of HA goes down then all the remaining hosts will run at 100% utlization....

What does Vmware suggests here? any ideas?

Also How can the utilization be more than 100%?

Thanks.

Jitendra Kumar

MCSE 2003, VCP, CCNA, ITIL Foundation, Netapp NS0-153

Personal Website - http://www.virtualizationtrainings.com

VCP, MCSE 2003, MCITP Enterprise Admin, CCNA, ITIL Foundation, Netapp NS0-153 (Storage Networking) Personal Website - http://www.virtualizationtrainings.com, http://www.hillsindia.com
0 Kudos
williambishop
Expert
Expert

You don't want to run 100% after failover, because if you do, then you are going to slow down your running guests. It's a common design to configure your fail-over to meet 85-90% on all hosts AFTER failover has occurred on a host(s), depending on the failover capacity you designed for. N+1 being the most common, meaning you can lose one server in the cluster, and the rest of the cluster will not hit that last 5-10% which will affect your environment. If it means only running 50% (say a two host cluster), then THAT'S what you do. Planning for this ahead of time, means a healthy cluster.

--"Non Temetis Messor."
0 Kudos
sbeaver
Leadership
Leadership

Back on Friday so just a quick visit. More news to come soon

Steve Beaver
VMware Communities User Moderator
VMware vExpert 2009 - 2020
VMware NSX vExpert - 2019 - 2020
====
Co-Author of "VMware ESX Essentials in the Virtual Data Center"
(ISBN:1420070274) from Auerbach
Come check out my blog: [www.virtualizationpractice.com/blog|http://www.virtualizationpractice.com/blog/]
Come follow me on twitter http://www.twitter.com/sbeaver

**The Cloud is a journey, not a project.**
0 Kudos
sbeaver
Leadership
Leadership

I had those same question but when I saw the spike from backups get up that high. To answer the question about what vmware suggests I think you can use the warnings that are given when there are not enough resources to satisfy HA as a good indication. You can turn that warning off and ESX will continue to start VM's but the amount of time it takes to fully recover can make you think twice about it.

Steve Beaver
VMware Communities User Moderator
VMware vExpert 2009 - 2020
VMware NSX vExpert - 2019 - 2020
====
Co-Author of "VMware ESX Essentials in the Virtual Data Center"
(ISBN:1420070274) from Auerbach
Come check out my blog: [www.virtualizationpractice.com/blog|http://www.virtualizationpractice.com/blog/]
Come follow me on twitter http://www.twitter.com/sbeaver

**The Cloud is a journey, not a project.**
0 Kudos
Ken_Cline
Champion
Champion

What are you doing in oregon? Shouldn't you be somewhere hot and muggy vs. cold and drizzly?

Unless I miss my guess, Mr. Beaver is getting indoctrinated...but he shouldn't worry too much. I'm playing surrogate this week (I'm in Ft. Walton Beach, FL) and the weather's nothing to brag about (it's warm enough, but it's raining...)

Ken Cline

Technical Director, Virtualization

Wells Landers

TVAR Solutions, A Wells Landers Group Company

VMware Communities User Moderator

Ken Cline VMware vExpert 2009 VMware Communities User Moderator Blogging at: http://KensVirtualReality.wordpress.com/
0 Kudos
sbeaver
Leadership
Leadership

Hey Ken,

When do you fly back? Are you going to get anywhere near Orlando? We have not had the chance to catch up in while so if you get some time give me a shout.

Steve Beaver
VMware Communities User Moderator
VMware vExpert 2009 - 2020
VMware NSX vExpert - 2019 - 2020
====
Co-Author of "VMware ESX Essentials in the Virtual Data Center"
(ISBN:1420070274) from Auerbach
Come check out my blog: [www.virtualizationpractice.com/blog|http://www.virtualizationpractice.com/blog/]
Come follow me on twitter http://www.twitter.com/sbeaver

**The Cloud is a journey, not a project.**
0 Kudos
Ken_Cline
Champion
Champion

All the other folks who have responded are wrong Smiley Wink

The "right" answer (as always) is "it depends". If you're familiar with a lot of mainframe environments, you've seen systems running at 97% on a pretty consistent basis. That's because the workloads are predictable and the management tools are very mature. We can't say the same thing for the typical ESX environment. The typical ESX workload is very spikey/variable and very environment specific. Some people can run their hosts at 85% CPU and 90% RAM with impunity; others may have to be much more conservative. Things to consider include: do you have cyclical workloads that have to be accounted for (i.e. monthly / quarterly / etc. tasks)? are you running in an HA enabled cluster? do you need the ability to surge quickly (like, for example, when Apple released the iPhone)? what is your lead time for provisioning additional capacity (if you're running at 80% and it takes you six weeks to stand up new capacity, is that OK?)? and others.

Also, is your site the failover site for a DR peer site? If so, then you need to make sure you have enough capacity to handle that contingency. So...as you see, there is no one right answer. You've got to look at your own particular environment and determine how much reserve capacity you need to meet your business requirements. Then you have to present those needs to your management team and let them weigh the risks against the costs and figure out how much reserve they're willing to fund, then you provision. Then, every six months or so, you review your infrastructure to see how well you're meeting your established guidelines and repeat the whole process again...

Ken Cline

Technical Director, Virtualization

Wells Landers

TVAR Solutions, A Wells Landers Group Company

VMware Communities User Moderator

Ken Cline VMware vExpert 2009 VMware Communities User Moderator Blogging at: http://KensVirtualReality.wordpress.com/
0 Kudos
mreferre
Champion
Champion

I was about to post something .... but Ken did beat me on time ... Smiley Happy

My view is that, as Ken suggested, this is not a mainframe. Mainframes regularly runs at 95/100% utilization (most of the time northbound) simply because what you get out of a mainframe is VERY predictable. Not even at which VM uses which resources .... but which VM needs to respond in this xyz amount of time. the x86s platform is light years far from this level of service (not implying it will never get there tough). On the mainframe you know that, if you touch parameter xyz your application is going to respond zyx% slower or faster (depending on the parameter).

I usually say that on the x86 platform you touch a bunch of parameters..... and you keep your fingers crossed. Well I guess I am exaggerating a bit but I think you know what I mean.

There is a reason by the way for which this is the case. The mainframe is more expensive than any x86 platform available so there are obviously motivations to run it close to 100% utilization. If you think for example at the x86 platform a "simple" upgrade from dual core CPUs to quad core CPUs would double your resources capacity basically at the same costs (both for the hardware and for the software). Yes we could ask VMware to invest into ESX at the point where you can drive workloads at 99% utilization and keep your %READY close to 0 (Vs raising to the roof)........... but then you couldn't complain if VMware is going to charge you 20K$ per socket for the hypervisor..... Smiley Wink

Perhaps I digressed. Back to the question..... I would say 60/70% utilization is pretty much a number I would like to see out of my ESX host.

My 0.02 cents.

Massimo.

Massimo Re Ferre' VMware vCloud Architect twitter.com/mreferre www.it20.info
0 Kudos
PaulCooper
Enthusiast
Enthusiast

Thanks Ken and Massimo - you've saved me lots of typing Smiley Happy

What I would suggest Jitendar is that, assuming you have no information on how 'spiky' your applications are, and reservng capacity for DR isn't a requirement, is to start at a low number - say 60% - and if after a month or two you find that the host has plenty of capacity that isn't being used by spikes (and backups/virus scans of the VMs can really spike their usage) then I would start to increase it. If that doesn't suit your virtualisation style then I would agree with Massimo and be conservative, as long as you are still achieving your company's goals for virtualisation especially the cost model (cost reduction compared to physical servers etc).

0 Kudos
Ken_Cline
Champion
Champion

I'm flying Friday afternoon. My entire time will be here in FWB, so not so close to Orlando Smiley Sad

Heck, in your new role, I expect you'll be up in my neck of the woods on a not infrequent basis!

Ken Cline

Technical Director, Virtualization

Wells Landers

TVAR Solutions, A Wells Landers Group Company

VMware Communities User Moderator

Ken Cline VMware vExpert 2009 VMware Communities User Moderator Blogging at: http://KensVirtualReality.wordpress.com/
0 Kudos
mreferre
Champion
Champion

I agree.

I have a very naives approach to sizing I have to admit: http://it20.info/blogs/main/archive/2007/11/26/83.aspx

In my opinion (I know many disagrees on this) sometimes we put so much effort into sizing / planning for performance when with 2% of the entire work you could get an 80% accurate result. Sure you can get to a 100% accurate result but you have to invest into the remaining 98% of the work. Sometimes this is not cost effective especially if you think that with what you save (in men-hours / tools etc etc) you could triple your virtual infrastructure (from a HW and SW perspective). Obviously this is not the case for mainframe like platforms where it is worth investing in men-hours / tools etc to size properly because the hw and the sw tend to be more expensive so you want to size it "just exactly right".

Typically VMware customers do not use actual resource utilization as a metric of success (I might be wrong ... or I might have been very lucky). I have never met a customer complaining that his/her farm is running 60% utilization Vs 95% ......

These are my own opinions and not those of my employer.

Massimo.

Massimo Re Ferre' VMware vCloud Architect twitter.com/mreferre www.it20.info
0 Kudos