topic Re: Question about High CPU latency in ESXi Discussions

Question about High CPU latency

AdamW201110141 — Mon, 31 Jan 2011 21:13:50 GMT

Hey there,

Hoping someone here is able to help me out with a high CPU Ready state I am seeing in one of my clusters.

Our cluster specs:

8 hosts - ESXi 4.0

dual - 6 core Xeon per host

72GB memory per host

DRS and HA turned on

Guests resources assigned:

32 guests (4 per host) with 4 vCPU and 20GB Memory (ave use is only 15-17GB)

The guests are Win 2k3 servers. Most of the traffic is read traffic and application based traffic - and is busy. CPU load on each host is about 40-50% at peak total, but the guests are showing higher than I think we should see for CPU ready time. See the graph below. Wondered how concerned I should be about the CPU ready times?

My first thought is that if the server has 12 cores and we have 4 Guests each asking for 4 vCPUs in scheduling time with semi busy activity, then we'll always see higher than normal CPU ready time. Mainly because there's always one of the guest that has to be waiting for time on the CPUs. So assuming that is correct, if I were to lower the CPU count to 3 on each guest (or even 2 maybe), I should actually see the CPU ready time go to zero without affecting guest performance?? maybe even gain some performance??

Just checking my logic here. Making sure I'm not crazy (which my wife would probably debate).

Thanks in advance!

-Adam

Re: Question about High CPU latency

Troy_Clavell — Mon, 31 Jan 2011 21:18:08 GMT

your logic is good.. Lowering the vCPU count, probably even to 2 will, in my opinion, help reduce CPU %rdy times.

Re: Question about High CPU latency

NuggetGTR — Mon, 31 Jan 2011 21:36:17 GMT

CPU ready time is really relative to what the virtual server is doing, are you seeing performance issues on the application side?

Servers/applicationgs that field small but high volume requests like a web server would suffer quiet a bit from high cpu ready times. Servers/applications that do larger proccessing on small volume generally would not be affected by high cpu ready times.

From the look of the chart you supplied the virtual servers are only hitting 35% cpu usage at their peak, from that you could remove 1 or 2vcpu's and not really impact the performace of the virtual server too much if at all.

The general rule of thumb is to start with 1 or 2(depending on the type of application running) then add if needed.

Re: Question about High CPU latency

AdamW201110141 — Mon, 31 Jan 2011 21:51:55 GMT

The guests are running Citrix so a lot of the traffic is going to be logons, openning applications and normal business use. Nothing too crazy.

I inherited this environment about 6 or 8 weeks ago, so I can't really tell you all the reasons they chose to go with 4 vCPU on all the guests. I was told that's where they saw the best performance benchmarking during the architecture part. Unfortunately, that was long before my employment here so I haven't seen any of that.

In any case, I would also agree that starting with 1 or 2 vCPUs and moving higher based on need is best practice and one I would use if I was architecting.

Really was trying to make sure my logic was sound before I make suggestions on a fix. To date, users really don't have any issues related to performance on an ongoing basis. This is more of something I just noticed as I have been learning the environment and thought it might be a small win or minor change that could be made to make sure the environment is stable. And I don't like seeing all that orange mudding up my perf graphs.

Re: Question about High CPU latency

NuggetGTR — Mon, 31 Jan 2011 22:38:46 GMT

ok so it may very well have been tested previously and 4vcpu was the best performance, this could be true with Citrix the performance with the extra vCPUs may out way the decrease in performance from the small CPU ready time.

but you can always test it yourself

Re: Question about High CPU latency

ChrisDearden — Tue, 01 Feb 2011 09:24:43 GMT

The rule of thumb is that anything over 10% ready *per vCPU* is bad , so with vCPU you are going to get higher figures , so its nothing to worry too much about.

Jason Boche said it much better than I did

http://www.yellow-bricks.com/2010/01/05/esxtop-valuesthresholds/comment-page-1/#comment-5861

are you actually getting an application performance issues ?

Re: Question about High CPU latency

AdamW201110141 — Tue, 01 Feb 2011 15:17:45 GMT

Thanks for the link. Very Helpful. We get alerts for slowness in the Citrix environment for various guests, but nothing that is consistent. Mostly, there are other factors that contribute to the majority of the problems. I'm really more trying to make sure there are no issues with the environment, so when we do have issues, it's a lot easier to say there's no issue with the ESX farm. CYA type of thing.