Extreme WAIT times and slow vm performance

[powadha]

In a 3 host R710 ESXi 4.1 (3x16 logical processors) DRS cluster running 18 vm's, performance on some vm's is really bad.

Looking at ready times nothing strange is showing but WAIT times seems extremely high (some vm's maxing out at 20000 Millisecond for longer periods of time).

I see that most vm's have 2 vCPU and about 6 have even 4 vCPU. I guess the problem might be with the vCPU's but not sure (usage and ready times are really low). What could explain those high WAIT times?

Regards

[iw123]

Hi,

Do you have 16 processors across the cluster? How many vCpus are assigned to VMs in the cluster ?

[powadha]

Cluster:

Info states: 8 CPU x 2,526, GHz, E5540, 2x processor sockets, Cores per socket 4, Logical Processors 16 , Hyperthreading: active

Two machines have an E5640 proc rest of info is the same...

[iw123]

How many vCPUs in use in the cluster ?

[powadha]

48 vCPU in this cluster...

[MKguy · Accepted solution]

The %WAIT metric is irrelevant because it includes idle time:

http://www.yellow-bricks.com/2012/07/17/why-is-wait-so-high/

%WAIT
            The total percentage of time the Resource Pool/World spent in wait state.  I.e., the world is waiting for some VMKernel resource.  Note that this percentage includes the percentage of time the Resource Pool/World was idle.

If you really suspect a CPU bottleneck, %RDY and %CSTP are the main metrics to watch out for.

Looking at ready times nothing strange is showing

So how big are they really? Remember you have to put the vCenter chart summation ready values in proportion of the data collection interval (20 seconds in realtime charts).

But from I doubt you really have CPU issues with "only" 48 vCPUs in a cluster of 3 hosts with 24 cores/48 threads total, unless most if your VMs run at nearly 100% CPU utilization.

[powadha]

Thanks, I'll have a better look at the READY times

[powadha]

I vmotioned some machines by hand to a host which had the least and lightest equiped machines in the cluster. READY times dropped (turned out to be high after all). Seems that DRS doesn't take READY times in account, just USAGE?

Anyways, the client wants machines with 2-4 cpu's even if performance shows the machines don't need it. I guess adding hardware is the only way to solve this for longer periods of time...

I'll install VMturbo and see if those charts can convince them of rethinking the amount of processors per VM.

Regards