CPU LATENCY Counter ON ESXI 5.0.

[Blicks]

I am having trouble understanding
the results from the latency counter on the vcentre for our guest VM.

My understanding of the latency counter
is that it will increase when there is contention for physical CPU resources
and that it is a measurement of percentage of time that a guest has to wait to
get access to the CPU resource.  However,
all our hosts have more CPU physical cores than the total number of VCPU’s of
our guests.  Therefore, if my
understanding is correct, there should be minimal latency and there should be
no contention for physical CPU resources.
We are seeing a number of guest VM wares with a latency of above 10%
which I have read shows significant degradation of performance.

However, our CPU Ready counter
are well below 500 milliseconds.  Can
anyone shed any light as to why this might be the case?

[Iwan_Rahabok]

Blicks is the issue resolved? I understand it's been almost a year. I'm keen to know it too. I hope yours get addressed by following some of the suggestion in this thread.

Latency includes co-stop. What values do you get for co-stop?

I think Latency at 10% - 20% could still be ok, if both ready and co-stop are low. Do you actually get significant degradation?

I hope this KB helps: Poor virtual machine application performance may be caused by processor power management settings

Cheers!

e1

[Iwan_Rahabok]

TheLight

My understanding is it includes co-stop also and HT busy time.

CPU Latency, i.e. %LAT_C in esxtop, includes: ready, cstp, HT busy time and effects of dynamic voltage frequency scaling,

It doesn't include mlmtd though.

Note that this is the same as CPU Contention in e.g. vCenter Operations.

Hope that helps

e1

[GalNeb]

I have the same issues and it is not of the three stated sources.  No swap, No Ready, Power management is set to OS, OS set to High Performance, C-State and C1E are both disabled.  All procs are nailed at 100%.  Still during high disk I/O periods I see huge CPU Latency.  Even 100%.

[Iwan_Rahabok]

Very interesting.

I did a test on HP DL 380 G8. In my case, it's confirmed the CPU Latency counter dropped from 6-7% to 0.0x % when I changed the power setting to maximum at BIOS. I then changed the power management back to Dynamic at BIOS level, and the CPU Latency went back up to 6-7%

Happy to do a webex and conf call if you want.

My email is e1 @ vmware dot com

I'm based in Singapore.

[Shocko]

Disabling power consumption reduction features for the CPU definitely helps. I have see 3-6% reductions in the latency counter when disabled. However, I still see latency counter run high at times and it's very difficult to determine what the root cause is.

[Iwan_Rahabok]

"4 dual core CPU"? What CPU model is that? It looks like a box from 7 years ago.... This is not meant to be rude. I'm curious why someone would run 4 socket dual-core box. If it's indeed a very old Xeon, then it's hard to analyse as Xeon changes drastically in the past several releases.

[GalNeb]

For a thorough analysis showing similar bogus %Lat_C, take a look at this thread: High %lat_c CPU Latency for 1 VM

I have been documenting %Lat_C numbers for some time and they just plain do not make sense.  The only data I can find on this counter is what has been stated here:

Ready, Swap, Power Settings.  Well, I am here to tell you that there is something else entirely being counted by %Lat_C.  As you can see by the discussion and screenshots in the other thread, there is no Ready, there is no Swap and all the MB setting are set properly.  (c-state, C1E both off, Power to OS, OS to High Performance).  You can even throw in co-stop and it would still be zero.  One guy even asked me to look at NUMA counters and with them at near 100%, the vCPUs are not moving between PCPUs.

Please, can someone at VMware tell us what this thing is telling us when I am getting CPU Usage Alarms due to %Lat_C being high.

[CompNerd84]

I don't mean to drudge up an old thread, but I'm not finding any answers about this specific issue.

the only thing I can find is that cpuUsedPct seems to have a direct relation to cpuLatencyPct.