vCenter VM CPU usage metrics vs VM Linux OS CPU usage metrics

[EBoucq]

Hi,

I notice that sometimes, there's a difference between the metrics given by the vCenter and those from the OS inside a VM.

For instance, at the moment, we have a SLES 12 VM for which the vCenter shows a real-time cpu usage between 73 and 97% although a vmstat indicates a max of 61%

Any idea what could explain the difference ? Should we trust the vCenter when it comes to monitor VM CPU usage ?

Thanks in advance for your answers.

Eric

[Tibmeister]

This is a very classic situation due to the point of view of the two items.  vCenter get's it's statistics from ESXi, which sees what is being used by the VM on the physical CPU via the scheduler.  This is actually a very accurate view, when taking into account IOWAIT, CO-STOP, and READY.

Now from the guest OS perspective, this get's different because the VM uses watchdog timers to determine the load of a CPU, regardless if it's physical or virtual.  The OS will launch a watchdog on every CPU it sees (remember, this is from the VM Guest OS perspective) as a background task, then measures how long it takes for this timer to complete.  Since it's a background task, it is the last thing to execute on the CPU.  So, this measures how long the CPU takes to get to this watchdog task, therefore it's a good approximation of how busy the CPU is, based on clock cycles and such.

The issue comes in that the guest OS is not aware of the external items, such as IOWAIT, CO-STOP, etc.  So, when these items are impacting the VM, even though the guest OS isn't doing much, that watchdog takes a long time to return, so therefore the guest OS thinks the CPU(s) are more heavily used than what they are.  This really comes into play with over-sized CPU's experiencing high CO-STOP.  So basically, the in-guest metrics for CPU should be taken with a grain of salt.  Not completely ignored, but weighed much less than the ESXi metrics.

Now with that, you also have to understand that vCenter does not present the raw data, but rather a roll-up of data, further making the data inconsistent with the in-guest, because you are looking at two different time periods and polling periods.  esxtop is really the best way to troubleshoot VM performance, or vROps, because it stores the raw data and you can change from a 5-minute resolution down to 1 second if you are so inclined.

[vbondzio]

What Tibmeister said, but ignoring non-run states, most of the difference comes from ESXi CPU Usage being a qualitative counter that takes into account CPU frequency to approximate throughput. Most OSs default to utilization (i.e. anything but idle), so that would under-report due to Turbo Boost (and even if they would _want_ to use frequency information for a usage like counter, the necessary information for that isn't presented to a VM by default).

I do explain it in some detail here (~15-50min): https://via.vmw.com/MCL1635

[EBoucq]

Thanks for your answer !

It's still confusing though because I looked at this particular VM metrics with esxtop and I see a %USED at 370%.

So I don't know what to think when the Linux admin tells me that it sees a 50% CPU usage in his machine and doesn't have any performance issue with his server.

[vbondzio]

esxtop will show you the sum of the percentages of each world, so maybe that VM has 6-8 vCPUs. You can expand the VM via 'e' -> GID to see the individual world values (think threads in a process).

[EBoucq]

Thanks vbondzio !

The VM has 4 vCPUS. So now I see the usage on the 4 vCPUs. Should I do something since one of the vcpu is over 100% ?

[vbondzio]

%RUN / utilization is closest to what you will see in the guest, if one vCPU is capped but others are not, adding more vCPUs might not help if the application(s) are single / low threaded. 370% usage on 4 vCPUs should be more than 50% in guest utilization, unless the CPU has a very wide NF / max TB span and nothing else is running on that host, maybe it is also doing lots of IO (charged via %SYS to vmx)? Can you limit 'l' the GID in addition to expanding it and pastebin the output?

[Tibmeister]

I've seen the > 100% utilization for a CPU and only was able to observe that when HT was enabled on the host.  Since HT does show as a core, but only provides, at most, 50% increase in performance per core, I always have assumed it was due to this and the lack of ability to determine if the core being run on is a logical or physical thread, which would have to influence the calculation.

Showing that to users I've always been able to say "See, you don't need as many cores in a VM because I can drive a CPU past 100%!".  It's easier than explaining co-stop, single-threaded apps, etc.

Also, if you are seeing a single core on a 4 core being pegged, you definitely are running a large single-threaded application, and I will bet you see high co-stop values as well.  I would reduce to 2 cores and monitor, and if the aggregate of both cores is between 75% and 85% over a 30 day period, you are optimal in that configuration.  I say 30 day because you want to account for odd spikes in workload and patching.

[EBoucq]

Thks again for your explanation ! Very useful!