VMware vSphere Guest SDK ready time not calculated correctly?

[inivanoff1]

I downloaded vSphere Guest SDK (for vSphere 5.5) from the official VMware web site: VMware-GuestSDK-9.4.0-1280544.tar.gz. In my case I downloaded the Linux version.

Everything seems fine. I also found a ready-to-use Python script for interacting with libvmGuestLib. Thanks to this script I was able to use the VMware vSphere Guest SDK accessor functions for virtual machine data. The whole thing behind the scripts is that vmguestlib.py is used to define all the functions from the API and vmguest-stats is the tool to get all virtual machine counters in some human readable form.

And what does vmguest-stats exactly do? It calls 3 functions:

    OldElapsedMs = gl.GetElapsedMs()

    OldStolenMs = gl.GetCpuStolenMs()

    OldUsedMs = gl.GetCpuUsedMs()

Then sleeps for a while (set to 2 seconds in the delay variable) and calls the same functions again:

    NewElapsedMs = gl.GetElapsedMs()

    NewStolenMs = gl.GetCpuStolenMs()

    NewUsedMs = gl.GetCpuUsedMs()

Last, a calculation is done within the period:

    UsedCpu = (NewUsedMs - OldUsedMs) * 100.0 / (NewElapsedMs - OldElapsedMs)

    StolenCpu = (NewStolenMs - OldStolenMs) * 100.0 / (NewElapsedMs - OldElapsedMs)

    EffectiveMhz = gl.GetHostProcessorSpeed() * (NewUsedMs - OldUsedMs) / (NewElapsedMs - OldElapsedMs)

But there is some kind of mistery with the most important value for me -- the stolen time. If you are running a VMware guest and you need information for the stolen time of your VM (called ready time by VMware in some of their documents), you have to either use vSphere or esxtop, or get the value from the VMware vSphere Guest API. I am running the Python script on several machines along with the statistics from esxtop and the values do not have any differencies. For example, the values for %RDY (from esxtop) and CpuStolen (from the Guest SDK API) are almost equal (around 0% -- 0.5% -- 1%).

I installed and configured two new virtual machines on a new physical host and they are the only virtual machines on the host. I set up the Python script along with all other monitoring scripts and for the first two weeks the CPU usage and CPU stolen time were around 0%. The virtual machines are not in production yet, so they are running mostly in idle. And one day I saw my monitoring system has generated some triggers for 25% and above stolen time on both the virtual machines (reported from the Python script)! This is a very high value for a ready time. My first thought was that the two virtual machines are waiting each other for CPU time. My assumption was not good enough: the virtual machines, as I said, were running most of the time in idle. And neither esxtop, nor vSphere was reporting for high CPU loads.

Then I checked the CPU stolen values from esxtop and vSphere and they were around 0%: 0.35% maximum seen from esxtop (by pressing repeatedly the space for refreshing the screen) and maximum 176 ms (average 89 ms) of ready time seen from vSphere (for a period of 20 seconds, which is the default).

Then I run the Python script in order to get the raw values returned from the gl.GetElapsedMs(), gl.GetCpuStolenMs() and gl.GetCpuUsedMs() functions (in milliseconds - the way vSphere Guest SDK gives them):

	- Total elapsed time since the VM has started:	178939950 ms
	- CPU used time:	47615288 ms
	- CPU ready time:	64730408 ms

These are the results without additional calculations. As you can see the ready time is a larger number compared to used time. It seems my machine is not using CPU, but it is waiting for CPU and is not scheduled to run.

My question is: why does vSphere Guest SDK report such large values for CPU ready time while esxtop and vSphere monitoring are reporting almost 0% of all loads (neither CPU used, nor ready time or any other parameter is above 0-1%). My VMs are working in idle, but from vSphere Guest SDK it seems that they have a high stolen time values.

I tried an older version of VMware vSphere Guest SDK (9.0) and the results are the same.

Thank you!