BUG? cpu.usagemhz is inconsistent with vCenter on resource pool

[sbourdeaud]

Hi everyone,

I have the following setup in a lab:

A single ESX 3.5.0 123630 host

A VM on that host running vCenter 2.5.0 build 174768 with VI Client of the same build. That VM is also running Powershell 1.0.0.0 and vSphere PowerCLI v4.0 build 162509.

I created a DRS cluster so that I could get resource pools. I have two resource pools, "pool A" and "pool B", with a single VM each.

Now when I look at the performance graph for either one of those resource pools, I get a CPU Mhz average around 30 Mhz, yet when I query the cpu.usagemhz.average performance counter on either one of those resource pools, I get about twice as much Mhz!!!

If I look at a single sample, I see different values. The values shown by vCenter are actually those of the cpu.usagemhz.average of the virtual machine (if I query the counter for that VM, I get the same values as displayed in vCenter).

My questions are:

Can anyone reproduce this in their environment?

Why is the cpu.usagemhz.average higher when reported at the resource pool level using the vSphere PowerCLI than it is reported in vCenter?

How is it possible that a resource pool uses more CPU Mhz than the single virtual machine in that pool? Where are all the extra Mhz coming from?

Any help would be greatly appreciated.

Regards,

Stephane B.

PS: I have added a screenshot of the curve shown in vCenter.

Here is the result of the Powershell request for supposedly the same data (cpu.usagemhz.average on the resource pool).

Note that the values given by the API are around 60-70 Mhz while the vCenter graph shows an average around 30 Mhz...

$mypool = Get-ResourcePool -Name "Customer B"
$mypool | Get-Stat -stat cpu.usagemhz.average -Start 16/09/2009 -IntervalMins 5
MetricId Timestamp Value Unit
-

---

-

---

-

---

• 
  cpu.usagemhz.average 16/09/2009 18:25:00 74 MHz
  cpu.usagemhz.average 16/09/2009 18:20:00 64 MHz
  cpu.usagemhz.average 16/09/2009 18:15:00 73 MHz
  cpu.usagemhz.average 16/09/2009 18:10:00 64 MHz
  cpu.usagemhz.average 16/09/2009 18:05:00 76 MHz
  cpu.usagemhz.average 16/09/2009 18:00:00 62 MHz
  cpu.usagemhz.average 16/09/2009 17:55:00 71 MHz
  cpu.usagemhz.average 16/09/2009 17:50:00 66 MHz
  cpu.usagemhz.average 16/09/2009 17:45:00 76 MHz
  cpu.usagemhz.average 16/09/2009 17:40:00 61 MHz
  cpu.usagemhz.average 16/09/2009 17:35:00 74 MHz
  cpu.usagemhz.average 16/09/2009 17:30:00 62 MHz
  cpu.usagemhz.average 16/09/2009 17:25:00 72 MHz
  cpu.usagemhz.average 16/09/2009 17:20:00 64 MHz
  cpu.usagemhz.average 16/09/2009 17:15:00 74 MHz
  cpu.usagemhz.average 16/09/2009 17:10:00 68 MHz
  cpu.usagemhz.average 16/09/2009 17:05:00 68 MHz
  cpu.usagemhz.average 16/09/2009 17:00:00 60 MHz
  cpu.usagemhz.average 16/09/2009 16:55:00 74 MHz
  cpu.usagemhz.average 16/09/2009 16:50:00 77 MHz
  cpu.usagemhz.average 16/09/2009 16:45:00 64 MHz
  cpu.usagemhz.average 16/09/2009 16:40:00 64 MHz
  cpu.usagemhz.average 16/09/2009 16:35:00 74 MHz
  cpu.usagemhz.average 16/09/2009 16:30:00 64 MHz
  cpu.usagemhz.average 16/09/2009 16:25:00 73 MHz
  cpu.usagemhz.average 16/09/2009 16:20:00 75 MHz
  cpu.usagemhz.average 16/09/2009 16:15:00 64 MHz
  cpu.usagemhz.average 16/09/2009 16:10:00 72 MHz
  cpu.usagemhz.average 16/09/2009 16:05:00 66 MHz
  cpu.usagemhz.average 16/09/2009 16:00:00 72 MHz
  cpu.usagemhz.average 16/09/2009 15:55:00 64 MHz
  cpu.usagemhz.average 16/09/2009 15:50:00 74 MHz
  cpu.usagemhz.average 16/09/2009 15:45:00 61 MHz
  cpu.usagemhz.average 16/09/2009 15:40:00 74 MHz
  cpu.usagemhz.average 16/09/2009 15:35:00 64 MHz
  cpu.usagemhz.average 16/09/2009 15:30:00 70 MHz
  cpu.usagemhz.average 16/09/2009 15:25:00 62 MHz
  cpu.usagemhz.average 16/09/2009 15:20:00 74 MHz
  cpu.usagemhz.average 16/09/2009 15:15:00 64 MHz
  cpu.usagemhz.average 16/09/2009 15:10:00 74 MHz
  cpu.usagemhz.average 16/09/2009 15:05:00 64 MHz
  cpu.usagemhz.average 16/09/2009 15:00:00 72 MHz
  cpu.usagemhz.average 16/09/2009 14:55:00 65 MHz
  cpu.usagemhz.average 16/09/2009 14:50:00 74 MHz
  cpu.usagemhz.average 16/09/2009 14:45:00 64 MHz
  cpu.usagemhz.average 16/09/2009 14:40:00 72 MHz
  cpu.usagemhz.average 16/09/2009 14:35:00 66 MHz
  cpu.usagemhz.average 16/09/2009 14:30:00 74 MHz
  cpu.usagemhz.average 16/09/2009 14:25:00 63 MHz
  cpu.usagemhz.average 16/09/2009 14:20:00 74 MHz
  cpu.usagemhz.average 16/09/2009 14:15:00 64 MHz
  cpu.usagemhz.average 16/09/2009 14:10:00 74 MHz
  cpu.usagemhz.average 16/09/2009 14:05:00 66 MHz
  cpu.usagemhz.average 16/09/2009 14:00:00 74 MHz
  cpu.usagemhz.average 16/09/2009 13:55:00 75 MHz
  cpu.usagemhz.average 16/09/2009 13:50:00 66 MHz
  cpu.usagemhz.average 16/09/2009 13:45:00 70 MHz
  cpu.usagemhz.average 16/09/2009 13:40:00 66 MHz
  cpu.usagemhz.average 16/09/2009 13:35:00 72 MHz
  cpu.usagemhz.average 16/09/2009 13:30:00 63 MHz
  cpu.usagemhz.average 16/09/2009 13:25:00 73 MHz
  cpu.usagemhz.average 16/09/2009 13:20:00 64 MHz
  cpu.usagemhz.average 16/09/2009 13:15:00 72 MHz
  cpu.usagemhz.average 16/09/2009 13:10:00 66 MHz
  cpu.usagemhz.average 16/09/2009 13:05:00 74 MHz
  cpu.usagemhz.average 16/09/2009 13:00:00 64 MHz
  cpu.usagemhz.average 16/09/2009 12:55:00 72 MHz
  cpu.usagemhz.average 16/09/2009 12:50:00 67 MHz
  cpu.usagemhz.average 16/09/2009 12:45:00 75 MHz
  cpu.usagemhz.average 16/09/2009 12:40:00 66 MHz
  cpu.usagemhz.average 16/09/2009 12:35:00 70 MHz
  cpu.usagemhz.average 16/09/2009 12:30:00 63 MHz
  cpu.usagemhz.average 16/09/2009 12:25:00 73 MHz
  cpu.usagemhz.average 16/09/2009 12:20:00 66 MHz
  cpu.usagemhz.average 16/09/2009 12:15:00 73 MHz
  cpu.usagemhz.average 16/09/2009 12:10:00 64 MHz
  cpu.usagemhz.average 16/09/2009 12:05:00 74 MHz
  cpu.usagemhz.average 16/09/2009 12:00:00 62 MHz
  cpu.usagemhz.average 16/09/2009 11:55:00 72 MHz
  cpu.usagemhz.average 16/09/2009 11:50:00 66 MHz
  cpu.usagemhz.average 16/09/2009 11:45:00 74 MHz
  cpu.usagemhz.average 16/09/2009 11:40:00 63 MHz
  cpu.usagemhz.average 16/09/2009 11:35:00 74 MHz
  cpu.usagemhz.average 16/09/2009 11:30:00 62 MHz
  cpu.usagemhz.average 16/09/2009 11:25:00 72 MHz
  cpu.usagemhz.average 16/09/2009 11:20:00 69 MHz
  cpu.usagemhz.average 16/09/2009 11:15:00 68 MHz
  cpu.usagemhz.average 16/09/2009 11:10:00 73 MHz
  cpu.usagemhz.average 16/09/2009 11:05:00 68 MHz
  cpu.usagemhz.average 16/09/2009 11:00:00 74 MHz
  cpu.usagemhz.average 16/09/2009 10:55:00 63 MHz
  cpu.usagemhz.average 16/09/2009 10:50:00 74 MHz
  cpu.usagemhz.average 16/09/2009 10:45:00 65 MHz
  cpu.usagemhz.average 16/09/2009 10:40:00 74 MHz
  cpu.usagemhz.average 16/09/2009 10:35:00 84 MHz
  cpu.usagemhz.average 16/09/2009 10:30:00 85 MHz
  cpu.usagemhz.average 16/09/2009 10:25:00 60 MHz
  cpu.usagemhz.average 16/09/2009 10:20:00 75 MHz
  cpu.usagemhz.average 16/09/2009 10:15:00 64 MHz
  cpu.usagemhz.average 16/09/2009 10:10:00 71 MHz
  cpu.usagemhz.average 16/09/2009 10:05:00 68 MHz
  cpu.usagemhz.average 16/09/2009 10:00:00 72 MHz
  cpu.usagemhz.average 16/09/2009 09:55:00 62 MHz
  cpu.usagemhz.average 16/09/2009 09:50:00 76 MHz
  cpu.usagemhz.average 16/09/2009 09:45:00 64 MHz
  cpu.usagemhz.average 16/09/2009 09:40:00 72 MHz
  cpu.usagemhz.average 16/09/2009 09:35:00 72 MHz
  cpu.usagemhz.average 16/09/2009 09:30:00 67 MHz
  cpu.usagemhz.average 16/09/2009 09:25:00 64 MHz
  cpu.usagemhz.average 16/09/2009 09:20:00 76 MHz
  cpu.usagemhz.average 16/09/2009 09:15:00 62 MHz
  cpu.usagemhz.average 16/09/2009 09:10:00 73 MHz
  cpu.usagemhz.average 16/09/2009 09:05:00 77 MHz
  cpu.usagemhz.average 16/09/2009 09:00:00 64 MHz
  cpu.usagemhz.average 16/09/2009 08:55:00 64 MHz
  cpu.usagemhz.average 16/09/2009 08:50:00 77 MHz
  cpu.usagemhz.average 16/09/2009 08:45:00 72 MHz
  cpu.usagemhz.average 16/09/2009 08:40:00 62 MHz
  cpu.usagemhz.average 16/09/2009 08:35:00 66 MHz
  cpu.usagemhz.average 16/09/2009 08:30:00 72 MHz
  cpu.usagemhz.average 16/09/2009 08:25:00 62 MHz
  cpu.usagemhz.average 16/09/2009 08:20:00 74 MHz
  cpu.usagemhz.average 16/09/2009 08:15:00 64 MHz
  cpu.usagemhz.average 16/09/2009 08:10:00 72 MHz
  cpu.usagemhz.average 16/09/2009 08:05:00 72 MHz
  cpu.usagemhz.average 16/09/2009 08:00:00 66 MHz
  cpu.usagemhz.average 16/09/2009 07:55:00 71 MHz
  cpu.usagemhz.average 16/09/2009 07:50:00 67 MHz
  cpu.usagemhz.average 16/09/2009 07:45:00 72 MHz
  cpu.usagemhz.average 16/09/2009 07:40:00 64 MHz
  cpu.usagemhz.average 16/09/2009 07:35:00 74 MHz
  cpu.usagemhz.average 16/09/2009 07:30:00 62 MHz
  cpu.usagemhz.average 16/09/2009 07:25:00 70 MHz
  cpu.usagemhz.average 16/09/2009 07:20:00 61 MHz
  cpu.usagemhz.average 16/09/2009 07:15:00 69 MHz
  cpu.usagemhz.average 16/09/2009 07:10:00 70 MHz
  cpu.usagemhz.average 16/09/2009 07:05:00 70 MHz
  cpu.usagemhz.average 16/09/2009 07:00:00 72 MHz
  cpu.usagemhz.average 16/09/2009 06:55:00 64 MHz
  cpu.usagemhz.average 16/09/2009 06:50:00 66 MHz
  cpu.usagemhz.average 16/09/2009 06:45:00 74 MHz
  cpu.usagemhz.average 16/09/2009 06:40:00 73 MHz
  cpu.usagemhz.average 16/09/2009 06:35:00 62 MHz
  cpu.usagemhz.average 16/09/2009 06:30:00 72 MHz
  cpu.usagemhz.average 16/09/2009 06:25:00 60 MHz
  cpu.usagemhz.average 16/09/2009 06:20:00 62 MHz
  cpu.usagemhz.average 16/09/2009 06:15:00 76 MHz
  cpu.usagemhz.average 16/09/2009 06:10:00 70 MHz
  cpu.usagemhz.average 16/09/2009 06:05:00 78 MHz
  cpu.usagemhz.average 16/09/2009 06:00:00 70 MHz
  cpu.usagemhz.average 16/09/2009 05:55:00 66 MHz
  cpu.usagemhz.average 16/09/2009 05:50:00 78 MHz
  cpu.usagemhz.average 16/09/2009 05:45:00 62 MHz
  cpu.usagemhz.average 16/09/2009 05:40:00 73 MHz
  cpu.usagemhz.average 16/09/2009 05:35:00 64 MHz
  cpu.usagemhz.average 16/09/2009 05:30:00 72 MHz
  cpu.usagemhz.average 16/09/2009 05:25:00 64 MHz
  cpu.usagemhz.average 16/09/2009 05:20:00 74 MHz
  cpu.usagemhz.average 16/09/2009 05:15:00 62 MHz
  cpu.usagemhz.average 16/09/2009 05:10:00 73 MHz
  cpu.usagemhz.average 16/09/2009 05:05:00 66 MHz
  cpu.usagemhz.average 16/09/2009 05:00:00 72 MHz
  cpu.usagemhz.average 16/09/2009 04:55:00 63 MHz
  cpu.usagemhz.average 16/09/2009 04:50:00 73 MHz
  cpu.usagemhz.average 16/09/2009 04:45:00 62 MHz
  cpu.usagemhz.average 16/09/2009 04:40:00 71 MHz
  cpu.usagemhz.average 16/09/2009 04:35:00 68 MHz
  cpu.usagemhz.average 16/09/2009 04:30:00 66 MHz
  cpu.usagemhz.average 16/09/2009 04:25:00 74 MHz
  cpu.usagemhz.average 16/09/2009 04:20:00 64 MHz
  cpu.usagemhz.average 16/09/2009 04:15:00 71 MHz
  cpu.usagemhz.average 16/09/2009 04:10:00 76 MHz
  cpu.usagemhz.average 16/09/2009 04:05:00 76 MHz
  cpu.usagemhz.average 16/09/2009 04:00:00 61 MHz
  cpu.usagemhz.average 16/09/2009 03:55:00 64 MHz
  cpu.usagemhz.average 16/09/2009 03:50:00 74 MHz
  cpu.usagemhz.average 16/09/2009 03:45:00 74 MHz
  cpu.usagemhz.average 16/09/2009 03:40:00 60 MHz
  cpu.usagemhz.average 16/09/2009 03:35:00 70 MHz
  cpu.usagemhz.average 16/09/2009 03:30:00 66 MHz
  cpu.usagemhz.average 16/09/2009 03:25:00 73 MHz
  cpu.usagemhz.average 16/09/2009 03:20:00 64 MHz
  cpu.usagemhz.average 16/09/2009 03:15:00 76 MHz
  cpu.usagemhz.average 16/09/2009 03:10:00 70 MHz
  cpu.usagemhz.average 16/09/2009 03:05:00 78 MHz
  cpu.usagemhz.average 16/09/2009 03:00:00 64 MHz
  cpu.usagemhz.average 16/09/2009 02:55:00 73 MHz
  cpu.usagemhz.average 16/09/2009 02:50:00 64 MHz
  cpu.usagemhz.average 16/09/2009 02:45:00 72 MHz
  cpu.usagemhz.average 16/09/2009 02:40:00 62 MHz
  cpu.usagemhz.average 16/09/2009 02:35:00 72 MHz
  cpu.usagemhz.average 16/09/2009 02:30:00 63 MHz
  cpu.usagemhz.average 16/09/2009 02:25:00 74 MHz
  cpu.usagemhz.average 16/09/2009 02:20:00 65 MHz
  cpu.usagemhz.average 16/09/2009 02:15:00 74 MHz
  cpu.usagemhz.average 16/09/2009 02:10:00 62 MHz
  cpu.usagemhz.average 16/09/2009 02:05:00 78 MHz
  cpu.usagemhz.average 16/09/2009 02:00:00 72 MHz
  cpu.usagemhz.average 16/09/2009 01:55:00 64 MHz
  cpu.usagemhz.average 16/09/2009 01:50:00 66 MHz
  cpu.usagemhz.average 16/09/2009 01:45:00 72 MHz
  cpu.usagemhz.average 16/09/2009 01:40:00 62 MHz
  cpu.usagemhz.average 16/09/2009 01:35:00 75 MHz
  cpu.usagemhz.average 16/09/2009 01:30:00 61 MHz
  cpu.usagemhz.average 16/09/2009 01:25:00 75 MHz
  cpu.usagemhz.average 16/09/2009 01:20:00 72 MHz
  cpu.usagemhz.average 16/09/2009 01:15:00 75 MHz
  cpu.usagemhz.average 16/09/2009 01:10:00 80 MHz
  cpu.usagemhz.average 16/09/2009 01:05:00 64 MHz
  cpu.usagemhz.average 16/09/2009 01:00:00 70 MHz
  cpu.usagemhz.average 16/09/2009 00:55:00 67 MHz
  cpu.usagemhz.average 16/09/2009 00:50:00 74 MHz
  cpu.usagemhz.average 16/09/2009 00:45:00 64 MHz
  cpu.usagemhz.average 16/09/2009 00:40:00 74 MHz
  cpu.usagemhz.average 16/09/2009 00:35:00 64 MHz
  cpu.usagemhz.average 16/09/2009 00:30:00 72 MHz
  cpu.usagemhz.average 16/09/2009 00:25:00 64 MHz
  cpu.usagemhz.average 16/09/2009 00:20:00 74 MHz
  cpu.usagemhz.average 16/09/2009 00:15:00 64 MHz
  cpu.usagemhz.average 16/09/2009 00:10:00 72 MHz
  cpu.usagemhz.average 16/09/2009 00:05:00 68 MHz
  cpu.usagemhz.average 16/09/2009 00:00:00 74 MHz

[sbourdeaud]

I think I figured it out.

I noticed that when I look at the VM cpu.usagemhz.average, I always get two sample values for a single time stamp.

Researching this issue led me to the following thread: http://communities.vmware.com/thread/226980

I then noticed that the value reported by get-stat -stat cpu.usagemhz.average on a resource pool corresponds to both Instances being added up. I can only assume this is a bug in get-stat. It should actually take only the aggregate value (which has instance set to null).

Any VMware PowerCLI developpers out there care to take a look at this?

Regards,

Stephane B.

[admin]

Are you sure this is not related to the object having several instances? Can you pipe your output into "group Instance" like this:

[vSphere PowerCLI] C:\> get-stat -stat cpu.usage.average | group Instance

Count Name                      Group
----- ----                      -----
  432 0                         {VMware.VimAutomation.Client20.FloatSampleIm...
  432 1                         {VMware.VimAutomation.Client20.FloatSampleIm...
  432 2                         {VMware.VimAutomation.Client20.FloatSampleIm...
  432 3                         {VMware.VimAutomation.Client20.FloatSampleIm...
  432 4                         {VMware.VimAutomation.Client20.FloatSampleIm...
  432 5                         {VMware.VimAutomation.Client20.FloatSampleIm...
  432 6                         {VMware.VimAutomation.Client20.FloatSampleIm...
  432 7                         {VMware.VimAutomation.Client20.FloatSampleIm...
  432                           {VMware.VimAutomation.Client20.FloatSampleIm...

=====

Carter Shanklin

Read the PowerCLI Blog
[Follow me on Twitter|http://twitter.com/cshanklin]

[LucD]

I can reproduce the problem you're seeing with PowerCLI 4 on a VC 2.5/ESX 3.5 environment.

The values the Get-Stat cmdlet is returning for a resource pool on the cpu.usagemhz.average is approx. twice the actual value.

If you do the same exercise with my Get-Stat2 script (see ) you get the values that correspond with what you see under the Performance tab in the VIC.

Note that my Get-Stat2 script returns the timestamp in UTC time, you will have to add the offset of your location to have the same timestamps as Get-Stat is returning.

I don't understand why you are seeing 2 instances being returned for that metric on a resourcepool.

That shouldn't be the case.

Or do you by any chance have 2 resource pools with the same name but in different clusters ?

[sbourdeaud]

Hello and thanks for your reply.

Here is my code and the output on both the resource pool and the vm in the resource pool:

$mypool = Get-ResourcePool -Name "Customer A"

$mypool | Get-Stat -stat cpu.usagemhz.average -start (get-date).addhours(-1) -IntervalMins 5 | group Instance

Count Name Group

12 {cpu.usagemhz.average, cpu.usagemhz.average, cpu.usagemhz.average, cpu.usagemhz

$myvm = Get-VM -Name "custAvm01"

$myvm | Get-Stat -stat cpu.usagemhz.average -start (get-date).addhours(-1) -IntervalMins 5 | group Instance

Count Name Group

11 {cpu.usagemhz.average, cpu.usagemhz.average, cpu.usagemhz.average, cpu.usagemhz

11 0 {cpu.usagemhz.average, cpu.usagemhz.average, cpu.usagemhz.average, cpu.usagemhz

Note that I get two instances for the VM as explained in the thread mentionned in my first reply.

Again, if I add those two instances for a single sample, I get the value returned by the get-stat command on the resource pool.

Regards,

Stephane B.

[sbourdeaud]

Hello and thanks for your reply.

No, I do not have two resource pools with the same name, and I only get one sample for the resource pool.

It is for the VM that I get two samples for every time stamp.

I believe it is the get-stat command that erroneously adds those two samples instead of keeping only the aggregate sample as explained in the thread I referenced earlier.

Regards,

Stephane B.

[LucD]

Correct.

With my Get-Stat2 script you will get the correct values.

[yboychev]

I can also reproduce this as described. Simply the statistics returned for the resourcepool is almost double to the one for the avg ones returned for the single vm. I've logged the apropriate bug in our bug tracking system. 10x for catching this for us!

\Yavor