mem capacity.usage counter gone as of vSphere 5.0U2?

[np123]

I run a weekly host performance report including the memory capacity.usage performance counter.

[LucD]

No, it should still be there.

Do you see the counter when you do ?

Get-ResourcePool MyRP | Get-StatType | where {$_ -like "mem.capacity.usage*"}

[np123]

I accidentally hit Enter before I finished typing my initial post, but you've got the gist... thanks.

Our interval is at stat level 1, and always has been. I wonder if this counter was recently moved from level one into level 3. On our end the change seems to coincide chronologically with a vSphere update from U1 to U2. If I'm not mistaken, our report stopped pulling this metric after that vSphere update, and before the ESXi hosts were updated from U1 to U2.

This doc still says level 1: http://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.wssdk.apiref.doc%2Fmemory_counters.h...

Where did your image come from?

[LucD]

This time I hit Enter too soon :smileygrin: That was the cpu.capacity.usage counter.

I corrected my original reply, can you run the Get-StatType cmdlet to check if the counter is returned ?

[np123]

Nothing returned there. Likewise, "The metric counter "mem.capacity.usage" doesn't exist for entity..." with

get-stat -entity vmhost.domain.tld -stat mem.capacity.usage

[LucD]

Is that entity a resourcepool ?

The SDK Reference seems to only list resourcepool a a valid entity

[np123]

That entity is an ESXi host, not a resource pool. No resource pools here yet beyond the defaults, so tried:

get-cluster clusterName | Get-ResourcePool | Get-StatType | where {$_ -like "mem.*"}

and got:

mem.mementitlement.latest

mem.overhead.average

mem.consumed.average

mem.vmmemctl.average

mem.mementitlement.latest

mem.overhead.average

mem.consumed.average

mem.vmmemctl.average

[LucD]

Do you see the counter under the Performance tab in the vSphere client.

Click Chart Options, select Memory and then scroll down in the Counters field.

That limitation to resourcepools is indeed only for realtime stats, in the Historical Intervals that counter is also available for ESXi hosts.

[np123]

Yes, the counter is present in the vSphere client. It drops to 0 in the charts starting on the day vSphere was upgraded to U2.

Thanks for clarifying the Resource Pool realtime restriction.

[LucD]

Then there seems to be indeed a problem.

Since this counter is an aggregated counter, could it be that the aggregation jobs on the vCenter are not running correctly ?

[np123]

The rollup job history looks clean. Glancing at a few graphs in VC, other counters appear to be unaffected.

[LucD]

I'm out of ideas for now.

It seems to be a vCenter problem for that specific build you are using. I couldn't find any KB articles.

I suggest you open a call.

[np123]

Fair enough, thanks Luc. If I find anything interesting I'll report back.

[np123]

As support request 13269194501 nears its 10-week anniversary...sounds like Stat Level 4 might revive the capacity.usage counter. I'm waiting to hear back about the relevant difference between the consumed and capacity.usage counters. The consumed counter is still available, although it too has Per Device Level 4 in the documentation.

[LucD · Accepted solution]

I did some further research on this and it seems that the API Reference is not correct.

According to the guide, this counter should only be available for resourcepools.

With the following script I discovered that Get-Stat returns metrics for this counter for resourcepools, ESXi hosts and VMs, starting from Historical Interval "Past Week".

And the Statistics level for Past Month and Past Year is not set to level 4 in my test environment.

In fact I wonder what the "device level" would mean for this counter ?

I used this script for my tests.

$clusterName = "MyCluster" 
$objTypes = "ResourcePoolWrapper","VMHostWrapper","VirtualMachineWrapper" 
$cluster = Get-Cluster -Name $clusterName $items = Get-Inventory -Location $cluster | 
where {$objTypes -contains $_.GetType().Name}

Get-StatInterval | %{
  Get-Stat -Stat mem.capacity.usage.average -Entity $items -IntervalSecs $_.SamplingPeriodSecs -MaxSamples 1 -ErrorAction SilentlyContinue |
  Select Timestamp,IntervalSecs,Value,@{N="Entity";E={$_.Entity.Name}},@{N="Type";E={$_.Entity.ExtensionData.GetType().Name}}
}

The test environments had thse versions running: vCenter 5.0.0 build 755629 and ESXi 5.0.0 build 914586.

[np123]

Interesting, thank you. My ESXi build matches yours. My vCenter 5.0.0 build is 913577. I mentioned "device level" because support included a link to the 5.1 Memory Counters page we've been looking at, and said this counter, among others, was "selected to be set at level 4 on a per device basis". I too wonder what "device level" would mean for this counter. Honestly, I'm a bit fuzzy on the meaning of perDeviceLevel in general. I've read the definition, but this is my first encounter with the concept.

More from support:

I traced a bug report on this dating from the 5.0 GA. (PR 792952)

It appears that the stats level required including and later than 5.0 Upgrade 2 is 4, i.e. Ther stats level need to be set to 4 in the vpx_stat_def table of the database for this feature (mem capacity.usage) to be processed. Therefore it is disabled by design unless the stats level is explicitly increased. A total of 38 counters were configured like this in Upgrade 2.

List of counters moved to level 4.

- cpu.capacity.contention

- cpu.capacity.demand

- cpu.capacity.entitlement

- cpu.capacity.provisioned

- cpu.capacity.usage

- cpu.corecount.contention

- cpu.corecount.provisioned

- cpu.corecount.usage

- mem.capacity.contention

- mem.capacity.entitlement

- mem.capacity.provisioned

- mem.capacity.usable

- mem.capacity.usage

- mem.reservedCapacityPct

- net.throughput.contention

- net.throughput.packetsPerSec

- net.throughput.provisioned

- net.throughput.usable

- net.throughput.usage

- power.capacity.usable

- power.capacity.usage

- power.capacity.usagePct

- datastore.throughput.contention

- datastore.throughput.usage

- disk.capacity.provisioned

- disk.capacity.contention

- disk.capacity.usage

- disk.scsiReservationCnflctsPct

- disk.throughput.contention

- disk.throughput.usage

- virtualDisk.throughput.cont

- virtualDisk.throughput.usage

- storageAdapter.OIOsPct

- storageAdapter.throughput.cont

- storageAdapter.throughput.usag

- storagePath.throughput.cont

- storagePath.throughput.usage

Unfortunately the Bug documentation is not publically available but I'll try track down something for you.

[LucD]

The support guy might have a point on a closer look at my data.

On a system that was upgraded in January this year, I can find the metrics for that counter in the Past Year interval, but in the Past Month and Past Week the metrics are missing.

So it looks as if after the upgrade the counter is indeed not collected if you are not running in statistics level 4.

[np123]

Glad I'm not the only one. Hopefully they'll update the documentation accordingly. Thanks again.

[np123]

Support has confirmed:

The levels column should read '4'.

In short, the counters I outlined in a previous mail are set by default at level 1.

The reason is, as outlined, to take the pressure off the DB.

To enable any or all the metrics listed, stats level must be set to '4'

But only for short term debugging.

The documentation leaves the impression that the data will be collected at level 1. This is not the case.

A query we ran against the DB confirms:

Default Level: 1

Collection Level: 4

Per Device Level: 4

[np123]

use VIM_VCDB

select name, STAT_LEVEL from vpx_stat_def

where NAME = 'capacity.usage' and GROUP_NAME = 'mem'

Presumably stat_level would've been 1 before the U2 upgrade. It's 4 now.