strange memory usage

[dmitrif]

Hi all,

I'm getting some strange performance numbers on a root resource pool of a host systems on a cluster:

mem.usage.average {-270309, -276587, -239023, -261793...}

mem.usage.minimum {3796, 3396, 3196, 3296...}

mem.usage.maximum {5796, 5696, 4596, 4896...}

Any idea why the average values would come out negative?

I use C# VI SDK to retrieve it, but I get similar numbers using PowerShell VIToolkit.

[stumpr]

Type conversion from floats? Not a lot of experience with C#.

[dmitrif]

Thanks, but I'm pretty sure this is not a conversion error, I looked at the values when I receive them from VI API.

They are negatives in that array of performance metric samples.

My guess would be the fact this is a host system on a cluster, maybe they are simply not applicable?

On related note, I do receive some cluster compute resource counter ids as "-1" (one or two from 32 available), I consider them garbage.

[stumpr]

Yeah, I converted them from unsigned, numbers are too high...could be a bug. Not sure why you'd get valid min/max values but invalid averages.

[Steve_Jin]

What versions of ESX/VC do you have? You can also try ManagedObjectBrowser to see what you get there.

[dmitrif]

ESX 3, VC 2.5

and how do I use mob for getting performance metrics, please?

The most I can get is runtime.memory.overallUsage 3,948,937,216 and maxUsage 30,044,848,128

But mem.usage.* set is suppose to be presented in hundredth of a percent, as a value between 0 and 10,000.

[dmitrif]

This is a 32GB of RAM cluster, BTW, with six vms, and the numbers I refer to are received from a root resource pool of an instance of ClusterComputeResource (resourcePool MOR).

[Steve_Jin]

To access MOB: https://<host>/mob/

You will need to provoide xml as input to the method calls.

[dmitrif]

Thanks, Steve, are you saying I'd need to invoke PerfMgr methods through mob with xml as parameters?

How is it different from me doing so from my C# code or from PowerShell VIToolkit?

Sorry, I'm not sure I understand a value of doing it this way, I'm pretty sure I'd get the same negative values...

[Steve_Jin · Accepted solution]

Trying MOB is just to isolate the problem since there was doubt whether it's a problem of the system or the C#/Powershell related conversion problem.

[lockenkeyster]

Are you still seeing this issue? I am on 3.5 Update 2 and doing a get-stat with the VI Toolkit pulls up #s like these for memory usage of a resource pool:

mem.usage.average 2/22/2009 1:50:00 PM -1.931662E+10 %

mem.usage.average 2/22/2009 1:45:00 PM -1.802813E+10 %

mem.usage.average 2/22/2009 1:40:00 PM -1.43774E+10 %

mem.usage.average 2/22/2009 1:35:00 PM -1.623498E+10 %

mem.usage.average 2/22/2009 1:30:00 PM -1.74483E+10 %

mem.usage.average 2/22/2009 1:25:00 PM -1.580548E+10 %

mem.usage.average 2/22/2009 1:20:00 PM -1.716913E+10 %

mem.usage.average 2/22/2009 1:15:00 PM -1.845762E+10 %

mem.usage.average 2/22/2009 1:10:00 PM -1.465658E+10 %

mem.usage.average 2/22/2009 1:05:00 PM -1.401233E+10 %

I was trying to see if it was the particular version of PowerShell or a bug in the Toolkit except that the numbers are also very strange in the vCenter Performance tabs, ruling both options out. This occurs for all of our Resource Pools but no other entities in this vCenter instance.

[dmitrif]

Last time I checked (a week ago) it was the same:

{-303613083648, -374849142784, -421120704512, -299473305600...}

I had to hack my code to report zeros instead of this garbage...

[lockenkeyster]

So this thread has helped me out a bit:

Essentially what I ended up doing was upping the Statistics Levels in VirtualCenter to Level 2. (Administration -> VirtualCenter Management Server Configuration -> Statistics -> Edit...)

This seems to be pulling the correct data in, although some queries still report back negative numbers. For example, the get-stat from before still gives:

C:\Program Files\VMware\Infrastructure\VIToolkitForWindows> get-stat -entity $pool -stat mem.usage.average

-realtime -maxsamples 10

MetricId Timestamp Value Unit

-

---

-

---

-

---

-

---

mem.usage.average 2/23/2009 3:10:00 PM -2.024003E+10 %

mem.usage.average 2/23/2009 3:05:00 PM -2.152852E+10 %

mem.usage.average 2/23/2009 3:00:00 PM -2.066953E+10 %

mem.usage.average 2/23/2009 2:55:00 PM -1.953136E+10 %

mem.usage.average 2/23/2009 2:50:00 PM -1.824287E+10 %

mem.usage.average 2/23/2009 2:45:00 PM -1.730872E+10 %

mem.usage.average 2/23/2009 2:40:00 PM -1.795296E+10 %

mem.usage.average 2/23/2009 2:35:00 PM -2.732673E+10 %

mem.usage.average 2/23/2009 2:30:00 PM -2.474975E+10 %

mem.usage.average 2/23/2009 2:25:00 PM -2.074469E+10 %

But a newly accessible statistic (active.average) returns:

C:\Program Files\VMware\Infrastructure\VIToolkitForWindows> get-stat -entity $pool -stat mem.active.average

-realtime -maxsamples 10

MetricId Timestamp Value Unit

-

---

-

---

-

---

-

---

mem.active.average 2/23/2009 3:00:00 PM 202024 KB

mem.active.average 2/23/2009 2:55:00 PM 190838 KB

mem.active.average 2/23/2009 2:50:00 PM 178256 KB

mem.active.average 2/23/2009 2:45:00 PM 169169 KB

mem.active.average 2/23/2009 2:40:00 PM 175460 KB

mem.active.average 2/23/2009 2:35:00 PM 267035 KB

mem.active.average 2/23/2009 2:30:00 PM 241870 KB

mem.active.average 2/23/2009 2:25:00 PM 202722 KB

mem.active.average 2/23/2009 2:20:00 PM 180352 KB

mem.active.average 2/23/2009 2:15:00 PM 195032 KB

I'm still playing around with these stats that are now there for memory, but it looks like we're heading in the right direction. I am having to make custom settings for the graphs to get them to show up in the vCenter due to the disparity in values between stats (e.g. 10 isn't going to show up on a graph with y axis from 0 - 100000). Also, this process may take a while to trickle data in as it is the first time we are collecting many of these counters.

Long story short - maybe check your Statistic Levels? Let me know.

[dmitrif]

Thanks for the hint.

My collection level is 3 and this is the very first thing I check when having trouble with performance metrics.

Level 3 covers all of them, except min/max.

Also I never had problems with active memory numbers, only usage and only on this object, this is definitely a bug in ESX server, just for some unknown to me reason nobody from VMware comments on it.

[njain]

"mem.usage" counter is not a valid counter for Resource Pool. With the release of VirtualCenter 2.5 Update 4, this counter has been now removed for resource pool.

Please refer to the resolved issue "VI Client Displays Memory Statistics for Resource Pools only When the Statistics Level is 2 or Higher" under "VirtualCenter, VI Client, and Web Access" resolved issues for details:

http://www.vmware.com/support/vi3/doc/vi3_vc25u4_rel_notes.html#resolvedviissues

[dmitrif]

Thanks for the info.

What's about virtual disk latency metrics and all managed objects CPU/memory entitlements, have they been removed too? I do not receive them at all...