High "Memory Consumption" Alert from VM?

[CaverDave]

Host = ESXi 3.5 w/ 2x Quadcores/16GB RAM (very under utilized)

VM = W2K3 Standard w/ 2x vCPU/4GB Memory

I've had this VM running trouble free for almost a year now! :smileycry:

Last week, a 3rd party software package started sending alerts that a memory consumption treshold (95%) had been met. Unfortunately, outside of configurable threshold, frequency, and email addy, that's all I have. I have a call into their engineers to find out exactly what method/library they use.

Via perfmon, I see occassional spikes in memory use, but it averages (via TaskMangler) 2.5GB free "Physical Memory" most of the time. I've got ProcMon fired up on it now to see if that shows anything. Not sure it matters, but I recently turned on Windows Updates on the box & it inhaled several, but has been restarted numerous times since then...

I'm going on a hunch that their (likely) antiquated methods of retrieving this metric is getting munged by the fact it's virtual?

Has anyone experienced similar? Have any wisdom regarding memory reporting on VMs?

[mittim12]

What do your consumed and active memory counters in vCenter say?  Are you able to look back at vCenter reports around the time the alert is generted?    Does the VM experience any type of paging? 

[CaverDave]

mittim12 wrote:

What do your consumed and active memory counters in vCenter say?

Edit: Report attached. Infrastructure Client connecting to ESXi 3.5 host shows a "Resources" box & lists:

• CPU usage: 424MHz
• Host Memory usage: 1.75GB
• Guest Memory usage: 310MB

mittim12 wrote:

Are you able to look back at vCenter reports around the time the alert is generted?

Unfortunately, ESXi will *only* keep an hours worth of "Real-Time" performance data. Haven't been able to get it backed off to a "5 minute" config (should make for less granular/longer logging?). Not sure if a sheduled export to XLS is possible? That would give me a bit more to go on...

mittim12 wrote:

Does the VM experience any type of paging? 

I have noticed the Page File spike (via TaskMangler) from 1.25GB avg up to 1.8-2GB... is that what you mean?

I'll be happy to provide additional info if needed!

[4nd7]

Hello,

The ballooning process forces the guest operating system to page. If you overprovision memory to the virtual machine, the host will try to reclaim unused memory pages by using ballooning.

I would decrease the memory allocated and see how the ballooning activity will be after.

Thanks!

[mittim12]

4nd7 is correct when there is memory contention on the ESX host then memory will be ballooned from guest.   If your third party application is set to trigger based on page file usage the ballooning could certainly cause this. You can reduce the unused memory in the machine generating the error or reduce the memory contention on the host so that ballooning is not necessary

[CaverDave]

I'm pretty sure there's no overprovisioning/contention on the host... unless the Host Memory usage (14.09GB used of 15.99GB) is reporting incorrectly? Or am I reading it wrong? Guest Memory % is near/over 100%

Also, I need to check the specifics, but these alerts appear to have started about the time I flew 2 more VMs (XP w/ 1vCPU/1GB & 2008R2 w/ 2vCPU/4GB) up on this host!

[mittim12]

The machine isn't always ballooning so that would seem to point to high utilization during different parts of the day.  You can always login and use esxtop to verify how much ballooning and swapping is happening on a per host level.  

[4nd7]

In my opinion the memory tax for idle virtual machines is applied even if there is no contention on the host

[mittim12]

I've never seen any type of ballooning take place unless there was contention for resources or a limit was placed on the VM.    Nice blog post below on idle tax

http://www.boche.net/blog/index.php/2009/01/29/idle-memory-tax/

[CaverDave]

You can always login and use esxtop to verify how much ballooning and swapping is happening on a per host level.  

I'm trying this, but have hit snag (which I'm assuming is due to ESXi version?) where all esxtop is displaying appears to be CPU related (NO %MEM or %SWPD columns). I tried toggling the fields via "f", but only see A-H (ID, GID, NAME, NWLD, %STATETIMES, Event Counts, CPU Alloc, & CPU Summary Stats) :smileyconfused:

[4nd7]

Hit 'm' first

[4nd7]

mittim12 you are right. I have just tested it and even when there is memory over commitment there is no idle tax. There has to be contention.

[CaverDave]

4nd7 wrote:

Hit 'm' first

Gotcha! Now I'm lost again (little previous 'nix/VMware training)

[mittim12]

After you hit M for memory you will need to hit F and make sure the MCTL counter is displaying.   That should give you ballooning information.

Here is a link to counter descriptions

http://communities.vmware.com/docs/DOC-5240

[CaverDave]

My apologies for the delay in responding, but was not well late last week.

Also, just to clarify... this is a SINGLE standalone ESXi 3.5 host (DEV environment) with ONLY local storage.

Does this look any better? All memory options toggled, but still appears to be a few missing? Even after reading the linked doc on mem counters, I'm still not comprehending what I'm seeing :smileyconfused:

Message was edited by: CaverDave

[4nd7]

The fields don't fit on your screen. Either turn off some, or move the MCTL fields by pressing 'o' and shift+j.

[CaverDave]

Thank you 4and7!

Is this any better?

[4nd7]

The host is in a high state (http://www.van-lieshout.com/2009/05/esx-memory-management-%E2%80%93-part-3/), meaning that it has enough memory, but it used ballooning and swapping to get there.

I would reduce configured memory of the virtual machines.