Guest VM shows 90% memory usage, doesn't match vSphere

[Steven_Gray]

We have a monitoring software that is alerting that a virtual server has high memory usage, when the admin of that server logs in to the Windows Server 2008 box he see's windows is reporting 90% memory usage:

However when we look at the Performance tab in vSphere we see that vmware thinks the VM is only using on average 12% memory:

What explains the difference that not only I can understand better but also explain to the server admin and management as to why the huge difference in numbers?

Thanks in advance.

Steven

[frankdenneman]

Steven, you are looking at the average active memory.

That reports the activity of the vm in the physical memory.

The consumed memory of the virtual machine matches the physical memory usage reported in the windows UI.

Might I guess that you are running either a Java based application or an application with a large cache (oracle for example)

[Steven_Gray]

Frank,

This is actually a front end to run sales reports and such. The database however is located on a second server. No java or oracle.

Steven

[aaronwsmith]

Another thing to keep in mind here is the 2008+ uses memory for file I/O caching to improve performance and reduce disk I/O.  So in your snapshot the VM is almost using 1 GB for file cache.  That cache is discarded as needed to service application memory needs, but it makes your machine look memory starved.  Might want to see if you can change how memory is monitored on your VMs to account for the cache.

[aaronwsmith]

For more info:

http://rickardnobel.se/windows-2008-r2-memory-counters-in-task-manager/

[Steven_Gray]

I don't believe I'm following, is the server really using 90% memory and need to be increased and if so why does vSphere show a much different percent usage?

[aaronwsmith]

Under 2008+, true memory free = cache + free.  So in your case, the 4GB VM is using 3GB with 1GB truely free.

The memory stats seens in vCenter reflect active memory access.  That is, the hypervisor knows when a VM is accessing memory and can report on the amount of memory that's "active" in this regard.  But it cannot report on how much memory the GuestOS consider to be consumed/free.

You're basically reviewing 2 completely different stats that will not line up because they are reporting on different things. 

Pretend I'm an operating system with running applications that collectively conumes 2GB of RAM, but I only actively/regularly access data stored in 10% of that 2GB allocated/consumed.  So one stat will tell you I'm using 2GB of RAM, whereas the other will tell you I'm only actively/regularly using 10% of it.

[Steven_Gray]

Aaron,

Thanks for the reply that kind of makes a bit more  sense. I see in 2008 it shows about 1GB cached but even still as you  pointed out Windows shows that as being "used". One thing I'm still  getting tripped up on is that even is you "subtract" the 1GB cache from  the 4GB that still wouldn't equal the 12% that vCenter is showing.

Is there something else in the background or another explanation that would show the difference?

[aaronwsmith]

And it probably never will.  As I said before, vCenter/ESXi does not have that level of insight into how the Guest OS is managing its memory.  Only thing the hypervisor knows is how much is granted to the VM (4GB), consumed (again 4GB), active (which is where I still think you're trying to make a square peg fit into a round hole with trying to associate memory consumed *within* the Guest OS to what ESXi knows is being actively used/accessed by the VM), etc.

Active memory will only show you what the hypervisor sees the VM is actively using, regardless of how much is consumed and regardless of how much the Guest OS shows is free/cached/etc.

[aaronwsmith]

These links might help if you read through them:

http://www.yellow-bricks.com/2010/12/20/vcenter-and-memory-metrics/

http://communities.vmware.com/docs/DOC-5600

The DOC-5600 states (which was also noted in this thread by someone else) "mem.usage.average = The percentage of memory used as a percent of all available machine memory."

Basically, Usage is just a percentage representation of active / granted.

That is, if you take the active stat and divide it by the granted stat, the % should be very close the "Usage" value you're focused on.  It's basically another way of showing you how much/little memory is really being used by the VM.

As noted on both links, Active memory is an estimated value by the VMkernel based on memory pages touched by the VM.  The fact that it's an estimated value enforces the the hypervisor does not have any insight to the Guest OS's memory manager.