Consumed Host Memory is higher than Guest usage(in...

bradduane · ‎03-18-2014

I am running ESXi 5.1. I do not have any custom reservations configured for Memory. I am running in "Sandy Bridge" EVC mode.

I notice higher than expected consumed host memory on many of my guests. This example guest is running Server 2012 R2 with current VMware tools version running.

VMware reports

Guest OS reports

My understanding is that the "Standby" and "Modified" on the guest contains the "Cached" memory, and the guest "Cached" memory is the typical answer to this question that I see when people are onyl looking at the "In Use" memory. In my case, I am factoring in the cached memory, my guest shows 3805 MB of truely free memory, yet my host shows 7.68 GB of host memory consumed, with only 68MB of overhead consumption.By my calculations, the VM should only be consuming about 4.28GB of memory.

Message was edited by: bradduane Replaced text values with screenshots.

DavoudTeimouri · ‎03-18-2014

Hi,

ESXi has memory management and memory sharing and swapping techniques.

For example, ESXi resource scheduler can reclaim memory from a guest VM by using balloon driver that installed with VMware tools.

Or, the host can share memory pages between similar guest OSes by TPS.

For more information, please read this document: http://pubs.vmware.com/vsphere-51/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-51-resourc...

-------------------------------------------------------------------------------------
Davoud Teimouri - https://www.teimouri.net - Twitter: @davoud_teimouri Facebook: https://www.facebook.com/teimouri.net/

bradduane · ‎03-18-2014

I appreciate the extremely fast response, but I think it was made too hastily and without much detail. Yes I have looked through these documents. My understanding is that the balloon driver is used to swap some of the "least used" (but actually used) guest memory pages. This does not seem to apply here because this memory is actually "Free" on the guest, it should not need to be swapped. If the balloon driver also is there to allow a host to reclaim "Free" memory, then why is this memory not being reclaimed by the balloon driver if it has been consistently free for so long? The TPS feature also does not seem relevant here, because if anything, it would reduce the host memory usage, not explain why the host memory usage is actually so much higher.

jrmunday · ‎03-18-2014

Is this a SQL server by any chance? It's normal for SQL to "consume" all the available memory that has been allocated (unless configured by the DBA's).

Label	Description
Consumed	Actual consumption of physical memory that has been allocated to the virtual machine.
Overhead Consumption	Amount of consumed memory being used for virtualization purposes.

From a guest OS perspective;

Standby = Memory that contains cached data and code that is not actively in use

Modified = Memory whose contents must be written to disk before it can be used for any other purpose

As an example, my vCenter MSSQL server with 32GB RAM allocated has 31.16GB consumed where 11.2GB is in use, 15.6GB standby and 5.7GB free (figures rounded).

This looks like normal MSSQL behavior to me.

Cheers,

Jon

vExpert 2014 - 2022 | VCP6-DCV | http://www.jonmunday.net | @JonMunday77

DavoudTeimouri · ‎03-18-2014

TPS and ballooning were as example!

Just I mean, memory management is different between ESXi and guest OSes.ESXi using advanced memory sharing techniques.

Memory is consumed to your guest OS and guest OS has access to all memory addresses (In its opinion). But host used memory percent is the virtual machine memory capacity that it's using from physical memory.

Try to reserve all your VM's memory, then the values will be different with the current values.

-------------------------------------------------------------------------------------
Davoud Teimouri - https://www.teimouri.net - Twitter: @davoud_teimouri Facebook: https://www.facebook.com/teimouri.net/

bradduane · ‎03-18-2014

This is an Active Directory Domain Controller, so also a database, but a relatively small one.

What I don't understand, is that even in your example, if the guest OS shows 5.7GB of free memory, that is memory that SQL is not using, it can't be using it if the guest hasn't allocated it. And if SQL is not using it, the guest is not using it, then why would the host be using it?

Is this because the database is constantly cycling the "Free" memory" So this 5.7GB that was free 10 minutes ago is at a completely different memory location than the 5.7GB that is free right now, causing the host reclaimation of guest memory to not be able to keep up?

bradduane · ‎03-18-2014

I think I found my answer in the "Understanding Memory Resource Management" guide here: http://www.vmware.com/files/pdf/mem_mgmt_perf_vsphere5.pdf

Section "When to Reclaim Host Memory" page 15:

"In the high state, the aggregate virtual machine guest memory usage is smaller than the host memory size. Whether or not host memory is overcommitted, the hypervisor will not reclaim memory through ballooning or swapping. (This is true only when the virtual machine memory limit is not set.)

If host free memory drops towards the soft threshold, the hypervisor starts to reclaim memory using ballooning. Ballooning happens before free memory actually reaches the soft threshold because it takes time for the balloon driver to allocate and pin guest physical memory. Usually, the balloon driver is able to reclaim memory in a timely fashion so that the host free memory stays above the soft threshold."

I checked esxtop, pressing "m" for memory view, and I can see that the hosts VMKMEM shows "high state", so the only way memory is currently reclaimed is by page sharing. Only when I start to run out of host memory, this state changes to "soft" or "hard", will ballooning or swapping take place to free up the guest "free" memory. esxtop show my "minfree" as 940 MB, while it shows that I have 6960MB of free physically memory.

View solution in original post

bradduane · ‎03-19-2014

I have confirmed with VMware that this is the reason the memory wasn't being reclaimed. They provided me the following link that has a nice low-level diagram showing how memory is managed and when:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=201764...

Consumed Host Memory is higher than Guest usage(incl Cached) and overhead?