All VMs High Consumed Memory?

[AndyR8939]

Hi All,

Have been looking into memory consumption on our ESXi infrastructure as need to add a few more VMs but was unsure of the amount of RAM free.

We currently have 3 ESXi hosts with a total of 126Gb ram between them and running 34 VMs totaling 110.5Gb ram.  I was looking to see if I could reduce the ram assigned to any of the VMs so i did some reading up and it seemed that I could look at the Consumed Memory for VMs and see if they actually need all of it.  IE, I had a VM with 4gb assigned to it, but consumed never goes above 1.5gb, so technically I could reduce it to 2gb of Ram?  Hope that makes sense?

Anyway, I had a look at all my VMs and not sure why but nearly every VM is running with consumed near enough the exact amount of RAM thats assigned?  is this right?  I can see one VM which drops to 1.8gb consumed of 2gb assigned but thats about it?  Am I doing something wrong with the config of them to have them all using this much consumed ram?

I've included a couple of screen grabs showing current usage and a graph of the last week consumed vs active.

Thanks guys,

Andy

[AndyR8939]

forgot to say, the VMs are a mixture of Windows 2003 Server and RHEL 5.5.

[AureusStone]

Hi Andy.  You are not doing anything wrong.

Consumed memory will show you how much memory has been allocated to the VM.  Because your memory isn't overcommited there is no reason for vShere to allocate less memory.  If there was a sudden spike of activity on one of your servers, your guest would not need to request more memory.

Active is going to tell you how much memory is being used by the guest.  Also note that every VM also has a memory overhead.  You can add that counter to see your overhead usage.

So if you have a VM with 2gb of ram, with 2gb active and 100mb of overhead the VM is actually using 2100MB of host memory.

It is always a good idea to not allocate more memory then required.  You can save some memory if you decrease the memory allocation to any guests which are not using them.

[AndyR8939]

Thanks.

What about the hosts then showing up as 90% memory being used?  As my VMs are basically not using all the memory (2gb consumed, 300mb active for example) if I moved more VMs to a host and it exceeded 100% memory on the host, would the host then start TPS and reduce the consumed memory?

Reason I'm asking is I have a host with a hardware error which i need to take down to repalce a part, and I was hoping to vmotion VMs to it, but not sure if I do it will kill the host if it exceeds 100% memory?

Thanks,

Andy

[AureusStone]

Click on your virtual cluster and then click on the host view.  Have a look at the % Memory column.

If the average usage is below around 60% you should be fine.  At 60% if you take one host down the other hosts should increase to about 90% utilisation.  Looking at how much memory you have, and how much you are using you should be fine.

TPS is not used for reclaiming memory from the guest.  It's role is to reduce memory consumption.  It does this by creating a hash key for every memory page on a host.  When it finds duplicate pages, it removes them and references back to the one page.  TPS is a core feature of vSphere and it is constantly running in the background by default.

[AndyR8939]

Thanks.

The reason I mentioned TPS is because I've using Nethalm CPUs which by default on ESX4.1 didn't enable small pages by default, I think on large pages?

Anyway, think I'm half way there as I read another post where someone has a similar problem and they were advised by Vmware that once the host memory hits 100% it will enable TPS small pages.  This user saw a drop from 90% to 60% by doing this so I gave it a try.  Ended up getting an extra 16gb of VMs on the host and took a host memory drop by 2gb.  Most of VMs are now showing half the amount of host memory consumed.  Shared memory has gone through the roof so looks like its kicked in alright for now.

Thanks guys.

[benq201110141]

Hi Andy,

I'm also trying to understand the consumed memory usage (being so much above active memory usage).

can you please elaborate on your last post about small pages.  what needs to be done on the cluster/hosts for this setting to take effect?

I have 3 hosts in the cluster with all 3 hosts usging around 90% (% Memory om the hosts tab of the cluster).  we've noted that when we fire up a handful of vms simultaneously, the host mem usage shoots to 97% for all 3 hosts, and consequently a number of existing vm connections are dropped.

so basically we're trying to acheive what you have mentioned in your last post, whereby reducing the amount of 'consumed memory' to more accurately reflect the 'actual memory' usage of the vm.

cheers

ben

[CaptainJF]

For disabling large page, you have to do this on your host

configuration / advanced setting / mem

set Mem.AllocGuestLargePage to 0

[benq201110141]

Thanks CaptainJF.

Sorry, I should have been more clear as to what the intent of my question was, and that is, is disabling large page a valid fix for this issue of consumed memory being higher than what the active memory on the vm is?

i read another thread where one respondent suggested, it would be normal behaviour for the consumed memory to be as high or close to the assigned memory for the vm, -IF- there was no memory contention taking place ie enough physical RAM on the hosts to satisfy the assigned memory of all vm's.

but in my case, I have about half the physical RAM to ram assigned for vm's, so i gather the consumed memory shouldnt be having like it is

our environment is:

3x ESX 4.1.0, 260247 (HP DL380 G7)

total memory available in cluster =  79.96GB

total memory assigned to vms (limit) = 150.87GB

total Consumed memory (snapshot usage of vms) = 65.67GB -- (so my 3 hosts are currently sitting at 85%, 92%, 82% %memory usage)

total Active memory (snapshot usage of vms) = 17.18GB

suggestions?

[AndyR8939]

Hi Benq.

The problem we had was we had was our 3 hosts had a total of 126gb ram between them, and all our VMs were totaling 110gb assigned to them.  The 3 hosts were therefor showing around 90% memory usage all the time, but active usage was way lower than that.   I also noticed the on the VM section of our cluster, that all VMs had there host mem basically matching the assigned memory.

In previous setups this host mem section has generally been much lower than assigned memory because of the memory sharing ESX does, so doing this digging is when I found out about small/large paging.

Because we have Nethlam CPUs in the host servers, by default ESXi 4.1 was using large memory pages, where in the past the default was always small pages.  This explained why we werent getting as much memory sharing going on.

The suggestions were to either put that key in to force small pages, or to put a host over 100% consumed memory at which point it forces it into small pages mode.  I did this by moving some extra VMs and once I did the host which had previously been running at 90% went down to 70%, even though I'd moved and extra 8gb of VMs onto it.  When I move them back, after about 30 minutes the host mem goes back up to 90% as large pages comes back in.

I beleive this is corrected in ESXi 4.1 update 1 where small pages is on by default so we are planning to upgrade to this soon.

Hope that helps?

Andy

[benq201110141]

Hi  Andy,

thanks for the info.  Re the ESXi 4.1 update 1, i could only find reference to a fix for the swap file page size, nothing about small/large paging for TPS purposes. (http://www.vmware.com/support/vsphere4/doc/vsp_esx41_u1_rel_notes.html#resolvedissuesvmotion)

could you point out the link to the referenced fix if you are aware of where it might be stated?

cheers

Ben

[AndyR8939]

Hi benq,

Sorry think I've got my wires crossed between two issues we have been having.

We've also got a problem with some of our NICs not functioning correctly when enabling vlans in ESXi and VM Support advised us to upgrade to 4.1 update 1 as this was addressed in that, so I think I've got confused with that sorry.

The original thread I was reading about TPS was - http://communities.vmware.com/thread/211585?start=0&tstart=0

In there were a few VMware support guys who suggested they were looking at resolving the TPS issue in the next release, but not sure if this happened or not.  There is also a blog post - http://blogs.vmware.com/uptime/2011/01/new-hardware-can-affect-tps.html - explaining the issue and VMware stance on it.

Hope that helps,

Andy