[vSphere ESX 4.1] Windows Server 2008 R2 Memory Utilisation

[piercelynch]

Good afternoon,

I see a number of open threads regarding this topic, but not specifically related to vSphere 4.1.

As part of an implementation of an entirely fresh vSphere 4.1 deployment, a series of Windows Server 2008 R2 virtual machines have been deployed. However, it would appear that due to the feature of Windows Server 2008 'caching' available memory, vSphere isn't particuarly efficient with its resource usage at hypervisor level with regards to memory resources.

I see this as a major issue, as quite frankly the hypervisor not being unable to return what is simply unused memory from the guest virtual machines to the host for use by other guest VMs quite a bit of a constraint in the use of virtualisation.

Having read through a number of threads, discussions and blogs floating around the internet - I have come to no definitive resolution to this issue. I understand there was a known issue in 4.0 with TPS no neccessarily reporting the correct guest memory usage, but this appears to be marked as resolved in 4.1.

I have also tried setting the Mem.AllocGuestLargePage to 0 in the advanced settings of each host, but this has made no visible difference to the utilisation of the memory at host level.

In short, I am looking for others ideas and advice with regards to how potentially the memory of a vSphere host can be better managed when having multiple Windows Server 2008 R2 virtual machines.

Any input would be greatly apprecaited.

[AndreTheGiant]

If a guest OS is more aggressive on memory (like also Linux) I think that there isn't much to do.

TPS will not work on memory that is use to file caching, cause is different for each VM.

The hypervisor can only reclaim memory with the ballon driver of the VMware Tools.

Andre

[piercelynch]

Hi Andre,

Thanks for your response.

In this instance though, there is very little benefit in virtualising a Windows Server 2008 instance as the RAM utilisation doesn't really suit a virtualisation model.

Is there any way in which the 'caching' or prefretch of RAM in Windows Server 2008 R2 can be disabled so that it doesn't do this?

[AndreTheGiant]

You can try to change the last setting this page:

http://blogs.technet.com/b/markrussinovich/archive/2008/11/17/3155406.aspx

Your hosts have high memory utilization?

Because if they not have, it's all fine.

When memory goes high, then with the ballon driver more memory can be recaimed for the VMs, also the Windows 2008.

Andre

[piercelynch]

Hi Andre,

In this instance, its not a case that the VMs have high memory utilisation - thats kind of the issue. Problem lies, for example, with a newly installed Windows Server 2008R2 VM with 4GB of RAM allocated. This is a completely clean install, with no services running yet windows has 'available' 3.3GB with 2.8GB cached, and around 500MB in use.

So, in short - that's the issue Windows isn't really utilising this memory which could be used by another VM, and in my opinion kind of kills one of the key benefits of virtualisation.

Does that make sense?

[AndreTheGiant]

It make sense IF this resources are required or some other VMs really need them.

How many free RAM do you have on your host?

Andre

[RParker]

So, in short - that's the issue Windows isn't really utilising this memory which could be used by another VM, and in my opinion kind of kills one of the key benefits of virtualisation.

There is still sharing among the VM's. You are new to this, so give it some time. Don't expect everything to be perfect from day 1. With experience and understanding and a little patience will understand the nuances of how ESX works, so don't get upset because your perception isn't perfect.

Windows 2008 may use more memory, but maybe you assigned too much memory to Windows 2008 VM in the first place. If it's there, it will use it. That's what Windows 2008 is designed to do. It's called "cache". Its different than previous Windows OS.

Plus if your host isn't using the available RAM, then the VM's have no reason to start sharing and ballooning since RAM is NOT scarce. When ESX begins to notice that the RAM is being used, it will then apply those features, and memory management will be better. Until then, this isn't a shortcoming of VM Ware, as much as it is a Windows feature. You can't expect VM Ware to fix an OS, when that OS reports that it NEEDS all that RAM. That's not a virtualization flaw.

Maybe you should use Windows 2003 instead.

[golddiggie]

From what I've seen of Server 2008 R2 and how ESX/ESXi 4.1 handles the resources (the guest OS) demands, it's rather greedy. Even though the VM might report that it's only using nMB RAM when you open a console window on it, ESX allocates xMB to it. I have a few Server 2008 VM's, a mix of 2008 x64 and R2. The one that's running Exchange 2010 reports using more host memory than the others do at just over 3GB (all set to the minimum 4GB of RAM). My 2008 R2 AD DC (also doing DNS/DHCP duties) reports using under 1GB of host memory, with using less than 125MB of the Active Guest Memory. My vCenter Server (running 2008 x64) is using just under 2GB (also running VUM, with SQL on another VM).

Maybe an upcoming VMware Tools update will resolve the issue (could be in the works)... Or perhaps an update from Microsoft will resolve the issue. Or there could be settings that you need to make (some might do them as SOP that are not in your SOP list) that also helps the issue. It would be interesting to compare configurations to see if there's anything different between our setups...

I would suggest keeping an eye on the VM's Resource Allocation over just what you're seeing reported for numbers. I believe that will give you a more clear insight into what's going on... My Linux web server (OS is CentOS 5.5, running Gallery 2.x to show my digital photo's) uses just over 600MB (host memory) of the 1GB I've given it. Active Guest Memory is under 50MB though.

I would also watch the settings over time, not just at that moment. See what happens when you reboot the guest R2 server (perhaps after an update session has run)... I rebooted my host the other day, and saw the Exchange 2010 VM host memory utilization drop to about 1GB. Since I'm now running a few consoles within the VM, the usage has gone back up. Still, I'm not getting memory alarms, so it's all good... In fact, after the reboot, my total memory use on the host went from just under 15GB (I have 16GB total in the host) to just over 11GB, with the same VM's running.

In my experience, you'll run short on available host memory before anything else (provided you planned properly on the storage side). I've seen this in several environments, even when we thought we would be going over the top in host memory installed, it quickly becomes evident that it was a good idea to 'over spec' that resource. On a fresh environment, I've seen memory use on each host (three with 48GB each) to between 50% and 65% with just ~40 VM's on the hosts (total, balanced across the three)... CPU usage was typically in the 1-5% range (with one of the E5620 Xeon's being 'dormant' on each host)...

Biggest recommendation I can make to anyone looking to build up a new host... Get as much memory as you can... Take what you think will easily do the job, and then double it (at least)... Considering how we were running four hosts before the new environment with 16GB of RAM each (so ~64GB total) and using about 85%-90% of each host's RAM... We might have added a couple more VM's, but nothing extensive... In fact, I think we left a few off when we migrated them over to the new hosts...

Also, generally speaking, I don't bother to look at the VM's memory usage. I keep an eye on what ESX/ESXi is giving it to make sure I'm not heading for trouble. Mostly because I know that Windows can be a resource hog, no matter where it sits... Out of curiousity, do you install any apps to the C drive, or do you use a secondary volume/drive for that? What size do make the C drive during VM creation?

Network Administrator

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

[cmiller15]

I have yet to see a good answer for this.

What I want to know, has anybody found a way to keep Server 2008 R2 from allocating all available RAM, or is VMWare working on a way to better manage memory, maybe through VMWare Tools to show what is being used and to release RAM that the 2008 R2 guest would have used otherwise.

We are in an environment with vSphere 4.1, 14 hosts, and well over 250 guests.  2 of those hosts are newer Dell R710's that we are deploying Server 2008 R2 guests to.  The guests have SQL 2008 and require 4gb or 8gb of RAM.  Most of the time the guest shows only using 1-2gb of RAM, but because 2008 R2 eats the entire 4/8gb, we lose sharable memory and also limit the number of guests we can run on each host.

We ran into a situation with HA that we had more RAM allocated then what could be failed over.  Server 2008 R2 max memory utilization is the culprit.

Has anybody found a work around for this?  I realize over-commit is an "option", but with HA enabled on the cluster, I've not found a way to make that work.

[lhcrnetadmin]

I am interested in hearing about this as well.

My exchange server has 4Gigs of ram on it. It's consuming 4gigs on ram on my HOST but its showing my active Guest memory as 1.5Gig.

Also, inside my VM it is showing that the memory used is maxed out at 4gig. This causes issues with my Whatsup Gold reporting. I'm not sure if I need to lower the VM's ram since its only using 1.5gigs or increase ram since inside the VM shows its maxed out.

[admin_harting]

Hello cmiller15

We are also using such an evironment in that size. About 20 Hosts and 300 VMs. And we would like to have a solution for that issue. Maybe some of the vmware technician will read this.

I'm looking forward to get a solution...

Alex

[LucasAlbers]

I am very familiar with this sort of issue.

So as the previous posters mentioned windows is caching the file system aggressively.

The is dependant on the cache setting which is determined by the mode of the system.

So windows  has two modes that determine the working-set parameters of the system file cache.

The cacheset utility will allow you to look at this setting, and covers what windows is doing on this page.

http://technet.microsoft.com/en-us/sysinternals/bb897561

You can change it on a system to see the affect.

You can also get an equivalent affect by changing the mode from file server to application mode.

Another article about windows memory usage:

http://blogs.msdn.com/b/ntdebugging/archive/2007/11/27/too-much-cache.aspx?PageIndex=2

For sql server, hard code it to use less memory if you want it to use less memory.

Then just check your monitors to very your cache hits are good per sql server doc's.

Your other solution(s) is to use windows server 2008 r2 core, it seems happy with a very very low working set.

This other issue covers this item, (standby memory), and how to force tps.

http://communities.vmware.com/thread/259618

[Conz]

You can also get an equivalent affect by changing the mode from file server to application mode.

Where is this setting hidden ?

I might just be overlooking it as I'm running a localized install but my google skills are failing me as well on this one.