High Host Memory Usage on some VMs

[PTJC]

We have a handful of VMs which are using 97% of their allocated memory all the time, they are all 64bit Windows Server VMs and none of them are actually using very much memory from the guest size. Any ideas?

ESX 3.5 Update 2 Cluster of 4 Dell 2970s w/ Quad Core Opterons.

[RParker]

It makes sense.

Windows 2008 and Vista take available memory and use most of it for cache. So your guest is using a small amount of memory but the AVAILABLE RAM is mostly consumed by the cache.

So if your OS is either of these, that's why the RAM is being consumed.

[PTJC]

I apologize, I should have put which versions of Windows Server. The VMs in question are almost all Windows Server 2003 Enterprise x64 R2 SP1/SP2. The guest sees about 20% of its RAM as available, but Virtual Center shows a steady 95-98% Host Memory usage. There are no Reserves for these VMs and their Shares are all set to normal.

[RParker]

Well if the system advanced properties are for background process and service priority then the same issue applies. It reserves more RAM for caching and services that run in the background. So it's pretty much the same thing still.

Changing these to foreground might fix it, I did notice a spike in memory for 64-bit systems also, but I figured it was the way 64-bit memory architecture worked. It's much more efficient, so I assumed that 'grabbing' the memory up front (which on a physical system isn't an issue) makes the OS more efficient over time.

[kjb007]

Remember that host memory and guest memory are different. When a guest allocates memory , the host takes memory and gives it to the vm. As a guest uses and stops using that memory, it does not mark the memory as "no longer in use", so ESX can not reclaim it, so by default, the host memory usage will steadily go higher and higher, up to the max allocated to a vm, plus some overhead. This will remain that way until the vm is restarted, or if there is memory contention in the ESX host, at which time ballooning will kick in to reclaim some memory.

-KjB

VMware vExpert

[PTJC]

I guess that I'm more concerned about the Guest Memory % figures, that is what is showing a high percentage. Let me give a better idea of what I'm looking at. Here are the top 25 VMs by Guest Memory %. The top 5 are my concern as they stay around 95%, the others listed are of comparable function and utilization yet they show pretty close to what the VM sees and there are 100 more VMs with significantly lower Guest Memory Percentages. These top 5 really are outliers and atypical to the environment. Is the Guest Memory % just not displaying correctly, or is there an issue with something else? Or is this just the balloon driver at work?

Host CPU - MHz	Host Mem - MB	Guest Mem - %	Guest OS	Memory Size - MB
67	2094	98%	Microsoft Windows Server 2003, Enterprise Edition (64-bit)	2048
19	1047	97%	Microsoft Windows Server 2003, Enterprise Edition (64-bit)	1024
171	2094	96%	Microsoft Windows Server 2008 (64-bit)	2048
29	2059	95%	Microsoft Windows Server 2003, Enterprise Edition (64-bit)	2048
155	2094	95%	Microsoft Windows Server 2003, Enterprise Edition (64-bit)	2048
29	2094	70%	Microsoft Windows Server 2003, Enterprise Edition (64-bit)	2048
66	2061	70%	Microsoft Windows Server 2003, Enterprise Edition (64-bit)	2048
2163	3763	61%	Microsoft Windows Server 2003, Enterprise Edition (64-bit)	4096
70	1867	57%	Microsoft Windows Server 2003, Enterprise Edition (64-bit)	2048
96	939	57%	Microsoft Windows Server 2003, Standard Edition (32-bit)	1024
419	7004	55%	Microsoft Windows Server 2003, Enterprise Edition (64-bit)	8192
26	2094	53%	Microsoft Windows Server 2003, Enterprise Edition (64-bit)	2048
252	1019	38%	Microsoft Windows XP Professional (32-bit)	1024
289	1597	33%	Microsoft Windows Server 2003, Enterprise Edition (64-bit)	2048
79	784	27%	Microsoft Windows Server 2003, Standard Edition (32-bit)	1024
878	343	25%	Microsoft Windows 2000 Server	768
234	1114	23%	Microsoft Windows Server 2003, Standard Edition (32-bit)	2048
212	1998	18%	Microsoft Windows Server 2003, Standard Edition (32-bit)	2048
92	1274	18%	Microsoft Windows Server 2003, Standard Edition (32-bit)	2048
132	2945	17%	Microsoft Windows Server 2003, Enterprise Edition (64-bit)	3072
85	1527	14%	Microsoft Windows Server 2003, Enterprise Edition (64-bit)	2048
81	1366	12%	Microsoft Windows Small Business Server 2003	2048
43	1024	11%	Microsoft Windows Server 2003, Enterprise Edition (64-bit)	2048
73	1614	10%	Microsoft Windows Server 2003, Standard Edition (32-bit)	2048
34	230	10%	Microsoft Windows Server 2003, Standard Edition (32-bit)	1024

[depping]

I've seen the same in an AMD environment. You might want to test the following of a 64Bit VM: Set "virtualized MMU" to disabled and see what happens. it appears that the % is slightly off. And Large Pages are used by default, TPS doesn't do Large Pages which also means you will have a high memory usage compared to 32Bit VMs.

I wrote an article a while ago:

http://www.yellow-bricks.com/2009/03/06/virtualized-mmu-and-tp/

Duncan

VMware Communities User Moderator | VCP | VCDX

-

---

Blogging: http://www.yellow-bricks.com

Twitter:

If you find this information useful, please award points for "correct" or "helpful".

[PTJC]

Nice article. I'll give it a try on one of the VMs and see how it goes. If that is what is going on we'll have to decide if 5-10% improved performance in select apps is worth the hit to our consolidation ratio.

Thanks again. I'll post my results (and assign points) next week.

[denz1217]

Hi All,

First of all, I'm still a newbie with regards to vmware but I have the same scenario with PTJC. HOwever, on my side, all are linux (64-bit) VMs running on ESX 3.5 update 4. Basically, all of the guest memory usage of the linux-based VMs are always high (90% range). If I ran, "free" or "vmstat", it appears that the usage is really that much if we take the buffer+cache into consideration. The only problem with this, are the alerts that gets generated. I know I can always customized it to be in the 98%-99% threshold but what is bothering me a lot is that, I've seen another ESX (3.5 update 3) that has several linux-based VMs but I don't see the same high guest memory usage in the virtual center. This is true even if both "free" or "vmstat" shows that most of the memory were eaten because of the cache and buffer. Any thoughts please

Thanks.

[denz1217]

I'll be trying to disable the virtualiazed mmu and see if this would lower the guest memory usage. I'll be posting theresults. Thanks.

[depping]

You could also disable Large Pages. check this topic:

http://communities.vmware.com/thread/211585?tstart=0

Duncan

VMware Communities User Moderator | VCP | VCDX

-

---

Blogging: http://www.yellow-bricks.com

Twitter:

If you find this information useful, please award points for "correct" or "helpful".

[denz1217]

Hi Duncan,

Thanks so much for sharing the link. It was very useful and informative. Anyways, I tried disabling the virtualized mmu and it worked. Now, I have to decide if I'll go with disabling the large pages or having the mmu turned off on all the affected VMs. I still need to digest all the details that were mentioned. Though, it was suggested in the link that disabling the large pages would likely be the way because mmu should be always be enabled whenever possible. Thanks again.