PTJC
Contributor
Contributor

High Host Memory Usage on some VMs

We have a handful of VMs which are using 97% of their allocated memory all the time, they are all 64bit Windows Server VMs and none of them are actually using very much memory from the guest size. Any ideas?

ESX 3.5 Update 2 Cluster of 4 Dell 2970s w/ Quad Core Opterons.

0 Kudos
11 Replies
RParker
Immortal
Immortal

It makes sense.

Windows 2008 and Vista take available memory and use most of it for cache. So your guest is using a small amount of memory but the AVAILABLE RAM is mostly consumed by the cache.

So if your OS is either of these, that's why the RAM is being consumed.

0 Kudos
PTJC
Contributor
Contributor

I apologize, I should have put which versions of Windows Server. The VMs in question are almost all Windows Server 2003 Enterprise x64 R2 SP1/SP2. The guest sees about 20% of its RAM as available, but Virtual Center shows a steady 95-98% Host Memory usage. There are no Reserves for these VMs and their Shares are all set to normal.

0 Kudos
RParker
Immortal
Immortal

Well if the system advanced properties are for background process and service priority then the same issue applies. It reserves more RAM for caching and services that run in the background. So it's pretty much the same thing still.

Changing these to foreground might fix it, I did notice a spike in memory for 64-bit systems also, but I figured it was the way 64-bit memory architecture worked. It's much more efficient, so I assumed that 'grabbing' the memory up front (which on a physical system isn't an issue) makes the OS more efficient over time.

0 Kudos
kjb007
Immortal
Immortal

Remember that host memory and guest memory are different. When a guest allocates memory , the host takes memory and gives it to the vm. As a guest uses and stops using that memory, it does not mark the memory as "no longer in use", so ESX can not reclaim it, so by default, the host memory usage will steadily go higher and higher, up to the max allocated to a vm, plus some overhead. This will remain that way until the vm is restarted, or if there is memory contention in the ESX host, at which time ballooning will kick in to reclaim some memory.

-KjB

VMware vExpert

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
0 Kudos
PTJC
Contributor
Contributor

I guess that I'm more concerned about the Guest Memory % figures, that is what is showing a high percentage. Let me give a better idea of what I'm looking at. Here are the top 25 VMs by Guest Memory %. The top 5 are my concern as they stay around 95%, the others listed are of comparable function and utilization yet they show pretty close to what the VM sees and there are 100 more VMs with significantly lower Guest Memory Percentages. These top 5 really are outliers and atypical to the environment. Is the Guest Memory % just not displaying correctly, or is there an issue with something else? Or is this just the balloon driver at work?

Host CPU - MHz

Host Mem - MB

Guest Mem - %

Guest OS

Memory Size - MB

67

2094

98%

Microsoft Windows Server 2003, Enterprise Edition (64-bit)

2048

19

1047

97%

Microsoft Windows Server 2003, Enterprise Edition (64-bit)

1024

171

2094

96%

Microsoft Windows Server 2008 (64-bit)

2048

29

2059

95%

Microsoft Windows Server 2003, Enterprise Edition (64-bit)

2048

155

2094

95%

Microsoft Windows Server 2003, Enterprise Edition (64-bit)

2048

29

2094

70%

Microsoft Windows Server 2003, Enterprise Edition (64-bit)

2048

66

2061

70%

Microsoft Windows Server 2003, Enterprise Edition (64-bit)

2048

2163

3763

61%

Microsoft Windows Server 2003, Enterprise Edition (64-bit)

4096

70

1867

57%

Microsoft Windows Server 2003, Enterprise Edition (64-bit)

2048

96

939

57%

Microsoft Windows Server 2003, Standard Edition (32-bit)

1024

419

7004

55%

Microsoft Windows Server 2003, Enterprise Edition (64-bit)

8192

26

2094

53%

Microsoft Windows Server 2003, Enterprise Edition (64-bit)

2048

252

1019

38%

Microsoft Windows XP Professional (32-bit)

1024

289

1597

33%

Microsoft Windows Server 2003, Enterprise Edition (64-bit)

2048

79

784

27%

Microsoft Windows Server 2003, Standard Edition (32-bit)

1024

878

343

25%

Microsoft Windows 2000 Server

768

234

1114

23%

Microsoft Windows Server 2003, Standard Edition (32-bit)

2048

212

1998

18%

Microsoft Windows Server 2003, Standard Edition (32-bit)

2048

92

1274

18%

Microsoft Windows Server 2003, Standard Edition (32-bit)

2048

132

2945

17%

Microsoft Windows Server 2003, Enterprise Edition (64-bit)

3072

85

1527

14%

Microsoft Windows Server 2003, Enterprise Edition (64-bit)

2048

81

1366

12%

Microsoft Windows Small Business Server 2003

2048

43

1024

11%

Microsoft Windows Server 2003, Enterprise Edition (64-bit)

2048

73

1614

10%

Microsoft Windows Server 2003, Standard Edition (32-bit)

2048

34

230

10%

Microsoft Windows Server 2003, Standard Edition (32-bit)

1024

0 Kudos
depping
Leadership
Leadership

I've seen the same in an AMD environment. You might want to test the following of a 64Bit VM: Set "virtualized MMU" to disabled and see what happens. it appears that the % is slightly off. And Large Pages are used by default, TPS doesn't do Large Pages which also means you will have a high memory usage compared to 32Bit VMs.

I wrote an article a while ago:

http://www.yellow-bricks.com/2009/03/06/virtualized-mmu-and-tp/

Duncan

VMware Communities User Moderator | VCP | VCDX

-


Blogging: http://www.yellow-bricks.com

Twitter:

If you find this information useful, please award points for "correct" or "helpful".

0 Kudos
PTJC
Contributor
Contributor

Nice article. I'll give it a try on one of the VMs and see how it goes. If that is what is going on we'll have to decide if 5-10% improved performance in select apps is worth the hit to our consolidation ratio.

Thanks again. I'll post my results (and assign points) next week.

0 Kudos
denz1217
Contributor
Contributor

Hi All,

First of all, I'm still a newbie with regards to vmware but I have the same scenario with PTJC. HOwever, on my side, all are linux (64-bit) VMs running on ESX 3.5 update 4. Basically, all of the guest memory usage of the linux-based VMs are always high (90% range). If I ran, "free" or "vmstat", it appears that the usage is really that much if we take the buffer+cache into consideration. The only problem with this, are the alerts that gets generated. I know I can always customized it to be in the 98%-99% threshold but what is bothering me a lot is that, I've seen another ESX (3.5 update 3) that has several linux-based VMs but I don't see the same high guest memory usage in the virtual center. This is true even if both "free" or "vmstat" shows that most of the memory were eaten because of the cache and buffer. Any thoughts please Smiley Happy

Thanks.

0 Kudos
denz1217
Contributor
Contributor

I'll be trying to disable the virtualiazed mmu and see if this would lower the guest memory usage. I'll be posting theresults. Thanks.

0 Kudos
depping
Leadership
Leadership

You could also disable Large Pages. check this topic:

http://communities.vmware.com/thread/211585?tstart=0

Duncan

VMware Communities User Moderator | VCP | VCDX

-


Blogging: http://www.yellow-bricks.com

Twitter:

If you find this information useful, please award points for "correct" or "helpful".

0 Kudos
denz1217
Contributor
Contributor

Hi Duncan,

Thanks so much for sharing the link. It was very useful and informative. Anyways, I tried disabling the virtualized mmu and it worked. Now, I have to decide if I'll go with disabling the large pages or having the mmu turned off on all the affected VMs. I still need to digest all the details that were mentioned. Though, it was suggested in the link that disabling the large pages would likely be the way because mmu should be always be enabled whenever possible. Thanks again.

0 Kudos