Should host 'consumed memory' equal the full amount of guest memory?

[LiamCurtis]

Hi All,

On ESX4, I have a predeployment box with two windows 2008 x64 guests. I gave one 4gb of mem and the other 8GB. After the vm's have booted, I see that the 'consumed memory' on the ESX host for each VM is the full amount I have allocated to the guest, and it never frees up memory thereafter.

I have read the guides on how memory works and I am still a bit confused. I thought that VMWware utilized an overcommit feature so that the full amount would not be used on the host, and that pages would be shared by default.

The host is a dl380 g6 with 2 x nehalem cpu's . I was reading somewhere else that there is some kind of bug with vmmu and tps /large pages with nehalem's, but not even sure if that applies here.

Any help greatly appreciated...

[MKguy]

Are both VMs running on the same host (In order to utilize the benefits of page sharing)?

Have you disabled the backing of large pages on the host? TPS won't work with large memory pages. Set Mem.AllocGuestLargePage to 0 and VMotion the VMs off and back to the host or cold boot them.

See http://communities.vmware.com/thread/211585 for details for the TPS issue.

Also, have you installed the VMware tools and how much actual memory usage does the Guest OS report?

[LiamCurtis]

Thanks for the reply MKGuy!

Both VM's are running on the same host. I have not set the Mem.AllocGuestLargePage to 0. I have installed VMWare tools.

The guests correctly report memory. On my VM with 4gb, approx 1024mb is being used by GOS. Active Guest memory in the vsphere client shows up as 1966mb, which is fair enough. I am concerned with the consumed host ('machine') memory using all of the memory at once, which is showing up at 4083MB...

My question is...is this normal behavior? This seems to negate one of the major selling points between VMWare and hyperV as all of the memory is being used at once for the guest and it does not seem to be released.

Or...does VMWare automatically start doing its magic as more VM's consume host memory...

[MKguy · Accepted solution]

I too am using DL 380 G6 Nehalem Servers with 64-bit 2008 guests and don't experience this issue. I have set Mem.AllocGuestLargePage to 0.

Usually, when I boot a Windows VM, the host memory usage of that VM is very high in the beginning (OS crawls through all of the memory during startup), but that gives way to a reasonable value within an hour or a bit more, so that the host memory consumed value in ESX very roughly equals what the GuestOS is reporting to be in use.

You can also check the esxtop memory view and take a look at the actual TPS values.

For example, one of my hosts currently running 23 VMs (mostly W2k3 32bit though) reports the following:

PSHARE/MB: 10788 shared, 2004 common: 8784 saving

The total provisioned memory to these VMs is about 26GB, but the actual memory usage of the host is only at about 19GB (this includes the service console and vmkernel memory too).

If TPS doesn't work for you, like it did for me when i had Mem.AllocGuestLargePage set to 1, you should see pretty much zero shared memory in esxtop.

I would advise you to set Mem.AllocGuestLargePage to 0 and see if that resolves the issue (VMotioning the VMs off and back etc). You can still set it back to 1 without issues later on, you don't even need to reboot the host if I remember correctly.

[LiamCurtis]

Holy smokes...that was it! Setting Mem.AllocGuestLargePage to 0 did the trick! Now I am seeing host consumption of only 615MB and as low as 0MB on the Guest Memory! This is what I was expecting to see. Now the trick is for me to figure out how TPS and vMMU really affect me and my environment (and what the heck they truly mean)

I owe you a beer....!

Thanks again for your reply and your help.