I guess I will answer my own question on this one. Just to rehash. I had a number of VMs, pretty much all of them, that were showing high Guest Mem % values. High being over 95%. The VMs themselves were not actually using the memory. The hosts were Vsphere 4 loaded on IBM 3650 M2 Nehalem machines. After asking my question I decided to open a call with VMware and also found a number of other threads where people were having the same issue.
Vmware had me make a change in the advance settings on each host. Goto advanced settings, then Mem, then look for Mem.AllocGuestLargePage and set it from 1 to 0. Move your VMs off the server before doing so then move them back and the probelm goes away after about 10 minutes or so. This is only a workaround until they have the patch in Update 1 sometime in the fall. It does turn off a feature and below is the explanation I received from the tech. Hopefully it will not hurt for the current machines I have on there now. I do plan to have large database servers with multiple CPUs and large amounts of RAM here in the future. Hopefully the patch will be out soon.
Here is the response I got from VMware
Hello,
After you make that change on all your hosts, and vmotion or power cycle your VMs, wait a few minutes, and you will see the memory performance graphs go to normal. Observe your performance of your VMs, to determine if you are seeing any performance hit, example of a VM that would use large memory pages, would be a SQL server that has 8GB or more of RAM assigned to it. Large workloads like this, would probably see a performance hit with large memory pages disabled.
Here is some technical info about large memory pages: Large Pages Computers manage memory in groups of bytes called pages. The x86 platform supports several page sizes, the most common being 4KB (small) pages and 2MB (large) pages. When an application accesses a large memory region frequently or accesses a significant amount of memory within the large region, it can be advantageous to work with large pages. This allows circuits in the CPU to access the memory faster. In a virtual environment, hypervisor support is needed to fully realize this performance boost . By default, ESX will use large pages for all memory that it manages when running on a platform with hardware nested page tables. This mitigates the extended TLB miss penalty that is possible on such hardware. Even when the hypervisor has backed a guest physical page with a large host machine page, configuring the guest to use large pages can offer further benefits and is always advised. Properly configuring the guest is essential for large page performance with hardware that does not implement RVI or EPT.
For such servers, vSphere defers to the guest operating system for hints as to which pages should be backed large. To see the value of vSphere's large page support, the virtual machine's operating system and applications were configured as described below prior to running the sales and distribution application load test1. The number of supported users was compared against an execution using the same virtual machine configuration but with large pages disabled using vSphere advanced configuration settings2.
* Windows: Use Group Policy Editor to give SAP process owner rights to "Lock pages in memory"
* MS SQL Server: Automatic if the virtual machine's memory size is ≥ 8GB * SAP: Profile should contain em/largepages=yes
Hope this helps someone else.
Thanks Rod.
