Memory Overcommitment Allocated vs. Active

[Vinsane]

I'm really slamming my head against the desk on this one, I've read tons of articles about this, but none quite get me there. Can anyone please debunk this once and for all for me?

1. We're attempting to overcommit memory in our 2 host cluster, call it HOST01 and HOST02.

1. Our hosts have 256GB physical memory available, and one of our hosts(HOST02) has over 96% of physical memory "allocated" to the running VMs. However, the amount of ACTIVE physical memory these VMs are using is VERY low in the < 20GB range.

1. We wanted to test over commitment and see what happened hoping the "memory states" of High Low Hard etc, were based on the amount of active physical memory actually being used.

1. Once we vMotioned 10 or so VMs to HOST2 and got it over 100% allocated, esxtop, 'm' started blowing up in SWAP /MB, ZIP /MB and MEMCTL/MB, I saw it go from high to soft to hard and back to high. VMs were also showing active swapping etc.

1. We've since balanced the cluster back out where the memory allocated is around 70% and it appears some items are still going through memory reclaim techniques.

I haven't been able to find a definitive answer on the memory reclaim techniques, based off what I am seeing, it's almost as if VMware is oblivious to the amount of Active Physical memory being used by the virtual machines running on the host.

Once the PMEM /MB value gets below a certain % of what seems to be ALLOCATED memory to VMs it appears the memory state flips, and memory reclaim techniques kick in and start swapping, compressing, ballooning etc. Why would it do this is the host truly has plenty of physical memory available for use?

Am I missing something? This seems inefficient, why if I have 256 Physical Memory can't I allocated say 500GB of memory to my VMs if my VMs are only truly using 20GB physical memory? I should be able to allocate as much as I want to the point of exhaustion where minimal physical memory is available THEN start memory reclaim techniques.

[jonathanp]

Are you using memory reservation on your VMs?

Jon

[Alistar]

Hello there,

there is a difference between the "Granted" and "Active" memory statistics.

• Active memory is the amount of memory your VM is actively touching at the moment
• Granted memory is the amount of memory you have configured for your VM.

Now, if you do not have any memory reservations set, the memory that not actively used by a VM can undergo memory reclamation techniques (Transparent Page Sharing (which is slowly diminishing because of the introduction of large pages unless forced by memory reclamation), Balooning, Compression and eventually swapping) in favor of other VMs or even System Worlds - ESXi still needs some free memory for itself, see my blog post: How much memory should be free for VMkernel? | VMXP (have you seen how windows behaves when it is out of memory?). Also if your VMs on the host are large, be aware that one big VM can consume a neat amount of overhead (http://stealthpuppy.com/performing-vsphere-vm-ram-overhead-lookups-in-excel/) as well.

If you post the VM config/memory used screenshot we can help you out a bit more

[Vinsane]

I'd like to clarify some more

1. VMware Tools are up to date and recent

2. I am running ESXI 5.5

3. VMWare Tools are running

4. There's not a single VM with any memory reservations. My understanding is the default in this case is 32Mhz CPU and  just the overhead required.

5. My VMs are definitely over sized in terms of memory allocated vs. memory active, however there's not a HUGE VM in there, they range from 4-8GB memory.

Like I said, my host has 256GB physical, VeeamOne reports the VMs on that host "active memory" is 20GB.

Before vMotion vSphere "Memory % was 96". I vMotioned 10VMs to this host to push it over 100% allocated, and I ran ESXTOP

The MEM overcommit avg was around 0.05 and the SWAP, ZIP and MEM CTL were all actively rising, the state went from high all the way to hard and back to HIGH after it ran through it's techniques.

Additional SWWR/s and SWW/s were very active for a number of VMs, in fact I am still seeing activity there even with the memory state being high.

[PhillyDubs · Accepted solution]

http://www.yellow-bricks.com/2015/02/17/vsphere-60-breaking-large-pages/http://

http://www.yellow-bricks.com/2015/03/02/what-happens-at-which-vsphere-memory-state/

From my experience, most of the "issue" is because of large pages. When I trick vSphere in to using small pages everything works wonderfully and I can cram many times the amount of virtual machines on the server.

[Vinsane]

Duncan was kind enough to answer this question as well for me:

https://labs.vmware.com/flings/vm-resource-and-availability-service#comment-532728