Aristizabal
Enthusiast
Enthusiast

How is the ESX server reclaiming memory on idle VM?

Hello,

I am currently monitoring the bandwidth consumption for vMotion operations,  and I noticed that it actually only transfers the memory in use for any particular VM. Then I was surprised to see that the hypervisor (ESXi 4.1) can determine that a VM is idle so its memory consumption is less, therefore if I run vMotion less traffic is transmitted.

Looking at the VMware resource management guide I found that since the Hypervisor is unaware of freed pages on the Guest OS, it will use the balloon driver to reclaim memory whenever the resource is on high demand. In my particular case I have an ESXi server with plenty of available memory, more than 50 % is not in use; so after running a process on a Windows 2003 VM that consumes almost all the memory, I can see the private memory on the VM growing from the vSphere Client, and also from resxtop (SZTGT).

After a while the VM becomes idle and the VM private memory decreases, as well as the SZTGT value on resxtop. I wanted to confirm if the balloon driver was doing this, but from esxtop I cannot see any activity for the memctl MCTLSZ/MCTLTGT counters; the only counter that is not in 0 is MCTLMAX, which I believe is the maximum value the driver can claim.

The other mechanism that could be reclaiming the memory is the idle memory tax, but honestly I don't understand how the Hypervisor can decide what memory pages are not in use by the Guest OS.

A related topic is discussed on the following link:

http://www.yellow-bricks.com/2010/03/03/cpumem-reservation-behaviour/

But it is not clear whats effectively reclaiming the memory on an idle VM.  I appreciate any help on the matter.

Regards,

Juan.

Tags (1)
0 Kudos
15 Replies
sflanders
Commander
Commander

The balloon drive only kicks in when a host is resource constraint. As such, if the host has plenty of available memory and the single VM becomes idle, the balloon driver is not the device freeing up memory. In addition, please be advised the balloon driver only factors in if you have VMware Tools installed (just an FYI as I did not see that mentioned in the post). As for what is freeing up memory when the VM becomes idle, the answer would likely be the VM itself. When the VM no longer needs memory pages it frees them. In the case of something that caches memory (e.g. database or web server) even if the VM is idle memory may show being mostly used.

Hope this helps! === If you find this information useful, please award points for "correct" or "helpful". ===
0 Kudos
Aristizabal
Enthusiast
Enthusiast

Thanks for the reply Steve.

In my case I have VMware Tools installed, and what you mentioned about the balloon driver makes perfect sense; but what is kind of confusing to me is that after the VM becomes idle and frees pages, the Hypervisor somehow knows those free pages are not in used by the Guest OS, since when I vMotion the VM only the memory in use by it is transferred between the ESX servers.

As long as I understand, the Hypervisor wouldn't know about the freed pages on the Guest OS and  vMotion would end up transferring the entire memory of the VM.

0 Kudos
sflanders
Commander
Commander

Why do you think the hypervisor would not know? The hypervisor gives the memory resources to the VM. When the VM is done with them it will release them and the hypervisor would be aware of the freed memory. The balloon drive reclaims memory (i.e. forces VM to give up memory pages), but the VM can willing release un-needed memory pages and the hypervisor would know of this as it is the resource distribution engine.

Hope this helps! === If you find this information useful, please award points for "correct" or "helpful". ===
0 Kudos
Aristizabal
Enthusiast
Enthusiast

My conjecture is based on the document "Understanding Memory Resource Management in VMware ESX 4.1" from VMware, page 4:

"Virtual machine memory deallocation acts just like an operating system, such that the guest operating system frees a piece of physical memory by adding these memory page numbers to the guest free list, but the data of the “freed” memory may not be modified at all. As a result, when a particular piece of guest physical memory is freed, the mapped host physical memory will usually not change its state and only the guest free list will be changed."

And also from the following article:

http://www.yellow-bricks.com/2010/03/03/cpumem-reservation-behaviour/

Where it is discussed how the memory deallocation works.

0 Kudos
mittim12
Immortal
Immortal

0 Kudos
Aristizabal
Enthusiast
Enthusiast

I already read this article, but it is never stated that the memory tax for idle VMs will use the balloon driver or swapping, and as it was mentioned by Steve on this thread these mechanisms are only used when resources are limited on the ESX, which is not my case.

Although the article is interesting, some of the questions asked on the comments section are not addressed, like:

"Duncan, i’ve been reading up on memory allocation and I still feel something is missing on the deallocation subject.

Since deallocation can only happen with TPS, Ballooning and Host swapping (according to VMware) how can my VM memory usage drop from 75% usage to 25% without ballooning (stats min/avg/max 0/0/0)?"

0 Kudos
mittim12
Immortal
Immortal

The below was taking from that article.

To summarize; How does ESX reclaim idle memory or free memory from a virtual machine? The answer is simple. ESX has two idle memory reclamation mechanisms:

  1. Balloon driver
  2. vSwap

The ESX host shouldn't reclaim memory until there is contention on the host and when there is contention the balloon driver will kick in.   Also see Jason Boche's blog on idle memory tax.

http://www.boche.net/blog/index.php/2009/01/29/idle-memory-tax/

Aristizabal
Enthusiast
Enthusiast

Thanks, interesting article.

My problem is that after the VM memory consumption ramps up to the maximum, it slowly goes down. Then when I vMotion the VM the ESX server only transfers the active memory, so if my server has plenty of memory resources and I haven't seen any ballooning activity on the VM (not reported on the memctl counters from resxtop) how is the Hypervisor aware of the memory that has been freed inside the guest OS?.

0 Kudos
mittim12
Immortal
Immortal

Well  I'm not sure really how to answer that.  I know the active memory counter is an estimated  counter and I was always under the idea that the vmkernel had no true idea of the guest memory management.

0 Kudos
Aristizabal
Enthusiast
Enthusiast

The way I increase the memory consumption on the VM is by using program that zeroes  out all the unused memory on the guest. Could it be that transparent page sharing TPS detects that the zeroed memory is repeated and it gradually moves the private memory of the VM to the shared space, so it optimizes the memory consumption?

Now, I noticed that when I vMotion the VM the amount of traffic generated is very close to what the private memory consumption is reported on the vSphere client and to the esxtop counter SZTGT. So vMotion is presumably only transferring the private memory of the VM plus the shared memory, that happens to be all zeroes. Is it possible that vMotion is aware of the TPS optimization?

0 Kudos
AureusStone
Expert
Expert

Yeah basically.  If you use all 0's TPS will compress that down to one 4kB block.

Perhaps you can use random data instead.

If there is no memory contention vSphere will generally not reclaim memory.  It is more efficient to have the memory assigned to the guest incase there is a sudden burst of activity.

0 Kudos
Aristizabal
Enthusiast
Enthusiast

Using random data is a good idea to prove that TPS is in fact compressing the memory, but I would like know if vMotion is aware of the compressed memory so it only transfers the private memory plus the compressed version of the shared memory.

0 Kudos
AureusStone
Expert
Expert

vMotion is not aware.

When the memory is compressed it does not keep a copy of the uncompressed memory.  If it did that it would not be saving memory.

TPS works by getting a hash value for all of the memory pages on an ESX host.  Then it compares them and finds any duplicates.  It then removes the duplicates and references the one memory page.

Aristizabal
Enthusiast
Enthusiast

Thanks for the clarification.

In fact, when I used random data TPS couldn't save much and I ended up transferring all the VM memory when vMotion was issued.

0 Kudos
rickardnobel
Champion
Champion

Aristizabal wrote:

how is the Hypervisor aware of the memory that has been freed inside the guest OS?.

I would say that it can not be aware of this in other ways than "pushing the OS somewhat" by using the balloon driver. However, as for the internal guest memory - when Windows closes an application the pages will be put on the "free list" and will not be visible to the Vmkernel as free. But, sooner or later the Windows memory manager does erase the data inside the free list and put the pages to the "zero list".

Since the memory now is erased, i.e. just zeros, it will be spotted by the TPS and reclaimed by the vmkernel.

My VMware blog: www.rickardnobel.se
0 Kudos