Guest RAM allocation on boot

[jbohlke]

Back when I had ESX servers that were memory constrained there where times when I could not start a VM because there was not enough free memory.  From what I understood this was due to the fact that ESX had to reserve the full memory size of the VM on boot until the OS was loaded.  I was doing some testing with the VirtualCenter appliance on my ESXi 5.1 server that has 4G of RAM.  The appliance needed 8G of RAM to work so I set it for 8 assuming it would not start.  I was very surprised when it successfully started and even more surprised when it reported 7G of active RAM.  Has something changed in the guest boot RAM allocation process that allows machines to start even when there is not enough available RAM?

[elgreco81]

Hi,

Windows has a security messure that zeroes out all memory pages, making ESXi to allocate all memory to that windows machine. However, this memory can be "shared" in the way that most of those pages are now equal and Transparent Page Sharing can consolidate all that equal memory into a few pages of physical RAM.

Now, I know you are not talking about a Windows machine, but I wanted to use the example to explain the point.

7 GB of active RAM is really a lot of RAM to be active. Normally, active RAM is quite low. Active RAM is the memory pages being readen in any given moment so normally is a copule of hundred MB or less.

Allocated is not necessary active memory.

HA admission control let's you choose if you can power on or not, VMs that violate your resource cluster policy. Maybe that was what happened before (just guessing here).

Other things to consider regarding VMs is "shares". If contention occurs and you are short on memory, memory will be given to the machine with more shares in a certain percentage (that will depend on total number of shares and the number assigned to that specific VM)

Then you also have things like compression, swapping and balloning that will most likely affect your enviroment performance (not always balloning or swapping means a performance issue) that can also help you to overcommit and "overuse" your RAM by unloading some pages to disk or by compressing the pages used.

I suppose that not all ESX versions had all these capabilities, but memory overcommit is being around for a long time now. If I had to guess, I would say that is more a thing of admission control than memory contrains.

Anyway, hope my explanation helped you!

Regards,

elgreco81

[sparrowangelste]

the vm is using swap space to run. invisible to the vm.

there is also Memory Compression:

Memory Compression: If the virtual machine’s memory usage approaches the level at which
host-level swapping will be required, ESXi will use memory compression to reduce the number of
memory pages it will need to swap out. Because the decompression latency is much smaller than the
swap-in latency, compressing memory pages has significantly less impact on performance than
swapping out those pages.

Regular Swapping: If the host cache becomes full, or if a host cache has not been configured, ESXi
will next reclaim memory from the virtual machine by swapping out pages to a regular swap file.
Like swap to host cache, some of the pages ESXi swaps out might be active. Unlike swap to host
cache, however, this mechanism can cause virtual machine performance to degrade significantly due
to its high access latency.

[jbohlke]

I understand memory sharing when the VM is already online, in previous ESX versions you had to have at least enough free RAM resources (the same or larger than the RAM granted to the VM you want to power on) to power on the VM regardless of what OS it was running, I think this was prior to admission control?  So is it possible that TPS is handling this at boot time as well, or is this requirement replaced by admission control now?

[elgreco81]

Hi,

I don't have much experience with older versions of ESX/ESXi but from this kb, I can see that Admission control has been around at least since 3.5.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100700...

TPS will hardly manage that at boot time...if I watch stadistics in a View enviroment, when users are powering on the VMs I can see a high use in memory and then a progressive decrease. Checking shared memory, I can see that growing. So, from my opinion, TPS still needs some time to "see" and process the memory pages.

Regards,

elgreco81

[jbohlke]

This is what I was expecting:

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&e...

But I did not see that error, is granted RAM reserved at boot time on ESXi 5.1?

[elgreco81]

Hi,

For what I see that KB talks about a specific situation where your VM's reserved memory has a bigger value than your memory limit for that VM.

The answer to your last question, if I had to guess I would say NO. If granted memory were reserved at boot time, we would always see the error described in the KB you referenced unless memory limit => memory granted.

Regards,

elgreco81

[jbohlke]

I guess that is my point, I am pretty sure this was the case to ensure that the VM could boot in older ESX versions.