esx4 U1 memory issue during vmotion

[ufo8mydog]

Hi there,

We have 4 ESX hosts in a cluster. Recently we needed to remove one of the ESX hosts out from the cluster and so the virtual machines were vmotioned off that machine. The problem is that they were vmotioned 4 at a time. What happened was that the cluster could not assign new ram quickly enough to the guests as they were vmotioned and they resorted to using swap; I/O became intolerably slow and they needed a hard reset to clear the swap.

With 3 hosts the 'Memory Granted' metric is around 45GB out of 48GB on each host, so close to capacity; but the Memory Active component is only 12GB meaning that the balloon driver should have easily inflated in other guests and reallocated the RAM to the new guest. I know that many people oversubscribe their environment so what gives here?

All ESX are on the latest patch level and so are vmtools on each VM.

[weinstein5]

Do you have any reservations set on the VMs?

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[ufo8mydog]

The reservations for memory are all set to 0 (which I believe means 256MB) since they are all equally important.

Should they be set higher?

[golddiggie]

When you view the properties of the VM (Edit Settings), under the Resources tab, what is the limit set to for the Memory setting? If the Allocation isn't set to normal shares, 0 reservation and the "Unlimited" box checked, try those settings. That should allow the VM to properly use the resources on the host, without grabbing too much in the reservation and giving you grief.

VCP4

[ufo8mydog]

Hi golddiggie,

The Reservation is 0MB and Limit is set to unlimited on all machines. Any other suggestions?

[ufo8mydog]

Does anyone else have some suggestions?

[ufo8mydog]

So I got off the phone to vmware, they claim if i set a minimum reservation for each VM (of say 1GB) then when they are vmotioned onto a crowded machine they will be guaranteed at least 1GB of real memory. Is this likely to actually happen? (Note I can't test it with the production virtual machines).

How does everyone else's clusters react when active memory is well below consumed memory, but consumed memory is oversubscribed?

[joshp]

When you set a memory reservation you are guaranteeing that amount of physical memory is going to be assigned/consumed to the VM. For example, if you set the VM memory limit to 1GB and you set the reservation to 1GB ESX will not create an ESX VM swapfile at machine boot (or during vMotion) because the VM will never need to swap out to disk through the hypervisor. So during the vMotion, the Virtual Machine with a reservation of 1GB will consume at minimum 1GB of physical memory on the destination host. Thus, if you are performing vMotions of 4 VMs each with 1GB memory reservations the ESX host will quickly need to come up with 4GB of physical memory (plus overhead).

VCP 3, 4

www.vstable.com

[ufo8mydog]

Thanks for the reply - so if I set up memory reservations, and during a vmotion event it needs to come up with the memory (because there are 4 going on) - but for some reason it cant do it quickly enough - does it then allocate swap to either the vmotioned VMs or to other VMs on that host?