VM memory ballooning and swapping

[AartK]

Hi,

I don't know if the following situation is normal.

2 days ago several vm's began swapping memory because an admin powered up a beefy test VM. So the Esxi overcomitted the ram 45000MB/32700MB to serve this VM's.

This testmachine has been powered off and now the The ESXi memory usage is: 23400MB/32700MB.

So enough memory left to feed the other VM's.

But when I check the "resource allocation" tab, I still see the VM's ballooning and swapping.

This is resolved when I reboot the VM.(Vmware tools checking memory usage at start up, anyone can confirm?)

Should I restart all vm's that are swapping memory?

PIC1: VM with memory swapping

PIC2: Same VM after restart (adjust mem size from 2048 towards 1024)

thx,

AartK

[Silverchenau]

What's the problem? I see 0 usage on both ballooning and swapping after you reboot your machine.

My Vmware blog: http://geeksilverblog.com

[AartK]

Yes but some machines are in production environment and they are still swapping memory. (cannot reboot till the weekend)

So is there any other way around besides a reboot?

I thought vmwaretools could handle this situation without a reboot.

[Silverchenau]

hmm, this is interesting. As far as I can recall, the swapping and ballooning memory can be automatically swapping back once physical memory is available. It has nothing to do with vm OS, it's just vmkernel job.

Please give me esxtop screen shots and see how many vms are using ballooning?

If you don't know how to do it, pls follow http://geeksilver.wordpress.com/2010/05/17/esxtop-my-understanding/

My Vmware blog: http://geeksilverblog.com

[AartK]

Hi,

I already rebooted about 6 vm's last night. There are still a few vm's left with ballooning and memory swapping.

I just ran esxtop against 2 esx hosts.

[Silverchenau]

Looks like you still have 11GB memory free.

It think it's issue of your resource pool and memory limitation.

Move your ballooning VMs to root level of your resource pool.

Right click VM and check out Options for memory limitation. Did you setup anything over there?

Can you have snapshots for your VM memory limitation setting here?

My Vmware blog: http://geeksilverblog.com

[AartK]

Indeed, there is still physical memory left on the ESX hosts.

Let me explain the situation.

This virtual environment is not under my control, I've been asked to check the virtual environment since the admins have limited knownledge of the vmware environment.(not saying I have the knowlegde :smileysilly:)

Anyway its a cluster with 2 esx 4.0 hosts with vmware Enterprise license.

Outside the cluster there is vmware esx 3.5 standard, *Esxi 4.0 evaluation and a esxi 3.5.

They don't fully benefit the Enterpise license.

At the moment they use HA,vmotion, svmotion and vmware DRS is set fully automated (aggressive) on the cluster niveau.

They didn't made resource pools.

The vm's don't have limitations set (mem&cpu) and have no reservations set either (mem&cpu).

So the vmware settings are kinda default settings.

shares = normal for all vm's.

esx host memory for service console = 500MB

esx system resource allocation = default settings

[Silverchenau]

Well, that's not what ESXTOP said. You can clearly see some of your machines has 2GB, but only granted with 1.5GB or even less. That's why it started to use ballooning and swapping.

I am not fully understand how this would happen if there is no setting on memory limitation and resource pool.

It's just doesn't make sense to me.

I strongly recommend to post screenshots of one of your ballooning VM so I can see by myself just in case something is missing.

My Vmware blog: http://geeksilverblog.com

[AartK]

This is GID 465 on esx03.

I attached screenshot from GID 465 VM (ws03-game).

Can it be that the admins set a memory limit and removed this limit after the problems and this limit is still applying towards the vm's?

I don't see another option besides a bug.

[Silverchenau]

please connect your ESX with ssh

open that vm.vmx file

looking for sched.mem.max. Make sure it's your memory size of that vm.

My Vmware blog: http://geeksilverblog.com

[AartK]

sched.cpu.min = "0"

sched.cpu.units = "mhz"

sched.cpu.shares = "normal"

sched.mem.minsize = "0"

sched.mem.max = "2048"

sched.mem.shares = "normal"

So 2048MB, the same as set in the VM.

😕

[AartK]

From frankdenneman:

Let’s focus a bit more on reclaiming. Reclaiming of memory is done by ballooning or swapping. But when will ESX start to balloon or swap? ESX analyzes its memory state. The VMkernel will try to keep 6% free (Mem.minfreepct) of its memory. (physical memory-service console memory)

When free memory is greater or equal than 6%, the VMkernel is in a HIGH free memory state. In a high free memory state, the ESX host considers itself not under memory pressure and will not reclaim memory in addition to the default active Transparent Page sharing process.

When available free memory drops below 6% the VMkernel will use several memory reclamation techniques. The VMkernel decides which reclamation technique to use depending on its threshold. ESX uses four thresholds high (6%), soft (4%) hard (2%) and low (1%). In the soft state (4% memory free) ESX prefers to use ballooning, if free system memory keeps on dropping and ESX will reach the Hard state (2% memory free) it will start to swap to disk. ESX will start to actively reclaim memory when it’s running out of free memory, but be aware that free memory does not automatically equal active memory.

______________________

Is there anyone who can tell me why the VM's are still in swapping/ballooning state when there is enough physical memory? And why I can't see the granted memory of a VM in vsphere client. (only in esxtop)

[rickardnobel]

I think it can be quite easy.

Once the large VM was started and needed lots of memory many of the other VMs were forced to balloon or swap out their memory. Now that the large VM is gone there are memory free, but...

the memory that has been swapped to disk by the Vmkernel has most likely not been needed by your guest VMs, and is still on disk for that reason. You can see by the counters that there are 0 swap reads/writes. It is just that some part of the VM memory is placed on disk and will be picked up when (if) ever needed again by your VMs. Since they have not accessed these memory parts points to that they are not important inside the VMs.

I would say that you should leave your VMs and not reboot them. Just watch for ongoing swap, that will hurt your performance.

[Deepeer]

Hopefully I can answer and clear some of your questions

>>> But when I check the "resource allocation" tab, I still see the VM's ballooning and swapping.

The "resource allocation" tab says "Ballooned" / "Swapped" and not "Ballooning" / " Swapping" . This means that , to make room for the beefy test VM , some memory of VM was ballooned / swapped .

>> This is resolved when I reboot the VM.

This is expected as memory held by VM before reboot is discarded during reboot since it is no longer needed ( including ballooned/swapped ) . You can also observe that much of the pages/memory are shared after reboot .

>>There are still a few vm's left with ballooning and memory swapping.

ESX does not automatically start "swapping in" the "swapped out" memory , as swapping is a costly affair and could have big impact on the performance of the host/VM . It "swaps in" when

A) The VM demands pages/memory .

or

B) Periodically when cpu cycles are free ( this is a slow process to minimise performance impact )

Since the VM has less "active" memory , I assume it is idle and not demanding pages/memory .

You can confirm that VM is not "swapping out" / "ballooning" from the "Performance" tab .

Hope this helps .