swap / balloon / memory issues

[pufferdude]

I recently installed Veeam Monitor 5 (full version) and it started alerting me to things I didn't notice in vCenter itself... I have two ESXi 4.1 hosts in a cluster, one with 24GB ram (dual quad core) and the other with 48GB (dual hex core). I have 22 VMs, divided about equally between the two hosts. About half of the VMs are alarming (via Veeam Monitor) with "VM Memory swap usage" and/or "High balloon memory utilization" alerts, and both hosts are alarming with "This host is swapping too much memory" alerts.

What I completely don't understand is that NEITHER of the hosts is memory contstrained in any way. If I add up the memory assigned to each VM on either host, it barely comes to HALF of the physical memory... there should be zero physical memory contention as I'm barely using 50% of the actual memory in each host... so I'm totally baffled as to what these alarms are actually telling me. They all repeat about every 10 minutes.

Can anyone point me in a direction to start looking? I'm not an expert in ESXi's memory handling (didn't think I needed to be, since i'm not over-committing at all). But I also have trouble believing that Veeam Monitor is reporting false alarms, so SOMETHING is going on, I just have no idea how to deal with it.

[a_p_]

Unless it's false alarm, the only thing I can think of at the moment are memory limits. Did you configure memory limits for the VMs?

André

[pufferdude]

I have no memory LIMITS enabled, but on some of them I have shares=high with a memory reservation... is that possibly the problem? Not all VMs have a reservation (only about 5 our of 22), and my intention was to reserve, say, 2048MB for my 8GB Exchange box, and give it "high" shares.

[VMmatty]

Have you looked at the VMs that are reporting swapping/ballooning within vCenter to confirm that the data is accurate?  I'm not doubting Veeam Monitor but just want you to confirm.

Was there ever a time when the hosts in question were memory constrained?  Perhaps during host maintenance or something?  Maybe it's just reporting on old data.

Matt

[bpierfy]

What OS are they running?

I've noticed that in Windows 2008 Server I have problems with large amounts of allocated memory.  The solution for me was to put a memory reservation on each VM (I was putting 2GB reservation on a 4GB VM, but I'm still looking for advice on proper amounts).

I think the issue is that ESX is able reclaim almost all of the memory allocated to a VM when it running that much memory (or in combination with win2008, not entirely sure what the cause is).  The excess memory is swapped out to other VMs.  But when the machine is then used again, it needs memory to perform even somewhat basic functions, so it swaps the memory back in.  I don't understand what is going on behind the hood exactly, as memory itself should take very little time to move, but I think it is related to the swap file that is created for each VM.  The slower your storage; the longer that will take.

I think it would be worth putting a reservation on those VMs and see if the alert goes away (maybe try half the allocated amount or 2GB).  I don't know if it will fix your problem but I did see the swap in/out rate go down after I made the aforementioned changes.

[Dev09]

Issue might be because of large page allocation on system. Please refer to KB#1021896.

Link: http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1021896&sl...

-Dev

[bulletprooffool]

Are the machines that are ballooning perhaps machines WITHOUT reservations (ie - you have reserved most / all resource on other VMs)?