VM ballooning while ESX plenty of free mem and no overcommitment

[feuillerat]

Hi everyone,

I'm experiencing a strange behaviour on ESX 3.0.3, 104629 server.

It is just installed, only 3 VM running on it,

48GB of physical RAM in the ESX,

and ...

some VMs are already using the balloon driver and the ESX even swaps.(see below)

I thought only low memory condition on the ESX would trigger the ballooning (around 75%), or memory overcommitment.

Can someone shed some light on this ?

here is an extract of an esxtop :

4:37:00pm up 9 days, 5:24, 65 worlds; MEM overcommit average: 0.00, 0.00, 0.00

PMEM (MB): 49149 total: 800 cos, 427 vmk, 4925 other, 42996 free

VMKMEM (MB): 47528 managed: 2851 minfree, 4058 rsvd, 43325 ursvd, high state

COSMEM (MB): 234 free: 4000 swap_t, 4000 swap_f: 0.00 r/s, 0.00 w/s

PSHARE (MB): 3884 shared, 281 common: 3603 saving

SWAP (MB): 109 curr, 96 target: 0.00 r/s, 0.00 w/s

MEMCTL (MB): 155 curr, 155 target, 5543 max

ID GID NAME NMEM MEMSZ SZTGT TCHD %ACTV %ACTVS %ACTVF %ACTVN MCTL? MCTLSZ MCTLTGT MCTLMAX SWCUR SWTGT SWR/s SWW/s

12 12 vmware-vmkauthd 1 2.20 2.20 0.34 0 0 0 0 N 0.00 0.00 0.00 0.00 0.00 0.00 0.00

18 18 VM1 7 3920.00 3328.65 270.82 3 2 2 0 Y 51.04 51.04 2548.00 8.28 1.31 0.00 0.00

21 21 VM2 7 1024.00 706.27 78.87 10 4 8 8 Y 38.15 38.15 665.60 3.31 0.66 0.00 0.00

22 22 VM3 13 3584.00 886.62 105.54 0 0 0 0 Y 66.00 66.00 2329.60 98.09 22.22 0.00 0.00

Olivier

[Troy_Clavell]

try to increase your service console memory to say 800MB

[feuillerat]

It is already at 800MB... (see in the esxtop extract)

[jparnell]

Check the resource allocation settings, and that the virtual machines don't have memory limits applied to them.

[Troy_Clavell]

sorry, I missed that. Make sure none of your VM's have any limits set.

[Troy_Clavell]

also, why is your SWAP partition 4GB? can you do a vdf -h and check for free space?

[feuillerat]

Nice suggestion,

but I sort of never apply limits to VM (kinda nightmare to follow up),

however 1 or 2 of the VM has a small memory reservation (350MB reservation on the one with 1024MB RAM allocated for instance.

Still I would not understand the relation between reservation and ballooning on an ESX with plenty of free RAM pages.

Any idea ?

To me that is the point of the problem.. my understanding of "ballooning" is that it is only used when memory is scarce on the ESX host.

[feuillerat]

# free -mt

total used free shared buffers cached

Mem: 780 547 232 0 101 298

-/+ buffers/cache: 148 632

Swap: 4000 0 4000

Total: 4781 547 4233

yes you may find the swap partition a bit big on the host.

# vdf -h

Filesystem Size Used Avail Use% Mounted on

/dev/cciss/c0d0p2 9.7G 1.3G 7.9G 14% /

/dev/cciss/c0d0p1 198M 32M 156M 17% /boot

none 391M 0 391M 0% /dev/shm

/dev/cciss/c0d0p6 3.9G 33M 3.7G 1% /tmp

/dev/cciss/c0d0p5 9.7G 759M 8.4G 9% /var

/vmfs/devices 1.9T 0 1.9T 0% /vmfs/devices

/vmfs/volumes/48d9d775-41244c85-3742-00237dcf1b34

40G 626M 39G 1% /vmfs/volumes/storage1 (1)

/vmfs/volumes/4a082549-80908dea-d21e-00237dcf1b58

299G 119G 180G 39% /vmfs/volumes/VMware_LUN1

/vmfs/volumes/4a114ef5-5ebee443-168e-00237dcf1b58

299G 70G 229G 23% /vmfs/volumes/VMware_LUN2

/vmfs/volumes/4a114f63-aa6ed9ad-6fb0-00237dcf1b58

299G 111G 188G 37% /vmfs/volumes/VMware_LUN3

[RParker]

To me that is the point of the problem.. my understanding of "ballooning" is that it is only used when memory is scarce on the ESX host.

Yes, correct. ESX forces VM to use their own internal memory to free up share resource, via balloon driver. So it must be a resource pool that has allocated a smaller subset of memory and therefore the VM's are forced to balloon because their 'container' only has so much memory free.

Reservations means that you reserved a portion of the memory for that VM, and it can't be given to any VM's until that VM is turned off. But somewhere on your host, VM settings or resource pools your VM's are being forced to use Balloon driver.

[feuillerat]

Yes I know that RParker,

but I am not using any resource pool, nor Limits on resources,

it's a very simple setup, just installed with plenty of RAM and only a couple of mid sized VM

so I have no idea what's triggering ballooning and swapping, hence my question...

[RParker]

OK, since you already know, then unless it's causing a performance issue, it's not really a problem. Normally this shouldn't happen, but if it's not causing any adverse affects, there is no cause for concern.

Ballooning isn't inherently bad.

[feuillerat]

Well thank you, it's true that it is not causing performance issue... so far...

but now I wonder how it is going to be when I have 10 more VMs on the host and memory usage goes around 65%,

ESX is already swapping with 10% overall memory usage.

What annoys me really is that it contradicts everything I have learn through the VMware Training, documentations, etc, etc,

to me it's a binary thing, if the host use less than 70% memory --> no ballooning, and usually not much swapping.

but this is really... strange !

So I can't buy the argument "sometimes it happens"....

I guess I will have to live with that, but I wanted to share the experience, and see if someone had "seen" such things happen on ESX 3.0.

Thank you for trying to help !