EPL
Contributor
Contributor

Memory Balloon question

Jump to solution

I have a vm that shows high memory Balloon average usage. None of my other VM's on my other servers show any ballooning, and no other vm on this host is ballooning. One of the VM's has 768MB of memory reserved, yet here are the stats for memory:

Memory Balloon Avg: 413632 Latest 423360 Max

Memory Granted Avg: 364228 latest 376708 Max

Memory Active avg: 55396 latest 86264 Max

Memory Usage avg % : 7.04 latest 10.96 Max

I do not seem to have memory contention issues on the ESX Host. I have 4.4GB used out of an available 12. So I'm trying to determine why time VM is ballooning?

Any ideas?

Thanks!

Tags (1)
0 Kudos
1 Solution

Accepted Solutions
mike_laspina
Champion
Champion

Your host is doing exactly what it is instructed to do. Reclaim the memory above the limit.

|----Config File................................../vmfs/volumes/43705541-380d451c-08eb-00163568aec7/Checks/Checks.vmx

\==+Memory Client Stats :

|----Touched...................................35.18 MB

|----Current Size..............................502.60 MB

|----Target Size...............................346.84 MB This is the memory it needs

|----Balloon Active............................1 KB

|----Balloon Size..............................265.39 MB This is what is reclaimed based on the instructed limit.

|----Balloon Target............................265.09 MB

|----Balloon Maximum...........................499.20 MB

\==+Memory Allocation :

|----Valid..................................true

|----Minimum................................0

|----Min Limit..............................unlimited

|----Maximum................................262144 This is what is limited to. It should be 512MB

|----Shares.................................normal

|----Units..................................kb

|----Config File................................../vmfs/volumes/43705541-380d451c-08eb-00163568aec7/Citrix/Citrix.vmx

\==+Memory Client Stats :

|----Touched...................................57.60 MB

|----Current Size..............................384.00 MB This VM needs more memory

|----Target Size...............................436.39 MB This is what it needs.

|----Balloon Active............................1 KB

|----Balloon Size..............................0 KB

|----Balloon Target............................0 KB

|----Balloon Maximum...........................249.60 MB

\==+Memory Allocation :

|----Valid..................................true

|----Minimum................................0

|----Min Limit..............................unlimited

|----Maximum................................393216 This should also go up with the granted memory.

|----Shares.................................normal

|----Units..................................kb

|----Config File................................../vmfs/volumes/43705541-380d451c-08eb-00163568aec7/epl-farm-05/epl-farm-05.vmx

\==+Memory Client Stats :

|----Touched...................................133.12 MB

|----Current Size..............................1024.00 MB |----Target Size...............................755.45 MB This VM is happy.

|----Balloon Active............................1 KB

|----Balloon Size..............................0 KB

|----Balloon Target............................0 KB

|----Balloon Maximum...........................665.60 MB

\==+Memory Allocation :

|----Valid..................................true

|----Minimum................................0

|----Min Limit..............................unlimited

|----Maximum................................1048576

|----Shares.................................normal

|----Units..................................kb

As you can see the memory allocation limits are impacting optimizations in a negative way.

It is instructed to force the VM to release free memory and swap it out the OS pagefile.

This is a common issue that many of us end up with.

It's easy to fix. Monitor esxtop and adjust the maximums to be at the granted memory or a little higher.

If you change a grant then check if you have a maximum memory allocation and adjust it as well.

http://blog.laspina.ca/ vExpert 2009

View solution in original post

0 Kudos
12 Replies
mcowger
Immortal
Immortal

Was is migrated OFF a host that was balooning?

--Matt

--Matt VCDX #52 blog.cowger.us
0 Kudos
mike_laspina
Champion
Champion

You need to verify that the VMware tools are installed and that the vmmemctl driver is loaded.

You will find the driver using Computer Management->View->Show Hidden Devices in Windows.

Expand Non-plug and Play Drivers and check the properties of the VMware Server Memory Controller.

In Linux use ps -A and it is the vmmemctl process that should be present.

A reservation will not stop the vmmemctl from inflating and deflating the memory of a reserved mem VM, it's based on memory resource demands of the VM.

It can be of benifit to disable the optimization if you a significant amount of free memory. There is a small overhead to run it.

Here is a good reference if you wish to know more.

http://www.kingston.com/branded/pdf_files/Memory_Provisioning_Recommendations_VI3.pdf

http://blog.laspina.ca/ vExpert 2009
AWo
Immortal
Immortal

You need to verify that the VMware tools are installed and that the vmmemctl driver is loaded.

Can ballooning happen without having the VMware Tools installed?

AWo

vExpert 2009/10/11 [:o]===[o:] [: ]o=o[ :] = Save forests! rent firewood! =
0 Kudos
mike_laspina
Champion
Champion

The vmmemctl driver is required for ballooning, while not verified, yes it can if the driver is installed.

With out the driver ballooning will not occur.

I will verify it for you tommorow.

http://blog.laspina.ca/ vExpert 2009
0 Kudos
EPL
Contributor
Contributor

What I don't understand is why would it balloon if there are no memory constraints on the ESX host?

according to the definition:

"When memory demands rise on the ESX host, the VMkernel will instruct the balloon driver to "inflate" and consume memory in the running guest OS "

The Vm is not memory constrained... it has not reached its reservation. I'm just wondering why it would balloon.

0 Kudos
mike_laspina
Champion
Champion

VMware ESX uses more than this one element to manage memory. Many of the elements are not publicly discussed. There are many factors that impact the memory optimization algorithms.

Here are several major impact elements that affect the optimization behavior.

1) Resource Shares

Shares can impact the memory tax algorithm applied to a VM

2) Limits

Limits can prevent poorly written apps from hogging ram.

3) Reservations

Reservation can create poor resource allocation and cause more ballooning than if they were not set.

4) Memory Grants

Too much allocation of memory to a VM will keep the optimization working harder.

5) Overhead

This is a big one, even a CPU requires overhead memory to operate outside of what is granted. If you allocate 1GB of ram 2 CPU's will need 100MB outside of it to operate the VM.

If you wish to go deep on the subject I would recommend reading the attached paper.

http://blog.laspina.ca/ vExpert 2009
mike_laspina
Champion
Champion

Verified. If you install only the vmmemctl driver ballooing still works.

If you run without ballooning you will seen an increase in demand paging. This memory optimization method is not preferred since it is LRU based and is external to the VM's OS knowledge. File swap should be a last resort. Disk is at least 1000 time slower the RAM.

http://blog.laspina.ca/ vExpert 2009
0 Kudos
EPL
Contributor
Contributor

All this information is really good and helpfull, and I certainly appreciate your help and your input.

I guess I'm just trying to figure out why this VM, and only this VM is ballooning. I've got about 10 other VM's some with more memory reservations, some with less. Also I am not using any custom resource pools.

I went into the Device manager like you said and disabled the balloon driver. Rebooted, and as expected there was no ballooning. Memory utilization was the normal 300MB or so out of the 768MB as it was previous. I let it run for a few hours then turned the Ballon driver to automatic and rebooted. As soon as I rebooted, the ballon avg was up to 445MB.

0 Kudos
mike_laspina
Champion
Champion

Ok.

Lets have a look.

Run esxtop on the host and select m and post a screen shot.

Run esxcfg-info -r > resource.log and post the file.

http://blog.laspina.ca/ vExpert 2009
0 Kudos
EPL
Contributor
Contributor

thanks for your help... attached are the files

0 Kudos
mike_laspina
Champion
Champion

Your host is doing exactly what it is instructed to do. Reclaim the memory above the limit.

|----Config File................................../vmfs/volumes/43705541-380d451c-08eb-00163568aec7/Checks/Checks.vmx

\==+Memory Client Stats :

|----Touched...................................35.18 MB

|----Current Size..............................502.60 MB

|----Target Size...............................346.84 MB This is the memory it needs

|----Balloon Active............................1 KB

|----Balloon Size..............................265.39 MB This is what is reclaimed based on the instructed limit.

|----Balloon Target............................265.09 MB

|----Balloon Maximum...........................499.20 MB

\==+Memory Allocation :

|----Valid..................................true

|----Minimum................................0

|----Min Limit..............................unlimited

|----Maximum................................262144 This is what is limited to. It should be 512MB

|----Shares.................................normal

|----Units..................................kb

|----Config File................................../vmfs/volumes/43705541-380d451c-08eb-00163568aec7/Citrix/Citrix.vmx

\==+Memory Client Stats :

|----Touched...................................57.60 MB

|----Current Size..............................384.00 MB This VM needs more memory

|----Target Size...............................436.39 MB This is what it needs.

|----Balloon Active............................1 KB

|----Balloon Size..............................0 KB

|----Balloon Target............................0 KB

|----Balloon Maximum...........................249.60 MB

\==+Memory Allocation :

|----Valid..................................true

|----Minimum................................0

|----Min Limit..............................unlimited

|----Maximum................................393216 This should also go up with the granted memory.

|----Shares.................................normal

|----Units..................................kb

|----Config File................................../vmfs/volumes/43705541-380d451c-08eb-00163568aec7/epl-farm-05/epl-farm-05.vmx

\==+Memory Client Stats :

|----Touched...................................133.12 MB

|----Current Size..............................1024.00 MB |----Target Size...............................755.45 MB This VM is happy.

|----Balloon Active............................1 KB

|----Balloon Size..............................0 KB

|----Balloon Target............................0 KB

|----Balloon Maximum...........................665.60 MB

\==+Memory Allocation :

|----Valid..................................true

|----Minimum................................0

|----Min Limit..............................unlimited

|----Maximum................................1048576

|----Shares.................................normal

|----Units..................................kb

As you can see the memory allocation limits are impacting optimizations in a negative way.

It is instructed to force the VM to release free memory and swap it out the OS pagefile.

This is a common issue that many of us end up with.

It's easy to fix. Monitor esxtop and adjust the maximums to be at the granted memory or a little higher.

If you change a grant then check if you have a maximum memory allocation and adjust it as well.

http://blog.laspina.ca/ vExpert 2009
0 Kudos
EPL
Contributor
Contributor

Mike,

This information is great, I now see where my issue is!

Wish I can give you +20 points.

I couldn't understand why it was ballooning because there was no contention. It seems as though one of my other administrators may have been playing with the individual VM limits, so it didn't even occur to me to check that setting!

Thanks again

0 Kudos