Consumed vs. Active Memory

[theowood]

We are trying to create a monthly report on VM memory usage and need help understanding the stats.

There are two key stats: Mem Consumed and Mem Active

What they mean exactly is not well documented and difficult to ascertain.

We want to determine how much additional memory we have available to allocate on our hosts. For example, we have a host with 16GB of memory.

It has five VM's on it that have all been running now for several months. Right now the total memory active is only 2GB on average and 2.5GB on peak.

However, memory consumed on the host is 12GB.

Memory granted to all machines total is 14GB.

If you look at these stats does this mean that we have plenty of room to add more machines?

It appears from these stats that we should adjust down the memory granted for each machine to something closer to what it is actually using during peak usage. Is this a good practice?

At what point will we be prevented from adding additional machines because of insufficient memory?

In other words, does VMWare look at the memory consumed statistic, memory granted or memory active when determining whether we can VMotion a machine to this host or startup a new machine?

[dbis]

Theo,

As you know memory granted it the memory allocated to your VM's.

Memory consumed is the memory allocated to your VM's minus and memory shared through page sharing. Looking at your numbers I assume most of your VM's are running the same OS.

Memory active is the memory that is actively being used by your guests.

You have currently 14GB allocated, you could add more VM's and overcommit the memory. Looking at your active memory utilization this does not appear to present a problem. If all systems would require all their memory at the same time, you woudl take a big performance hit due to swapping.

Keep in mind that SQL and to a lesser degree Exchange will use your memory if it is needed or not to cache data. So there will be memory in use, but not active.

It would be cleaner to reduce the allocated memory to your VM's and also make it easier to troubleshoot. If you constrain the VM's and not overcommit the memory on your ESX host, then you isolate any potential problems. You can use perfmon on your VM's to get a realistic idea on how much memory should be allocated.

Sincerely,

Daniel Bischops

[theowood]

What does VMWare look at to determine whether there is enough memory free that you can start a new VM? What about whether there is enough memory free that you can VMotion to the host?

[dbis]

Theo,

As you can overcommit memory, it will not prevent you from starting new VM's or using vmotion. Altough you can really kill performance (which causes errors in itself) if you overallocate too much, there is nothing stopping you.

The exception is when you set reservations on the memory for a VM. A reservation will grant a set amount of memory to the VM if it is using it or not.

The total reservation of your VM's cannot exceed the physical memory or it will not power on.

Daniel

[theowood]

That makes sense.

The part I do not understand is why active memory is generally so much smaller than consumed memory. At what point will the memory consumed grow?

For example, I have a guest VM which is actively using only 150MB of memory. It is granted 4GB of memory. It is consuming 1.8 GB of memory. However, if you watch the guest perf mon from Windows, it never goes higher than 1.5GB of memory committed.

Does this mean we could move the memory granted down to 2GB?

Why does consumed memory only show 1.8GB? Is VMWare sharing the rest of the memory?

[RParker]

Bingo! That's EXACTLY what that means, and exactly the reason why you should not give people memory because they simply ASK for it.

Give them half of what they want, prove they don't need it, and show them the results. You can add memory later \*IF* they are actually using it.

The architecture of ESX is such that core and memory are shared among like OS. So on an ESX host with ALL Windows 2003 guests are extremely efficient, in fact it's up to 30% shared across the guests when they are the same, so the more VM's you add, the more efficient it becomes.

[nsnook]

I appreciated the great explination and walkthru on this

[theowood]

There is significant risk when starting with half of the memory required that some applications may run out of memory in production. You are also going against vendor recommendations which may lead to support issues with the vendor.

If we take the approach of giving the application what the vendor required, doesn't VMWare automatically reclaim unused memory with the balloon driver if there is contention for memory?

When the balloon driver kicks in, will this lower your consumed memory, your active memory or both?

If someone has a script that will simulate an application with heavy memory usage that would help. The only scripts I have seen simulate heavy CPU usage.

[jhanekom]

Minor risk point: if you carelessly overcommit memory you don't have just to pacify vendors, you stand the risk that if one or more VMs develop a problem (say, a memory leak), they could bring down performance for the entire platform.

Ultimately, politics and the amount of clout you have as owner of the virtualisation platform will play a role, but if you can consistently prove yourself correct by using RParker's strategy judiciously, you'll buy confidence points over time.

Something else that tends to work well is to tell system owners that if they want a guaranteed amount of memory, they must fork out the money to buy it. That's the way it would work on physical systems, and gives you extra capital to expand your VMware environment.

(Once you see balooning in your environment, you're already getting performance degradation, as it will probably force VMs to swap pages to disk.)

[JonRoderick]

Thanks - clears up my understanding too - I think...

Just one thought then, how come Consumed Memory isn't a default counter if it's more representative of memory use than Active Memory? When I'm arguing with a VM owner over whether or not they need more RAM, should I be looking at Consumed Memory or Active Memory?

Jon

[Jeffrey79]

I've ben doing some research today about memory stats in ESX and my understanding is that it's Active memory that's important in those conversations.

Consumed memory is just the Granted memory minus the Shared memory. You can check that on any of your hosts or vms. Shared memory is relative to how much memory is used for memory page sharing on the host, which mostly has to do with what common OSs are being run on that host. So Consumed memory really has little to do with what the vm is actually using.

Think of this example - An ESX host running 9 Linux vms and 1 Windows vm. That Windows vm will have a very low Shared memory value due to it being the only Windows guest on the server. You allocated that Windows vm with 4gb of memory. So the Consumed memory comes out to be slightly less than 4gb. But that Windows vm doesn't run anything except DHCP. So the actual memory being used by the vm is next to nothing. It's the Active memory setting which tells you that.

I hope this helps to further clear things up for people. Someone please correct me if I'm giving bad info here. I'm still getting the hang of analyzing these stats.

[Alceryes]

Great information here!

Even though this is in the 3.5 forum the info is still relevant for today's ESXi (4.1).

Thx OP, and all who replied.