50 VMs per blade = high Memory Balloon & Memory Swap Used. Concerned?

[AGServerTeam]

We have "succeeded" in putting 50 VMs on one blade. The characteristics are as follows:

• Guests: 50 Windows XP Professional SP2 virtual machines, each with 1 CPU and 1GB RAM. The RAM is throttled back to 768 per, the CPU is throttled back to 1000MHz per.

• Host: 32GB, 4 CPU x3GHz. VMware ESX 3.0.2

As long as e-Trust (our antivirus vendor) doesn't kick off, people are able to be productive.

We began layering VMs onto this blade last Tuesday at ~ 4PM.

I have concerns and wonder if you could confirm those concerns or provide a different perspective. The chart below (more detail in the attached spreadsheet), indicates activity that I was expecting:

1. My memory balloon for the host increased from 92MB to 19,896MB (19.4GB).

2. My Memory Swap Used (Average) rose from an average of 519MB before the test started to an average of 1,414MB in the day after the 50^th guest was added.

My understanding of how VMware manages memory is as follows:

1. Memory de-duplication happens on an ongoing basis and as such isn't relevant to this discussion.

2. In order to relieve memory pressure, memory ballooning is engaged on the guests as a first resort.

3. In the event that memory ballooning does not entirely relieve memory pressure, swap file usage increases.

Based on this understanding, combined with our own observations and reading of blogs and discussion forums, memory ballooning is no big deal, but swap file usage is. One comment I read was that I want my swap file usage to be as close to zero as possible. Now, on my other blade (same hardware configuration) I have 26 VMs and as of this moment the Memory Swap Used is 131MB with my Memory Balloon at 0. So it seems to me that achieving 0 Memory Swap Used is impossible in typical environments.

So, what's the question? The question is, should I freak out that our Memory Swap Used and Memory Balloon are sky high, or is it no big deal, provided the end-users aren't impacted? Am I looking at the wrong counters? Should I be more worried about Memory Swap In and Memory Swap Out?

Attached is the Excel spreadsheet that details the relevant memory counters over the past week.

Thanks for any perspective you can provide.

Dane Hershberger

Infrastructure Architect

American Greetings

[chouse]

Curious why you allocate 1GB ram to the VM yet throttle it to 768? Why not just allocate 768? Doesn't throttling effectively limit the amount it can use, so that extra 256 goes wasted? Also, have you tried allocating less memory? Our VDI sessions are 512MB with no CPU or mem throttle and are running fine performance-wise.

A server with 43 VMs (my highest number of VMs on a server after DRS balances them out) and 24GB server memory, dual quad-core intel xeon procs and VMs all with 512mb mem has no swapping or ballooning at all. 53% memory used - 22.5 GB granted memory, only 5 GB "active", 1.3 GB shared.

[AGServerTeam]

1GB per VM versus throttling it back to 768: Balancing out the perceived but not yet quantified requirements versus what we've identified thus far as the actual memory usage. My understanding of how VMware memory works is that the memory is not going to waste, even if it is allocated, since it is not actually in use by the guest.

Are you suggesting I might see different memory behavior (lower ballooning/swapping) be decreasing the memory allocated per VM to 768?

[chouse]

I can't say for sure but it's definitely worth trying - dropping the VDI sessions down to 512MB, no throttle on memory, make sure the windows pagefile is system-managed, don't specify a size.

After you've changed and booted all the sessions, you'll need to wait several hours for the memory to calm down because windows will zero it out when it boots, making it look like it's using it all.

[Carlo_Giuliani]

We have a much less stressed VMWare environment, and I think we can afford to squeeze it harder. Yours is an interesting example. What values you are seeing for "memory active" and "memory shared common"??

You didn't actually say, but it looks like your host has 32GB memory. Is that right? In our case, we average about 20 VMs each of 3 hosts Each host has 32GB memory and 2xquad-core processors. We have very little balloon memory - about 5% of physical memory. And our "active" memory is extremely low - about 10% of physical memory in each host.

I am trying to convince people here we can afford to be stingier with the memory allocations, and my main evidence is the low "memory active" values. But to be sure, I would like to confirm that a stressed environment (like your) has higher memory active values. I am guessing that in your situation the sum of "active"+"balloon"+"shared common" will be pretty close to the total of "memory granted".

[cmorris05]

I am in the process of setting up a VDI enviroment for our Higher Ed center, to replace the classroom labs. One problem I am seeing more then memory usage, is CPU utilization. How does the blade handle all those desktops when the users ramp up their use? I am looking at using a dual Quad Core 3.16 Ghz with 16GB of ram to support at least 25 VMs.

We did a test using our existing servers (4 dual Dual core 2.66 GHz with 16GB RAM each) which supports our servers. When I got all the desktops connected and started running them up, we maxed out all the servers on CPU utilization. The servers were not doing much so they did not take much processor, but I was just wondering is this what I should expect?

Also, what version of ESX are you using to host the VMs?