memory usage & vm right sizing

[Jae_Ellers]

I'm trying to get my head around VM memory usage as it relates to DRS & ESX 3.

I have a cluster of 4 hosts with 48 Gb & 43 Ghz available. Currently 60 VMs running here.

CPU util is around 20%.

Memory util is around 74%.

Active memory shows around 5Gb.

The 74% number exactly matches the amount of RAM allocated to the vms vs the physical RAM in the server.

With a default ESX config, what's the over committment allowed? How is this adjusted? Where is this documented?

Is my memory usage apparently high just because there's no contention and therefore no ballooning or swapping is happening?

Has anyone developed a good process for determining if vms have the right amount of memory allocated?

I believe everything is working fine, but I need to understand how much I can push things before performance will erode.

[esiebert7625 · Accepted solution]

Are you using memory reservations at all? If you make reservations for your VM's that are equal to the amount of RAM assigned to them then swapping and page sharing does not occur. You can overcommit pretty heavily if you are willing to pay the performance penalty. If you do not set reservations then ESX create a VSWP file equal to the difference between the amount of physical memory assigned to the VM and the reservation it has. By default memory reservations are set to 0. So if you have a VM with 2GB of memory without a reservation it will create a 2GB vswp file when it is powered on. It will start using the vswp file if the server is out of physical RAM. If you set a 1GB reservation then it will create a 1GB vswp file instead. The vswp files are what allows for memory over-commitment.

Good Windows memory doc...

Understanding memory usage in Windows 2000 - http://www.microsoft.com/technet/prodtechnol/windows2000serv/tips/w2mem.mspx

Also see this post...

http://www.vmware.com/community/thread.jspa?messageID=549391��

These are good memory documents...

The Role of Memory in ESX Server 3 - http://www.vmware.com/pdf/esx3_memory.pdf

Memory resource management in Vmware ESX Server - http://www.vmware.com/pdf/usenix_resource_mgmt.pdf

[Jae_Ellers]

thanks. This is very helpful. I've read thru the whitepapers listed before, when I ran into swapping under ESX 2 and there wasn't much to do about it.

Now I'm getting close to saturating resources and we'll still need to bring some more vms into the cluster before the new blades show. I can either stop adding new vms to the cluster or deal with the situation with new features.

I have not configured many reservations.

Going forward I think using resource pools to segregate systems of different priority will be the way to go, then fine-tuning the pecking order in those with individual reservations. This way the performance drain on a swapping system can be focused on those less important or demanding systems.

With this discussion I finally get what all the knobs are good for. I knew what they did, but didn't get the cohesive picture.

1. Use resource pools to group machines & guarantee resources

2. Use reservations & limits to control individual system resources

3. Use priority to balance demanding vms

None of it matters without contention except for memory max - reservation = swap size.

Thanks. What's the e stand for? esiebert7625 doesn't exactly roll of the keyboard.