Understand how resource pools really work

[JaseM]

In researching how to properly nail down resource usage for resource pools in our clusters, I've found that there are some nuances that are not specifically documented regarding "Reservation" vs "Limit" as they relate to Resource Pools. If anyone is a Resource Pool expert please chime in. Specifically I'm looking for validation of what we believe is true.

Here is the case:

So based on my interpretation of the answers and our resource pool design, it sounds like the "Limit" is always an important value that decides the full aggregate amount of VM's memory (reserved memory + swapped memory + shared memory + virtual memory) in a given resource pool of all VMs and does not imply that memory is taken from a parent resource pool.

In our environment we don't allow "Expandable Reservation". Our question was, "Does the limit value matter". We assumed that with the "Expandable Reservation" NOT CHECKED that the limit value did not matter even though it's not greyed out. So if the limit is not greyed out, is it allowing resources to be taken from the parent? I'm understanding that this is not happening. In our case the limit does not mean resources taken from parent but rather seems that this defines the cap of memory from all the VMs in the resource pool can use (reserved memory + paged memory + shared memory + virtual memory).

Example(1): We have a resource pool called "Exchange-RP" with 10GB Reserved, Not Expandable, and a limit of 20GB. If we have 11 VMs each configured with 2GB of ram (The individual VMs have no reservation but the VM is configured for 2GB).

We can power on the first 10 VMs ( 10 x 2GB = 20GB ) but the 11th will not since this would exceed the limit.

In this example how are the 10VMs getting the 2GB each? I'm assuming each is getting 1GB of the physical memory and the other 1GB is made up of virtual memory ( swap, shared, etc, non-physical ram ). At no point is physical memory being drawn from the parent.

Example(2): Given the exact scenario above. If the unlimited box is now checked (limit is greyed = unlimited), this would mean that the 11th VM would now be allowed to be powered on and that whatever physical memory reservation there is just needs to be shared by all VMs in the pool. It seems that once we get too many VMs powered on that there will be too little physical ram per VM and we will start seeing excessive ballooning and then swapping in turn affecting performance.

Please confirm the Examples above is accurate of how memory is managed.

We are looking to validate that the RP limit value always matters as hard limit of all memory used by the VMs, not just physical(reserved memory). None of the VMware documentation speak to this.

Thanks,

Jase

[weinstein5 · Accepted solution]

Welcome to the Forums - Example 1: My understanding is the resource pool should act like a host in that you oversommit the memory assigned to the resource pool - so you should be able to power on that 11th VM - with 11 VMs all needing 2 GB and with the limit set to 20 GB - if all the VMs are using their 2 GB and no page sharing occurring I would expect to see Ballooning and per VM vmkernel swap files being used - now if you have reservations set 2 GB you would only be able to power on 5 VMs in this example because with 5 VMs you would be using all of the 10 GB of reserved memory - and the vmkernel will not power on a VM if it can not guarantee its reservation -

Example 2 - see above - even with alimit set you will be able to oversommit the memory in the RP - the only thing you will gain if you have the limit set to unlimited is that you will be bale to power up more VMs and impact all the users on that ESx host or cluster -

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[JaseM]

Hello Weinstein, Thanks for your answer. It looks like you've confirmed what I've figured out. These little limit vs reservation details (As it relates to the RP level) are not clear in the official documentation.

With that said, here is my next question:

If the limit value is used to cap the amount of all memory available to VMs in a resource pool, then how do we control how much physical memory is taken from a parent when the "Expandable Reservation" box is checked? Is it a free for all when sibling RP's compete?

Previously I thought the limit value was just used to inicate how much memory (Physical/Reserved) could be taken from the parent when the "Expandable Reservation" box is checked. Now I understand it ( Limit value ) is just for a cap on aggregate of VMs configured memory in a RP.

It seems its almost safe to say (At the RP level) that the limit value has nothing to do with the physical reservation amount.

Jase

[JaseM]

I did a full test case to get the answers, please see the attachment.

The limit value is a very important resource pool setting that dictates the hard limit cap of how much CPU or Memory will be available to all the VMs in a RP. This hard limit can be reached in a number of ways depending on wither we are using Expandable Reservation (fixed/not fixed) or if we are using a defined limit or unlimited.

It all boils down to a math game of comparing the RP’s defined reservation amount VS fixed/not fixed VS limit/no limit VS the configured amount of resources per VM. This is a game where we must decide if we want to over allocate or not over allocate.

CPU: When each VM gets a single vCPU, if the RP is unlimited each VM could use up to an entire core worth of CPU. On a 2-way Quad Core ESX host this means 8 runaway VMs could use all the CPU. With that said it is important to NOT use “Unlimited” for CPU on Resource Pools.

Memory: Because each VMs max possible memory usage is defined by the per VM configured amount, It’s important that the RP limit amount is the same as the aggregate of all child VMs configured memory amount. If the reservation meets this amount (limit) or less then that is a decision on how much to oversubscribe memory.