Resource Pool Setup Question

[elihuj]

We are currently not using resource pools within our environment. What we are looking to do is implement resource pools with memory reservations based upon our VM sizes. For example, we'll create a 4GB resource pool for VMs with 4GB of memory, an 8GB resource pool for VMs with 8GB of memory, etc. We would then set the memory reservation to half (of the VM) for each pool. So the 4GB VM will have a 2GB reservation. Our aim in doing this is reduce the size of our swap files. We have a few VMs spread out in our environment that are independently configured. For example, a 12GB VM with a 6GB reservation has a 6GB swap file. On a global level, this could save us a lot of datastore space.

A few questions in doing this though. What will happen for VMs that already have reservations enabled? Does the resource pool override individual VM settings?

And does anyone see this as having any adverse effects for an HA event? We have had issues in the past in a different environment with reservations (large 38GB+ reservations), and when we lost a host VMs were unable to power on due to there being no more resources available.

[weinstein5 · Accepted solution]

The resource pool reservations have no affect on the VM's reservation in the pool - they are independent

The problems you had with HA is one issue - another is by assigning reservations to all VM's you can have come to the point where an ESXi host will not have enough memory to satisfy the reservation causing the VM not to start.

[elihuj]

If we have an 8 host cluster, each using about 90% of its total 132GB of memory, and the VMs are configured in resource pools to reserve about half of their memory, that means we would need about ~60GB (90% of 132 = 118/2) of physical memory available to the cluster for all the VMs to stay powered on in the event of a failover then correct?

[VirtuallyMikeB]

Essentially, yes.  If you're using the percentage-based admission control policy (you should be), then simply use the following trivial formula, set, and forget.  The good folks at VMware have done the work for you.  The percentage-based admission control policy does all the figuring for you on the backend, including taking into consideration your VM and resource pool reservations and VM and system overheads.  It's hard to ask for more.

% reserved = ( number of host failures cluster tolerates / number of hosts in cluster ) x 100

Be sure to round up to the nearest percent and you're done!  Again, all the magic happens behind the curtain.

And just to be clear (emphasis mine),

"...we would need about 66GB of physical memory available to the cluster for all the VMs to stay powered on in the event of a failover then correct?

in an HA event, your VMs would be restarted on another host so those VMs would not stay powered on at all.  You probably know this but again, just trying to clarify.

All the best,

Mike

-----------------------------------------

Please consider marking this answer "correct" or "helpful" if you found it useful.

Mike Brown

VMware, Cisco Data Center, and NetApp dude

Consulting Engineer

michael.b.brown3@gmail.com

Twitter: @VirtuallyMikeB

Blog: http://VirtuallyMikeBrown.com

LinkedIn: http://LinkedIn.com/in/michaelbbrown

[weinstein5]

Using you example of an 8 node cluster 1ith 132 GB total - for a 4 GB VM with a 2 GB reservation there will need to be 2 GB of free RAM on a single host for that VM to start in the cluster - so now let us assume that each host uses 4 GB of RAM that means there will be 100 GB of RAM available for VMs - so theoretically using the above VM configuration you will only be able to start 50 VMs in the cluster.

Now lets work  through 2 resource pools - one RP (RP1) with 20 GB reservation and the other with 40 GB (RP2) reservation leaving 40 GB available to the cluster - the other key is that the resource pools are configured without expandable reservations - so this means that when a machine is powered in will reserve its 2 GB from the reservation in the resource pool - this means that in RP1 you will be able to start 10 VMs and in RP2 you will be able start 20 VMs - because the RPs are configured to not use expandable reservations  40GB that are available to the cluster will not be used. If expandable reservations are enabled the VM will be able to use the additional cluster memory.

[Gav0]

I don't think Resource Pools will solve your problem (If I understand what you are asking)

EG - Resource Pool 4GB - This Resource Pool contains 4 VM's each with a 4GB memory entitlement.  So you set the memory reservation of the pool to 8GB (half the total memory requirements of the VM's).

This will not allocate each VM a 2GB reservation (thus reducing the swap file size of each VM by 2GB).  The reservation guarantees this memory is available to any VM's in the Resource Pool and will satisfy any child reservations - it doesn't pre-allocate it to the VM's at boot up so the swap files will still be created.

To illustrate I created a Resource Pool with a 1 GB memory reservation.  Then created VM1 (512 MB reservation) & VM2 (no reservation) and powered them on (no OS installed). (Edit each VM has 512mb of memory configured)

As you can see, VM2 with no reservation defined at the VM level still has its full swap file value despite there being reservation remaining in the Resource pool:

To achieve your goal I think you will need to set a 50% reservation at the VM level.

[elihuj]

Thank you all for the replies. I was under the impression that setting a resource pool to have, say a 4GB reservation would set all VMs within the pool to have a 4GB reservation. You guys helped me to understand that resource pools help guarantee memory for all VMs within the pool, and do not pre-allocate on a per-VM basis. So it does not appear that this plan will work as I imagined.

Would this be accurate then? Resource Pool 1 has a 20GB reservation (no expandable reservations). We power on 5 VMs 4GB memory. Each VM will reserve its full 4GB from the pool. We add a 6th VM with 4GB. Will this VM power on and just not have reserved memory? Or will it not power on at all?

Mike - we are currently NOT using admission control. If you wouldn't mind, could I pm you about this? I have a few questions that fall outside the scope of this question.

Weistein5 - Thank you for your explanation. It does sound like we would run into some issues in powering on VMs if we go with the reservation route. Are there any recommendations or best practices to reduce the swap size for VMs?

Gav0 - Thank you for your examples. When you say setting a 50% reservation at the VM level, you do mean going to each VM and setting this correct?

Thank you again for your help.

[Gav0]

Gav0 - Thank you for your examples. When you say setting a 50% reservation at the VM level, you do mean going to each VM and setting this correct?

If your objective is to save disk space by reducing the size of the swap files then yes, you will need to configure the reservation for each VM.  It should be fairly easy to script this.

Depending on your storage vendor you could perhaps set up a dedicated Thin Provisioned LUN and configure the cluster so that all swap files are stored on that lun (no VM's just the swap files).  Ensuring all hosts have access to it will negate any vmotion penatlies (eg - having to recreate the swap file etc).  This is a fairly common scenario where you have configured replication or array based snapshots and there is no value replicating/backing the swap files.

[elihuj]

Thank you for that explanation Gav0. I do like your idea about a LUN dedicated to swap files. We could potentially run into problems with VMs starting if the datastore were to be lost though correct?

And in regards to what I said previously about reservations.. Resource Pool 1 has a 20GB reservation (no expandable reservations). We power on 5 VMs 4GB memory. Each VM will reserve its full 4GB from the pool. We add a 6th VM with 4GB. Will this VM power on and just not have reserved memory? Or will it not power on at all?

If we were to enable expandable reservations, the VMs could then pull memory from the remaining resources available to the cluster correct? (until the cluster runs out of resources)

[Gav0]

If the lun datastore is not available then ESXi will default to storing the swap files with the VM files, so as long as their is space on the VM datastore they will power up.  If the lun were to go oflline while the vms are running this might cause an issue but not something i've tested (yet).

I guess its down to how confident you are with your SAN and redundancy (multiple paths, switched etc).  We use Netapp and want to extend the retention of our snapshots and this approach is in the Netapp & VMware best practice guide so i'll be doing some testing soon.

Regarding your other qustion the answer is yes.  VM's with reservations will take their memory from the memory that is reserved for the parent Resource Pool.  If expandable reservations are ticked then if you add that 6th VM then the Resource Pool will take another 4GB from the cluster (or its parent - could be another resource pool) - the memory reservation of the Resource Pool will then change to 24GB.

[elihuj]

Nice, thank you Gav0. So with expandable reservations enabled, any of the VMs within the resource pool will be guaranteed a reserved amount of physical memory. If the VM requests memory outside of what the pool is allocated, it will pull memory from the remaining host resources and increase the reservation of the pool. This will continue until the host is out of resources then correct?

And I'm no longer leaning towards setting up reservations in this fashion. But if we did, and had 6 VMs (4GB/each) within a 20GB pool, there would be one VM with a swap size of 4GB correct? The other VMs wouldn't have swap files because their memory is reserved.

Thank you for your patience on this, much appreciated.

[weinstein5]

Nice, thank you Gav0. So with expandable reservations enabled, any of the VMs within the resource pool will be guaranteed a reserved amount of physical memory. If the VM requests memory outside of what the pool is allocated, it will pull memory from the remaining host resources and increase the reservation of the pool. This will continue until the host is out of resources then correct?

That is pretty much correct but the reservation on the Resource Pool will not be increased - it will continue until the resource on the host runout.

And I'm no longer leaning towards setting up reservations in this fashion. But if we did, and had 6 VMs (4GB/each) within a 20GB pool, there would be one VM with a swap size of 4GB correct? The other VMs wouldn't have swap files because their memory is reserved.

No - If you have 6 VM each having a reservation of 4 GB that means you will need 24 GB of physical RAM to support the reservation - so if you only have 20 GB available you will be able to start 5 of the 6 machines - when you attempt to start the 6th machine you will receive an error of insufficient resources - Remember the vmkernel swap file is equal to the amount of memory assigned to the VM minus the amount of memory reserved so the 5 machines that do start might or might not have a swap file depending on the amount of RAM assigned - if they are assign 4 GB of RAM there will be no swap file created when the VM is powered on but if the VMs are assigned 8 GB of  RAM each VM will have a swap file of 4 GB created when the VM is powered on -

When I taught for VMware I would use the following analogy to describe reservations - lets say you have a restaurant with 100 seats - the vmkernel is the manger and it will try to seat everyone in the restaurant - a guest (VM) wants to reserve 10 seats - the manager looks in the room and sees he has 10 seats unreserved and reserves them for the VM - if the VM does not use all the seats the empty will seats will be used by other VMs - 9 more VMs want reserve 10 seats that means all 100 seats are reserved - well lets look at what happens with a few scenarios -

1. All seats are reserved and another VM comes in with out a reservation - as long as there are empty seats in the dining room the VMs will be seated in the dining room
2. All the VMs that have reservation are using all of their reserved seats and VM without a reservation wants to come in - it will be allowed in but will have to sit one the patio in other words it will use its VMkernel Swap File -
3. All seats are reserved - the reserved seats might or might not be filled - a VM comes in with a reservation and request a reservation without expandable reservations enabled the manager will turn away the new VM and it will not start
4. All seats are reserved - the reserved seats might or might not be filled - a VM comes in and request a reservation with expandable reservations enabled the manager will call to the upstairs dining room (this is parent resource pool or top level of the cluster) to see if seats that are not reserved - if there are seats available the VM will come (start) - if there are not seats available then the VM will not start

Message was edited by: David Weinstein adding a description of how reservations function -

[Gav0]

Nice, thank you Gav0. So with expandable reservations enabled, any of the VMs within the resource pool will be guaranteed a reserved amount of physical memory. If the VM requests memory outside of what the pool is allocated, it will pull memory from the remaining host resources and increase the reservation of the pool. This will continue until the host is out of resources then correct?

Just to follow up on this point here's an example:

Resource Pool 1 with a memory reservation of 2400MB contains 3 VMs (VMTEST1,2 & 3) each with 1024MB of memory and a 1024MB memory reservation configured on the VM.  Expandable Reservations are disabled on the Resource Pool.

We can power up VMTEST1 & VMTEST2 but not VMTEST3 as there isnt enough Available Reservation (76MB):

Now lets turn on Expandable Reservation and we can see we now have memory resource available from the cluster (118091MB) to satisfy any VM reservations:

So if we power up VMTEST3 you will see the Used Reservation increase and the Available Reservation decrease (as the cluster now has less to offer):

You can see that the Configured Reservation (2400MB) doesnt change, only the Used Reservation on the Resouirce Allocation tab of the Resource Pool (3498MB).

If you Edit Settings of the Resource Pool you will see that the memory reservation value has changed (3498MB - and cannot be reduced while the reserved memory is in use):

This will revert back to its original configured value when the Resouce Pool no longer needs to borrow memory from the cluster (or other parent) - EG VM's are powered off.

[elihuj]

weinstein5 - Awesome analogy! That really paints a picture the way you described it.

Gav0 - Thank you for the examples. Especially with expanadable reservations.. that really helped me to see what happens when you use or do not use them.

Thank you again to everyone who contributed to this thread.