Allocation Pool vCPU and memory reservations not as expeccted in vCD 5.1

[PeterBlatherwic]

Hi,

We've been working with vCD 5.1 (5.1.1.868405, Appliance version), and have discovered apparent discrepancies between expect behavior of Allocation Pool vCPU and memory reservations / limits / shares, relative to our expectations and VMware documentation.

Based on documentation od the vCD User Interface, expected behavior of VMs in the Allocation Pool model were:

- vCPU reservation forced to 0 (no reservation) as was the case in vCD 1.0 and 1.5

- vCPU Limit set to vCPU speed configured for the Allocation Pool x number of vCPUs for the VM

- vCPU shares forced to Normal

- Memory reservation set to VM's memory size x % guarantee for the Allocation Pool (*think* that's how it was in vCD 1.x).

- Memory limit set to VM's memory size

- Memory shares forced to Normal

Instead, what we find is:

- vCPU limit is forced to Unlimited (not related to vCPU speed), NOT as expected

- Memory reservation always forced to 0 (no reservation, not related to % guarantee), NOT as expected

- Memory limit always forced to Unlimited (not related to VM memory size), NOT as expected

We tested this by creating VMs with various reservation / limits / shares, then starting them in the Allocation Pool vDC, then inspect actual VM setting in vSphere layer.  No discrepancies were found in either the PAYG or Reservation Pool models.

Has anyone else found this?? Could we be doing something wrong to cause this behaviour?? Is it a known issue??

We need to be able to give our customers good guidance on how to configure under each of the vCD allocation models, including Allocation Pool, in prder to achieve required performance SLAs.

Cheers,

PeterB

[PeterBlatherwic]

Nobody has responded to this yet.  Alas. 

Is anyone else seeing the same (or other) differences relative to what vCLoud 5.1 documentation and the UI say on Allocation Pool vDC model handling of VM / vApp resource reservations??  Known issues?  Work-arounds?  

Need to understand this to set customer expectations on SLA behaviour.  

Cheers,

PeterB

[jkasalzuma]

I am seeing the exact same behavior. Did you ever figure this out?

Thanks,

Joe

[PeterBlatherwic]

Sorry for the late reply Joe.

But, NO, we have not found any work-around, and nobody from VMware has replied.

-- PeterB

[IamTHEvilONE]

http://www.vmware.com/files/pdf/techpaper/vCloud_Director_Resource_Allocation-USLET.pdf

that's one.

If you want 5.1.2 to work like 5.1.1 and 5.1.0 for allcoation pools, you have to enable it under General settings (last setting on the page)

[PeterBlatherwic]

Thanks. 

On a first read of the Resource Allocation Models whitepaper, it does not seem to be agreeing with our findings.  Specifically, vCPU speed seems to have no effect on vCPU limit for the VM (it is always unlimited), % guarantee seems to have no effect on memory reservation for the VM (it is always 0), and VM memory size seems to have no effect on VM memory limit (it is always unlimited).  Maybe I'm just not reading it right -- will have another go.

Actually, I noticed the whitepaper does not show a setting for vCPU speeed at all, as we see in vCD 5.1.  Perhaps screenshots are from an earlier version of vCD?  The paper is about a year old now, but does sort of imply vCD 5.x.

> If you want 5.1.2 to work like 5.1.1 and 5.1.0 for allcoation pools, you have to

> enable it under General settings (last setting on the page)

I am not seeing any configuration for this in our vCD 5.1.1 setup.  I presume only available in 5.1.2? 

Also, are there functional differences with respect to how resourcing is applied in 5.1.2, different from previous versions?  Is 5.1.2 working as expected?

Thanks again

PeterB

[IamTHEvilONE]

So in 5.1.0 and 5.1.1 Allocation Pools are Elastic by default.  This means we track CPU allocations based on a fixed math.  CPU Count * Reservation * CPU Speed = consumption.

In 5.1.2, we added the ability to have vCloud Director mimic a non-Elastic allocation pool.  This is what life was like in 1.5.x generation, and what you might be use to.

We just set the reservation on the Resource Pool Level (from what I recall).  So if you upgrade to 5.1.2, you'll have the option to have Allocation Pools function like they did in 1.5.x.

Please ensure to differentiate the vCloud Org vDC GUI vs that of the actual Resource Pools.  Elastic Allocation vDC Pools do not 'track' CPU usage in the GUI.

[jkasalzuma]

Ya I have looked high and low for a solution. Just as Peter mentioned, the reservation pools in vSphere and the vCloud settings do not match.I have a case open currently and the current recommended fix is to upgrade to 5.1.2.

The concern I have expressed is if the reservations are not set within vSphere, what will happen if I upgrade, the issue gets resolved and now I have reservations set in vSphere. HA admission control could scream as the slot sizes have changed and thus HA may not operate as expected.

This is a pretty scary bug that on the surface has no major effect, but the potential is massive. Obviously a quick short term fix is to manually change the HA slot sizes but that is not a long tern fix.

I welcome you to look at my open SR. The more eyes the better for those of us that are experiencing this issue. When thinking about the consequences of applying a reservation to 1000's of Vms, the simple fix that VMware has been recommending isn't as simple as one would think, and actually might need to be rectified prior to upgrading.

[IamTHEvilONE]

If you are only using a single cluster to back a Provider/Org vDC for allocation pools, the suggestion is to upgrade to 5.1.2.  This will enable SCAP mode, and set the reservations on the resource pool.  We may need to use a tool to perform a remediation for the reservations, but that's pretty straight forward.

This is the release note about SCAP:

• Allocation pool organization virtual datacenters can be elastic or non-elastic: Starting with vCloud Director 5.1.2, system administrators can configure Allocation Pool organization virtual datacenters with Single Cluster Allocation Pool (SCAP), making them non-elastic. This is a global setting that affects all Allocation Pool organization virtual datacenters. By default, Allocation Pool organization virtual datacenters have Single Cluster Allocation Pool enabled. Systems upgraded from vCloud Director 5.1 that have Allocation Pool organization virtual dataceters with virtual machines spanning multiple resource pools have Single Cluster Allocation Pool disabled by default.To change the Single Cluster Allocation Pool setting go to System > Administration > General > Miscellaneous. Before enabling Single Cluster Allocation Pool, migrate any virtual machines on secondary resource pools to the organization virtual datacenter's primary resource pool.

[jkasalzuma]

Ok this adds more questions to the mix.

What tool is it that you mention?

Also, is there a definitive answer on what will occur at the vSphere HA level if SCAP is enabled and or we upgrade from 5.1.1 to 5.1.2?

What happens if we have multiple vSphere clusters backing a PvDC/OvDC?

This looks like a fix for the elastic/non-elastic change in vCloud editions, not really a fix for the current issue with reservations not being seen on the vSphere/VM level in vCD 5.1.1 which is why I am very interested in the tool you mentioned to test what will happen when we upgrade to 5.1.2.

[IamTHEvilONE]

The tool is for support only, and will only verify that reservations are set correctly on the pools and VMs which were deployed by vCloud.

However, given the questions you have, it may be best served by the support team.

I do believe in the Elastic model that the CPU consumption isn't tracked specifically.  The support team should be able to reproduce and verify for you.

[jkasalzuma]

I emailed my support engineer asking about this tool.

Ya I'm not concerned about CPU reservations. My main concern is RAM reservations.

[Techstarts]

Hi,

I've create a blog post for each of the model based on latest version.

Idea was to understand what happens inside vCenter.

It might help you clear your doubts

http://vzare.com/2013/08/30/pay-as-you-go-resource-allocation-model-behind-the-scene/

[jkasalzuma]

To clarify the issue I am seeing using the above PDF linked by IamTHEvilONE.

On page 20 of the "VMware vCloud Director Resource Allocation Models" doc it shows an example of how the reservations on the VM level should be set under an allocation type pool. you will see the % configured on the OvDC settings is the exact % of resources that should be set on the VM level.

So real world example in my environment is if the OvDC is set to 50% reservation of the allocated RAM, a VM with 1024 MB (1 GB) should have a reservation of 512 MB.

This reservation is not occuring as the Document states should happen.

However, the the "vCloud Director Administrator's Guide" (link below) on page 34 makes no mention of the reservation setting on the VM level. Nor does Techstarts blog he links above.

http://pubs.vmware.com/vcd-51/topic/com.vmware.ICbase/PDF/vcd_51_admin_guide.pdf

Now, this means that there are two versions telling a different story about the allocation Pool allocation model. In the example of the admin guide and Techstarts blog my environment is configured and operating correctly. However under the "VMware vCloud Director Resource Allocation models" doc and previous versions of vCloud docs, along with several blogs, my environment would be operating incorrectly by not setting the reservation on the VM level.

I hope the admin guide and Techstarts blogs are correct as I can close a case I have open and nothing is wrong with my environment. That would be Great!

However, if the latter is correct, it appears Peter and I have hit a bug with potentially major side effects.

[IamTHEvilONE]

so this was due to elastic mode being on.

Allocation Pools are elastic in 5.1.1 and 5.1.0 by default, and memory reservations are NOT set on a per VM basis.  The PDF i posted earlier pre-dates 5.1.0's release.

Allocation Pools are NOT elastic in 5.1.2 and anything 1.5.x and prior (not capable), and memory reservations should be set on a per VM basis.

[IamTHEvilONE]

As for the "reporting" in the GUI, the cached metrics weren't updating for other reasons.  If you want to see when they were last updated, you can grep the debug log file as follows:

egrep "Executed cache referesh stored procedure" vcloud-container-debug.log | awk ' {print $1$2 " " $10 " " $15 " " $16 " " $17 " " $18} '

Sample Output:

2013-08-3014:33:37,116 Executed populate_cached_vdc_metrics. Took 33 ms.

2013-08-3014:33:37,152 Executed populate_cached_sc_metrics. Took 35 ms.

2013-08-3014:33:37,175 Executed populate_cached_pvdc_metrics. Took 23 ms.

2013-08-3014:33:37,182 Executed populate_cached_vdc_sp_metrics. Took 7 ms.