1 2 Previous Next 19 Replies Latest reply on May 19, 2017 7:53 AM by thgreyprnc

    CPU reservations irrelevant - core scheduling more important

    VMwareBL Novice

      I am having a bit of a problem wraping my head around the whole resource reservations when it comes to CPUs and resource pools and what VMware is doing here basing it on Ghz. Memory is straight forward because you have certian amount and can allocated out of that until its used up but CPUs is hard to grasp since only one process can allocate at the time even if only 100Mhz the rest is not used.

       

      The CPU has two areas of concer as i see it "the clock speed known as Ghz" and "core schdule".  Lets say you have 4 core CPU with 4 Ghz for simplicity.  And i create two resource pools one with 3Ghz and other 1Ghz. How if any way does this affect scheduling of the cores? Thos wont make sure that 3 cores are always ready for one pool and one core for the other as i want.  What's the point of the Ghz  reservation unless you have massively high CPU consuming apps running?  When the core is scheduled to execute and is busy executing even 100Mhz the rest of the core capability is wasted and can't be allocated to something else as two processes can't execute at same time! So to me ghz is irrelevant when thinking from that perspective. The part that really counts is core scheduling.  This i found true even in my environment where i has 10% utalization yet system slooooooooow because of core overcommitment. I had 24vCPUs and assigned 38 to VMs. So even though the Ghz was at close to zero the VMs were sky high in "ready' time becasue there was no free core to execute in! Once that had core busy were using only little mghz but might as well have been using everything as its irrelevant. You see my point?  I have over 50 clients and each one of them has 10% CPU utalization yet close to perofrmance limit becasue of core allocation not Ghz!  If i add more VMs ready time sky rockets and performance drops like hell.

       

      Scenario: I want to split the server half for lab and half for production and i set 3Ghz for Production and 1Ghz for Lab using same CPU specs (4Ghz 4 core). Well my lab can still be grinding my production server to a halt if its scheudled accross all cores but not high clock!  Only way around that would be shares i think!  but even then there is no way to guarantee cores for certain pool right?  Like you can't create a Lab environment and make sure it never takes up more then certain amount of cores?

        • 1. Re: CPU reservations irrelevant - core scheduling more important
          weinstein5 Guru

          Welco me to the Community - the reservation is measure in GHz so that it is easier to understand and allow users to visualize on how much of the pool cycles is being reserved -

           

          The first point is a reservation insures there are going to be many cycles available for the vm/pool - so for your example if you a VM in the 3 GHz pool using 100 MHz pool - the rremainiing 3.9 GHz is available for any vm - including those in the 1 GHz pool - now as the VM starts using more cycles it will be taken away from other VMs until the VM in the 1 GHz pool - and if both VMs are using their reserved and require more cycle the CPU ready time will increase

           

          Shares will definitely a better way to because with reservations you can run in to situations where you will not be able to power a vm up -

          • 2. Re: CPU reservations irrelevant - core scheduling more important
            VMwareBL Novice

            Thanks! Take it you noticed its the 1st post ...

             

            See that's the problem though! Your comment: "so for your example if you a VM in the 3 GHz pool using 100 MHz pool - the rremainiing 3.9 GHz is available for any vm" The remnaining 3.9Ghz is NOT available to any other VM. Not if your 100Mhz is being executed by 4 threads on 4 cores! Then you may have 3.9Ghz of free Ghz but nothing can get to it!

             

            So if you have a VM in lab which has 4 cores and running multi threaded application and using only 400Ghz lest say, you think you have 3Ghz for your other VM except you don't becasue all of the 4 cores are busy executing the 400Ghz by the lab VM.  That's what i mean. The Ghz is nowhere near important as the free core and make sure there is a free one in my opinion and that you can't do unless you ensure that you dont over provision your cores.

             

            Also on another point, why would you even want to limit Ghz! This measure is one which in more basic terms is the amount of instructions your computer can carry out.  Thus why limit it, wouldn't it make more sense to leave it so that whatever thread is exeuting uses what it needs to finish faster and free up the core. Afterall as i said you can have two different threads using 3 Ghz core at same time 1.5Ghz each. Schedulign doesn't work like that nor does the CPU in general.

            • 3. Re: CPU reservations irrelevant - core scheduling more important
              Chris Wahl Master

              Hertz are a measurement of time. The CPU reservation/limit relationship is how much time the vmkernel scheduler allows the VM to run on a given core with respect to the clock rate of that core. This can be seen with ESXTOP when you look at RDY values for a limited VM.

              VCDX #104 (DCV, NV) ஃ WahlNetwork.com ஃ @ChrisWahl ஃ Author, Networking for VMware Administrators
              • 4. Re: CPU reservations irrelevant - core scheduling more important
                VMwareBL Novice

                I don't think so. I disagree with that as CPU Hrz is not a measure of time but cycles and is relevant to other varable. If you trully believe so can you find me an article proving so and i mean that with utmost respect couse i would like to know for sure but I think your wrong in this case. I have read ton of articles to support the argument. Here first one that came up on google.

                 

                http://wiki.answers.com/Q/Is_speed_of_CPU_is_measured_in_hertz

                 

                Maybe VMware made resource pools and reservations treat it as such but that's not what it is!  And real life scenarios prove otherswise as well.  I have 4Ghz CPU with 4 cores. I have a 2 VMs each with 4 cores (total 8) and one is in pool for 3Ghz other in 1Ghz. The VMs will have to wait on each other i dont care how you set the Ghz reservations.   The whole point behind this is that you can't use this reservation to guarantee perfromance to something. Maybe in ratio to another VM yet but not to guarantee it will not have performance hit!

                 

                Like the shares make perfect sense to me as its priority to get scheduled to the CPU but this reservations make none. i'm either missing something or there really is no point.  Its not like memory or storage you have x capacity and you allocate until full. core is capable of 1Ghz lets say and only 1 process can access it wether its using 1mhz or 1ghz so if its scheduled its scehduled. if i reserve 1ghz but grant 4 cores and have application using all 4 cores at low Mhz the other pools is blocked as cores are busy regardless that Ghz are free.

                • 5. Re: CPU reservations irrelevant - core scheduling more important
                  Chris Wahl Master

                  Frank has written quite extensively on this subject.

                   

                  http://frankdenneman.nl/2010/06/reservations-and-cpu-scheduling/

                   

                  Additionally, there is a rather complete white paper on the subject from vSphere 4 entitled "CPU Scheduler in VMware ESX 4.1"

                   

                  http://www.vmware.com/files/pdf/techpaper/VMW_vSphere41_cpu_schedule_ESX.pdf

                   

                  Cheers.

                  VCDX #104 (DCV, NV) ஃ WahlNetwork.com ஃ @ChrisWahl ஃ Author, Networking for VMware Administrators
                  • 6. Re: CPU reservations irrelevant - core scheduling more important
                    VMwareBL Novice

                    see that's my problem, his article doesn't help but adds to the confusion for me...

                     

                    quote: "Limit: A limit is a mechanism to restrict physical  resource usage of the virtual machine. A limit ensures that the VM will  never receive more CPU cycles than specified, even if extra cycles are  available on the host."

                     

                    Why would you limit a VM to 500Mhz  on a 1Ghz core when only one thread can access at a time anyway's?

                     

                    quote: "Now reservations act differently when setting it on a CPU than setting  it on memory. When the virtual machine does not use its CPU cycles,  these CPU cycles are redistributed to other active virtual machines"

                     

                    if this statement it true then the first quote makes sense as difference is redistributed but again, how can it be when the core is busy though? i reserve 500Mhz on a 1Ghz core and i have 2 cores executing 500mhz each .. that can't be. Maybe one after another but one is waiting while other competes if its executing at same time.

                     

                     

                    also another thing that comes to mind. if you have 4 cores 1ghz each and you allocate a VM with a single vCPU logic says the most it can ever use is 1Ghz since that underlying CPU hardware capacity and you gave it only 1 CPU? Would that be accurate to say or is it false assumption in VMware and this VM can have 4Ghz though that single vCPU by switching cores?

                     

                    I hear all the arguments but none of them address my question directly hence why i keep going back to the same question. Simple scenario:

                     

                    you have a pool with 4 Ghz and 4 core CPU supporting it. You create 2 VMs and one you allocate 3Ghz reservation but you give it 2 cores (i don't even know if that would make sense) then on second VM you give it 4 cores and only 1Ghz reservation. When the second VM is executing its 4 threads (assuming its multi threaded app) the first VM that has 3Ghz reservation has to wait as core is busy regardless that this one is limited to 1Ghz. so its not a "real" reservation but rather higher priority if anything to me.

                     

                    GUYS THANKS SO MUCH for discussing this with me as i can't seem to find the answer anywhere and i know your just trying to answer even though i might come off "argumentative" i assure you its not i just want to get to the bottom of this so that i can understand it.

                    • 7. Re: CPU reservations irrelevant - core scheduling more important
                      dwaynesinclair Lurker
                      vExpertVMware Employees

                      Why would you limit a VM to 500Mhz  on a 1Ghz core when only one thread can access at a time anyway's?

                       

                      It's an example and the 1Ghz single core is for illistrative purposes only. VMware Hypervisor is virtualizing the physical 1 Core/ 1Ghz CPU and providing a virtual 1Ghz vCPU to multiple VM's. Multiple VM's would each have a 1Ghz vCPU and their respective OS's would be capable of running up to 1Ghz but the Hypervisor limit will restrict CPU utilization to 500Mhz.

                       

                      Multiple VM's each with a 500Mhz CPU limit would happily share the single physical 1Ghz core assuming all VM's are not running at 100% all at the same time but no one VM could run more than 500Mhz. More than two VM's running simutaniously at 500Mhz would exceed the physical capability of the process and the Hypervisor would now share the physical 1Ghz CPU so 4 VM's running at 100% would each get 250Mhz each.

                       

                      The reality is -

                       

                      1. when CPU limits are placed on VM's, the physical CPU's are faster (eg 3Ghz) (Multicore (4/8/12/etc), and support Hyperthreadiing thus the limits are relevent to the VM and not the physical machine. Larger ESXi hypervisor servers can be built to ~3Ghz/40 Cores/512GB RAM+.

                       

                      2. Irrespective of a VM's memory allocation, limit, and reservation, all VM's dont run at 100% simutaniously.

                       

                      quote: "Now reservations act differently when setting it on a CPU than setting  it on memory. When the virtual machine does not use its CPU cycles,  these CPU cycles are redistributed to other active virtual machines"


                      if this statement it true then the first quote makes sense as difference is redistributed but again, how can it be when the core is busy though? i reserve 500Mhz on a 1Ghz core and i have 2 cores executing 500mhz each .. that can't be. Maybe one after another but one is waiting while other competes if its executing at same time.

                       

                      Reservations are not limits. Reservations as they pertain to the above example ensures the Hypervisor has 500Mhz of CPU processing power for the VM whether it uses it or not.

                       

                      In larger VM environments (100's-10,000 VM's) and because of the capabilities of multisocket, multicore servers, physical RAM becomes the linmiting factor well before CPU.

                       

                      From...

                      http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50-resource-management-guide.pdf

                       

                      vCenter Server or ESXi allows you to power on a virtual machine only if there are enough unreserved resources to satisfy the reservation of the virtual machine. The server guarantees that amount even when the physical server is heavily loaded. The reservation is expressed in concrete units (megahertz or megabytes).

                      For example, assume you have 2GHz available and specify a reservation of 1GHz for VM1 and 1GHz for VM2. Now each virtual machine is guaranteed to get 1GHz if it needs it. However, if VM1 is using only 500MHz, VM2 can use 1.5GHz.

                      Reservation defaults to 0. You can specify a reservation if you need to guarantee that the minimum required amounts of CPU or memory are always available for the virtual machine.

                      • 8. Re: CPU reservations irrelevant - core scheduling more important
                        VMwareBL Novice

                        First let me thank you for your response.  I understand all that in the context of text but not real life. I just don't see how "CPU reservation" is  areservation and guarantee at all. Maybe if you answer this for me it will help.

                         

                        if you have 4Ghz CPU with 4 cores lets say. Then i create 4 VMs each with 1Ghz reservation. Thus i am saying i guarantee this VM will have 1Ghz. Then i assign each VM with 4 cores so i exssentialy overallocated my cores and have 12 cores assigned when i have only 4 (assume no hyper threading). At this point how can i guarantee that one of these VMs will have CPU to run on and 1 Ghz if my other VM is running a hyper threaded app executing on all 4 cores but at very low hrz?  It can't. The VM which has 1Ghz reservation will have it CPU ready time shoot up indicating that its waiting and thus performance is impacted thus to me your not reserving CPU and guaranteeing it will  be there.  If the core is busy its irrelevant if its executing at 2hz or 1 ghz as one thread has access and that's it.


                        Thanks so much!

                         

                        Added: Also if i have overallocated cores but i want to ensure that one VM is not impacted, i really can't with CPU reservations only wtih shares i can ensuring it will be shceduled with higher priority then other VMs. At least that's what makes sense to me or seems to be like as when i set reservation on a VM in overallocated environment i still have CPOU ready go up and impact on that vm.

                        • 9. Re: CPU reservations irrelevant - core scheduling more important
                          dwaynesinclair Lurker
                          VMware EmployeesvExpert

                          I just don't see how "CPU reservation" is a reservation and guarantee at all. Maybe if you answer this for me it will help.

                           

                          Depending on if you are viewing your environment from vSphere vCenter or stand alone directly connected to the Hypervisor, Servers and Clusters have a tab called Resource Allocation which details the available resources for Memory and CPU, total capacity, reserved capacity. available capacity.

                           

                          Its a reservation because new VM's will not start if the environment does not have enough resources to meet the reservation. If 10Ghz CPU is available and 7.5Ghz is reserved, you cannot start a new VM if its reservation is greater than 2.5Ghz irrespective of how much the three existing and one new VM's actually use.

                           

                          Reservations/limits are normally applied on specific VM's or on Pools so that reservations are spread across all VM's within a pool. Ideally, the CPU and memory capabilities of your virtualized environment will meet your requirements. Limits and Reservations will assist to ensure the outlyers are controlled and guarenteed.

                           

                          I would recommend readling the link provided as well as "testing" various scenarios using a VM and resource pools on ESXi - ESXi can be virtualized under VMware Workstation or Fusion for personal testing/education.

                          • 10. Re: CPU reservations irrelevant - core scheduling more important
                            VMwareBL Novice

                            I totaly understand that! You can't reserver 20Ghz when you have 10Ghz and expect to power up those extra VMs past 10Ghz. However, what does that have to do with ensuring that a VM will have CPU resources to execute on when it needs to?


                            I understand it have implimation on being able to power up vms - but does the reservation guarantee that VM will have CPU to execute on when it needs to?  For me the answer is no as it all depends if the cores are busy executing threads form another VM which is what i tried to explain in the question above with high overallocation of cores.  If i reserve 1Ghz and i have to execute i am not guaranteed i will be able to right away unde the above scenario right? 

                            • 11. Re: CPU reservations irrelevant - core scheduling more important
                              dwaynesinclair Lurker
                              VMware EmployeesvExpert

                              Does the reservation guarantee that VM will have CPU to execute on when it needs to?  For me the answer is no as it all depends if the cores are busy executing threads form another VM which is what i tried to explain in the question above with high overallocation of cores.  If i reserve 1Ghz and i have to execute i am not guaranteed i will be able to right away unde the above scenario right?

                               

                              Virtualized environemts will be memory constrained well before CPU constrained. I'm not sure where you are at with your virtualization journey but I want to share with you I have performed many VMware Capacity Planning assessments and what I commonly see is average physical CPU utilization of around 1%-2% and max 5%-10% across physical environments.

                               

                              The answer to your questions is yes - reservations are honored and cpu will be available to meet the reservation. When the reservation is honored and the VM starts, the reservation will always be available to the VM. Other VM's without reservations will be dynamically constrained (limited) based on the reservation and remaining/available resources.

                               

                              Statistics on CPU scheduling are maintained by vSphere and certainly in a very small/ very constained environment there could be the possiblity of scheduling delays but real world environemnts are not likely to be constrained in this manner.

                               

                              As a test, use the Nostalgia VM available in the VA Marketplace as this in an old DOS app which uses all availale CPU you can dyanmically apply a limit to constrain the VM down to whatever CPU limit you like while the machine is running - This is what the Hypervisor is doing realtime in constrained environments.

                              • 12. Re: CPU reservations irrelevant - core scheduling more important
                                VMwareBL Novice

                                Actually i am a Lead Consultant with all the certifications in VMware and years of experiance. Problem here is that i am asking one question but other is being answered though so my question might not be clear.

                                 

                                Quote ' I have performed many VMware Capacity Planning assessments and what I commonly see is average physical CPU utilization of around 1%-2% and max 5%-10% across physical environments.'

                                 

                                That's true if your looking at the Ghz performance metrics. Which is EXACTLLY why i have this problem.  However, look at your CPU ready counter! you can have 5% CPU utalization but have MAJOR performance impact if your cores are overallocated!   This is a fact!

                                 

                                quote 'The answer to your questions is yes - reservations are honored and cpu will be available to meet the reservation'

                                 

                                My question was not if reservations are honored, my question was "if a vm wants to execute will there be CPU ready for it" - answer is no if you have overallocation in cores regardless of CPU reservations.

                                • 13. Re: CPU reservations irrelevant - core scheduling more important
                                  VMwareBL Novice

                                  Maybe this will help as its reall world scenario  which in part raised this thread:

                                   

                                  A month ago i went to see a new client who reports having major perofmrance issues.  I looked at his VC and noticed he is running lab and production in same cluster (which we can all agree on is not smart).  Anyways his comment was "but i created two resource groups and i have production 80% reservation while i gave lab only 20%" .  Meanwhile he had over 50 VMs in lab and only 20 production! He had allocated over 100 cores on a system that has only fraction of that. When i told him its the CPU, he said (which by the way is what you were saying earlier)  but look at the performance tab for CPU, its not even 10%!  Meanwhile his CPU ready counter was showing 1000's in ESXTOP!

                                   

                                  So while the ghz might be 5% it means nothing to your performance if your cores are overallocated and reservation will not guarantee you perfromance.

                                  • 14. Re: CPU reservations irrelevant - core scheduling more important
                                    dwaynesinclair Lurker
                                    VMware EmployeesvExpert

                                    Thanks so much for the clarification. Certainly specific workloads are optimized for more/smaller cores vs less/greater cpu so 10Ghz could consist of 4 Cores/2.5Ghz vs 10 Cores/1 Ghz. Reservations and Limits don't address core/workload optimization - this needs to occur at design and implementation. Today, CPU Affinity combined HT Sharing can help to assist but depending on the workload, a cluster that is architecturely optimized to meet the specific workload would help.

                                     

                                    Given the example you provided, appropriate CPU resources need to be allocated to respective environents. It's also important to be sure VM's are correctly sized (minimize vCPUs where possible).

                                     

                                    I do not know if ESX(i) is running on a NUMA server but I have attached the folowing kb which is a good match for the scenario you detailed - first validate your customer is on current versions of the hypervisor...

                                     

                                    http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1026063&sliceId=2&docTypeID=DT_KB_1_1&dialogID=380638546&stateId=1%200%20380656302

                                     

                                    http://blogs.vmware.com/vsphere/2012/02/vspherenuma-loadbalancing.html

                                     

                                    As the VMware environment scales, core optimized worlodads mixed with general purpose workloads is fine given the size provides spread. I would encourage you to get a vSphere Hypervisor futures update from your VMware account team.

                                     

                                    Thanks.

                                    1 2 Previous Next