VM CPU priority level and impact on host (dynamic shares)

[mjbeddek]

Let me explain,

Say I have two Windows VMs running Citrix and I have them configured with equal shares in CPU/Memory etc. I'll try and simplify below, i'm not really interested in throw more CPU at the problem as I'll just throw more VMs to counter that point of view, I'm interested in whether it's possible to alter the prioritisation / shares on the fly.

On one of the VMs (could be either) a user runs a process that gets a little hot under the collar, and starts to consume far too many CPU resources. Citrix CPU management or another CPU management tool chanmges the priority of the application too low so that other applications can run and appear responsive. Even though the priority level has changed, the server still consumes the full 100% of CPU

This essentially means that only 50% (ok less with overheads) of the physical CPU is available and whilst this server will run with this, it will run better with more if required. This server doesn't have a hanging application and all the processes are running at 'Normal' priority.

Is it possible to ensure that systems running at 'low' priority consume less physical resources than those running at a higher priority? Another alternate is that the shares of the VM get altered on the fly depending on what the VM is running internally and whether the 100% process is configured as low.

Does anyone have such a toolset / script? is it included? Have I made sense?

Thanks for any pointers as it could really help out here.

Mark

[weinstein5]

Remember shares will only come into play if the vmkernel can not deliver the necessary cpu cycles - so depending how your physsical environment is configured shares might not come into play - lets consider your two vms are dual virtual CPU VMs running on a dual quad core ESX host and remembering that the vmkernel will schedule the vcpu on a single core at a time - so with 8 available cores and the vm that needs the cpu will always get it because with 8 available cores the vmkernel should always have a should always have 2 cores available to the vm - so shares would not come into play since there are enough cpu resources -

[mjbeddek]

Ok.

Take this to the next step and that the vmkernel is going to have to start utilising shares, I have 20 VMs running and only 8 cores, and the VMs require more cpu cycles than can physically be delivered at certain points in time.

How can I configure the system so that the VM that's running a low priority process at 100% has less shares than other machines that currently have well behaving processes and are still at the normal level and overall running at 50%? Remembering that I have no way of knowing which VM might be running the '100% low priority process' as this could be anyone of them and depends on who logs onto what and who gets the stuck process. I would also require the number of shares to be returned to normal once the bad process has terminated or finished its work.

Remembering 19 50% + 1100% = 9.5 cores which is > 8 cores.

Thanks

[weinstein5]

Ok using your example - all the vms are fighting for CPU so there is contention - but instead of using percentage of cpu lets use what the vmkernel uses cpu cycles - assuming each core is 3 GHz - so the vmkernel has 24 GHz to distribute between your 20 VMs - if all have equal shares of 1000 then each vm will get 1/20 of the pool or 1.2 GHz - lets say the critical vm - the one you want to make sure has enough CPU cycles - will have a share of 6000 the other 19 each have 1000 shares so your total pool is 25000 shares - which means the critical vm will get 5000/25000 or 20% of the 4.8 GHz but if it is a single vCPU VM it can only run on one core so it will get a full core - the remaining 19 VMs will split the remaing 21 GHz - each getting 1000/25000 or 4% 840 MHz -

[mjbeddek]

OK, I can see this going backwards and forwards.

I don't have one VM that is more important than any of the others, ALL are equal except when one/two/three... (ANY one/2/3...) starts running a process with a 'low' priority that is using ALL (pedantic: nearly ALL) CPU within that guest.

Now, let's use GHz, as you said assume the VMkernel has 24GHz to allocate and I have 20 machines and they are running under contention for resources, none of the machines at present have a higher reserve or greater number of shares (1000). Now let's say machines 1-17 are running processes that are behaving themselves though the machines are quite CPU intensive.

Now all machines are created equal so all machines have 1.2GHz of CPU and they are using it however they would run better if they can use more (please don't come back and say I need another host:).

Now, does anyone have a script that can be run when a machine has one of these 'low' priority processes that is using 1.2GHz of the host CPU so that the shares on that VM are lowered. Say for example to 500.

In this instance I then have 3 (arbitrary number of VMs with bad low priority process, remember this can be any number of machines, I don't know which or how many until it happens and will change during the course of the day) * 500 (share has been dynamically lowered) + 17 (number of nicely behaved machines) * 1.32GHz ((24000-1500) / 17) = 24GHz.

3 * 500 + 17 * 1323 = 24000

As you can see the well behaving VMs now have 1.32GHz and suffer less because of these VMs with badly behaving processes causing contention.

Thanks for your help, hope this explains what I'm trying to achieve a little better.

[weinstein5]

I guess the question how can you identify that that it is a low priotity process that is using the ntire cpu or one of the regular processes? unless you can do this and feed to mechanism that can change the shares you run the risk of dropping shares to vm that actually needs them -

Also what you describe is 20 cpu intensive vms whether running low or high priority process is not really an ideal situation for virtualization - remember the idea behind virtualization is to balance the load of virtual machines that utilize at different at times one of the core four resources (CPU, Memory, Network, Disk) virtualized by the VMkernel - so with that being said if you want to run 20 cpu intensive on the same host - you are going to need enough cores to handle the load - or - and here it comes - another server

[mjbeddek]

I can do that using standard processes, in fact I have CPU management tools that change a process to 'low' when the situation occurs and can configure this to automatically run a script if required, just need to know if it is possible on the VMware side of things.

Something more inherent in VMware would be nice but if there is a VMware level script, API or something that someone has written I can wrap this into my tool.

Thanks

Mark