Throttle IO on a virtual machine - is this possible?

[intermax]

Hi,

vSphere is perfectly capable of limiting CPU, memory and network traffic but I cannot find any option / documentation on how to limit a VM's disk IO. We are using NFS as the storage backend, so I cannot configure shares. By the way, configuring shares is also not exactly what I want; I want to be able to limit a VM for example to have a maximum of 500 IO operations per second, not more.

The reason I'm asking this is because we have a lot of VMs on a few NFS datastores; there are a few VMs which generate so much IO that the other VMs are experiencing delays etc.

Is this possible somehow? Maybe by tweaking the vSCSI adapter?

Thanks,

Niels

[vmroyale · Accepted solution]

Hello and welcome to the forums.

At this time, there is no way to throttle IOps on the disk the way you describe. Probably asking the obvious here, but have you identified the source of the IO in the virtual machines to make sure that it can't be stopped? AV scans, defrags, bulk copies, etc could be factoring in.

Good Luck!

[RParker]

The reason I'm asking this is because we have a lot of VMs on a few NFS datastores; there are a few VMs which generate so much IO that the other VMs are experiencing delays etc.

This is not really the direction you should be heading, you should be looking at increasing the IO and bandwidth on your NFS server, because if your VM's are suffering that means you either don't have the right disks for the job, or something is seriously wrong with your NFS store. You never want to throttle IO, that can have adverse affects on many things inside of a guest OS, you want to made the shared storage as robust as possible, so your problem isn't the VM's taking too much IO, your problem is not enough IO to go around, and that's a big problem.

[intermax]

Thank you for your replys. Currently we are indeed in the process of upgrading the amount of disks on our NetApp filer. However, we have exactly -one- VM (out of 150) which is making around 2k-3k IOps, where all others are < 100 IOps. I was just wondering if there are options to throttle this particular VM... because now the extra shelfs are just to satisfy the 'misbehaviour' of this VM, which sounds silly. But that just isn't possible then I guess.

Of course I can create a separate volume for this VM on our storage layer and give this volume a lower priority than other volumes, but that isn't a very simple approach. Besides, the amount of NFS stores is limited.

[exto]

I thought I would add to this post as I think it is very interesting (and necessary), I would love to find an answer to this.

I have a very similar issue.    First I want to help everyone understand why I feel it is necessary to throttle IO.

In certain circumstances there are rogue processes or perhaps batch processes that will take pretty much ALL the IOps that are available to it...  

Let's say I have a very large database in a "test" environment.   Occasionally developers (or just bad code) may run against this database generating huge IOps.   I want to ensure that a particular "test" environment does not "hog" all the IO from our shared storage and thus affect other applications.    I feel the best way to do this is give a maximum IOps limit to the guest VM, this way the "test" environment will get consistent runtimes and the storage administrator does not get any surprises when rogue processes are running on our vms.   

To me this is very similar to the reason you would want to throttle CPU's, you want to ensure a consistent level of Service !

I see this being benefitial mostly for Database file systems, I would not plan on doing this on the operating system filesystem.

Does anyone out there have any ideas on how to do this at the Disk Level in vSphere?

Of note, I see that in RHEL 6.0 there is a feature called cgroups which allow you to put a limit on the IOps per volume.   Haven't tried this out yet as i'm still running RHEL 5.5

[depping]

The limit that you can set within your VM on a disk level should work. The local disk scheduler handles this one and should throttle your workload to what has been specified.

Duncan (VCDX)

Available now on Amazon: vSphere 4.1 HA and DRS technical deepdive

[exto]

I assume you are referring to the "Limit - IOPs" option on the Resources/Disk section of the VM Settings.

I have tried setting this to a very low level and enabling Storage IO Control on the LUN.   Changing the "Limit - IOPs" seemed to have no affect.    From further reading of the documentation it seems this limit might only be used under certain circumstances and not a hard Limit.