Disk Throughput VMs with high I/O

[epping]

Hi guys

i was wondering how to plan for vms with high I/O, backend array is not the issue here, i am concerned about only using one fibre path to the back end.

Am i right in thinking that a 2GB HBA can handle about 200MBps load ?

has anyone had any success with virtualising some monster servers.

thanks

[weinstein5]

depending on how many of these high i/o vms you can saturate a single path to storage -

[spex]

Most important is to know the io characteristics of your applications. Most of them (exchange, sql) use random io.

So the theoretical 200MByteps are limited by your number of spindels and not by your hba.

Example:

Using 8k io's (exchange) you need around 25 000 io's per second to reach 200MByteps limit.

A 15k disk does around 160io's per second (no caching). So you can handle around 150 disks at 100% util.

Your are lucky if you hit the limit first with your hba.

Sequential io's is an other story. One disk can deliver around 60 MByte per second...

Regards

Spex

[williambishop]

To add, you can, if you have multiple paths, also reflect this in the guest, by adding another virtual hba and pointing it to the other path explicitly.

[epping]

thanks for this

not done anything with virtual hba's yet.

can i have multiple vhba's pointing to the same disk - i always thought that esx could one have one path to target

thanks

[weinstein5]

That is correct - but with 3.5 Update 1 I believe there is now experimental support for round robbin load balancing on hbas -

[epping]

this i read someware that it can give u 30% throughput

[williambishop]

It does provide a very noticeable benefit, unfortunately, until they release it from experimental, I'm stuck with 95% of my environment running one path.

[weinstein5]

is everything on a single LUN? Do you have the ability to use mutiple HBAs? As you mentioned you can only have one path active to a LUN but if you have mutiple HBAs and multiple LUNs you can abalnce the load across all available paths to your AN -

[epping]

hi

i have multiple LUNs but still require large throughput to each LUN, my concern is one path will not be up to the job

[williambishop]

Do you have dual paths?

[epping]

hi, yes have 2 hba;s but can stick in more, no sure how this helps though as i thought i can only use one path to a lun in esx. am i wrong?

[williambishop]

Like I said, you can add in a second virtual HBA, and point each to one path.

[weinstein5]

your are correct - but you can balance the load across all HBAs by having different LUns use different HBAs so all of your SAN traffic does not run through a single HBA -

[weinstein5]

I am also curious as well, what do you mean by virtual HBA? Since it eems like he is referring to FC SAN - it is my understanding all you can use are physical FC HBAs installed installed in the ESX Host -

[epping]

ok done a bit of reading and virtual hba's maybe the way to go, the disk has to be in RDM mode but looks like i can use up to 4 physical hba's, i.e. a one to one relationship between virtual and physical --- i wonder if any mpio driver supports virtual hba's it would be intresting to see.

Ok read a bit more on this and i dont think it is possible, it looks like the virtual hba's are identification aliases only and can not be uses to route load through different paths.

on the plus side http://blogs.vmware.com/performance/2008/05/100000-io-opera.html

Message was edited by: epping

[sakacc]

Disclosure, I'm an EMC employee.

Great question!!! It's one that we (VMware and EMC) get often. We just completed a joint study on the topic (joining a lot of Exchange, SQL server, Oracle and SAP in VMware testing and docs at a wide variety of scales).

Check it out here:

There's also some good whitepapers (and more to come any day now) on 16K Exchange 2007 users (in increments of 500 - so applicable to lower end configs also), and 7000 concurrent SQL Server 2005 users (and the same for Oracle 10g/11g, Sharepoint and SAP) all in 3.5 configs. We have a "solutioneering" lab with more than 450 ESX servers where we do exactly this 24x7 (above and beyond interop, test solutions to the breaking point - not just marketeering, but for real world best practices.

You can virtualize monster IO workloads - just need to follow best practices and leverage community and vendor know-how.

Let me know if you're struggling to find the info, and I'll get it directly for you.

[christianZ]

Always by such test I would ask about capacity utilization - practically when you would have e.g. 50-60 % cap. utilization I would expect much lower numbers, IMHO. Anyway thanks for that - one said EMC doesn't make any workload test.

[sakacc]

Utilization was abjectly terrible in this use case (at least from a capacity aka TB) standpoint. The goal was "as fast as possible" which after all, sometimes is the driving single constraint for the customer. More often, people are looking for balance. If "best performance with near 100% utilization", we could drive the same response with a very small number of Enterprise SSD drives (we can do 3000-5000 IOPs apiece), but then again, those are very expensive.

We do a TON of published workload tests - but in general have found that generic workloads (SPC) are less valuable (though we do publish SQL Server/Oracle results using Quest Benchmark Factory, and Exchange LoadGen results (only exception which is pretty solid is the SpecNFS tests), but that application-centric workloads are generally the most useful.

We try to stay out of "benchmarketing" (just ends up being vendor pride-fights, not useful customer data), but in this case wanted to help VMware put an end to the "you can't put big IO worklods in VMs" myth by posting a ridiculously high result.