ESXi 5.5: VM Performance on RAID 10 Datastore - How many spindles??

[LabDevUser]

I/we have a lab/development box that is running on a hodge-podge of random combination of drives ranging from 2.5" SSD's to 3.5" 2TB HDD's. The time has come to streamline the hardware as we are adding a second (identical) development/lab host.

I found that a striped datastore with two spindles (4x 320GB 7200RPM HDD's in RAID-10) performed reasonably well however the VMs placed on that array were all fairly passive (active directory & management servers, vCenter appliance, etc.).

I know there is MUCH discussion online about this topic but I have not found anything definitive other than "it all depends" or "your mileage may vary". Is there a decent rule of thumb for number of spindles needed to get a reasonably performing RAID 10 array? Is it reasonable to think that I could put 25-30 VMs on a single RAID 10 array of HDD's or is that crazy no matter how many spindles are in the array? What we run into is the pros and cons of the performance advantages of giving a VM a dedicated HDD or SSD vs. the leftover/empty dead/wasted space at the end of the datastore that sits empty.

I'm going between two choices, a single larger RAID 10 array with 12x 320gb, 7200rpm, 64mb cache HDD's (six spindles in the stripe) with one or two hot-spares and one or two SSDs for LSI's CacheCade caching system on an LSI 9260-16i raid controller...

or

Two RAID 10 arrays, basically break the above in half and split it between either the same RAID card OR two 8-port RAID cards.

Would I be better with one larger RAID 10 array or two smaller ones? Also, is it reasonable to think that I could put 25-30 VMs on a single array such as this? Out of the 25-30 VMs only 3-4 will get fairly active or heavy use. All the VMs have small OS HDD's about 32-64GB in size. They have much larger multi TB HDD's on remote NAS or other datastores and any "heavy use" is usually done by writing to a VM guest created 24GB RAMDRIVE at the guest OS level so overall fairly low writes to the guest OS or primary HDD.

Thanks in advance for any thoughts or feedback.

[Alistar]

Hello there,

"a general rule of thumb" for direct-attached storage on ESXi host would be to have a RAID controller with some NVRAM cache ( > 512 MB) for the performance where this cache is flushed out to disks where available, and BBU (back-up battery unit) in case of power outage and the disks attached to it. RAID10 is the best possible choice for magnetic spindles in terms of performance, but could be hard to justify if you are on a tight budget.

Next is the spindles' RPM - the more you have the more IOPS you are able to handle. 7400 RPM SATA tops out at somewhere between 75-100 IOPS, whereas SAS 10k RPM at ~140 IOPS and 15k RPM at 175-210 IOPS (source). Using any RAID calcluator on-line can give you a rough estimate of how much combined IOPS will your drives be able to handle. Since many VMs running on a datastore will result in random reads/writes to disk, this will be the main metric to go by. I can also recommend making one datastore from all the drives on the same RAID controller - this way you'll avoid wasting the fragments of leftover space on your datastore.

Make sure you understand what exactly will these "high performance" VMs need - consult the owners, monitor them for a day or two during their peak times to be able to completely emulate the current performance under a virtualized solution.

Cheers, and good luck!

[LabDevUser]

I appreciate the reply so quickly! These are development/lab hosts so I'm not sure how to politely articulate this other than to say I don't  care so much what the end user needs or does not need regarding performance because when (if) this goes into production it will either be on bare metal or on a much higher performing host with SSD datastores vs. spindle HDDs. What I am worried about hardware failures and losing data, while having a reasonably decent performing ESXi host.

Up to this point the last guy who was here previously had everything in RAID5 and RAID6 arrays using very old SATA2 RAID cards and 2TB 3.5" retail HDDs. It was a disaster though it wasn't all his fault as that's what he was given. The RAID6 arrays were experiencing failures over and over. He has since moved on (promotion) and I have inherited this mess. It is currently one Dell Precision T7500 tower (not technically a Server but close enough for a development/lab environment). I have two more that were given to me last week so our dev environment is going to go from one single host to a cluster of three identical machines. Each has two hex-core CPUs and 96GB RAM so they are plenty capable of hosting a bunch of VMs assuming the right datastore configuration.

I am thinking about buying four of these per ESXi host:  iStarUSA BPU-126-SA 1 x 5.25" to 6 x 2.5" SATA 6.0 Gb/s Hot-Swap Cage - Serv...

I would then pair them with either 16 or 24 port 9260 or 9280 series RAID cars from LSI. Each series has 512mb cache and a BBU option as well as the ability to use an SSD for R/W caching and making use of a hot spare.

For hard drives 10k/15k would be nice but #1 they do not fit in the chassis and #2 they are not within our budget so it will most probably be a bunch of 320GB or 500GB 7200RPM 2.5" laptop drives creating the arrays. I would eBay a bunch of used HDDs.

My biggest concern with one 12-drive RAID10 array is both the vast number of random read/writes generated by all the VMs being on the same datastore AND (I don't know if this is even something I should worry about) the I/O of the RAID card through the PCI bus. A lot of the VMs on the host make use of 4GB-24GB RAMDRIVEs which makes the PCI bus even more saturated. We could probably buy 24-port cards since there are 4x 5.25" bays and 4x 3.5" bays internally. We could do the cheaper 320GB HDD's and build a RAID10 array with 20 drives plus two hot spares and two SSD's for LSI's CacheCade feature. Again, my worry is the VMs hammering on the array AND the PCI bus traffic. I get that more spindles is better but when you start to get that big (10+ spindles) I worry that it gets foolish because of the number of random read/writes. I have read many places online that in a situation such as this I might be better running 6-12 RAID1 mirror arrays and put each VM on its own datastore vs. put all on one giant datastore.

It's confusing and I'm not sure as to which way to go about this. There is plenty of room to fill up the 4x 5.25" bays with 24 2.5" drives and still room for 8x more 2.5" SSD's and things for ESXi host swap & cache and for Virtual Flash Resource capacity.

That's what I was getting at I suppose. I understand each has their own opinion. I am genuinely interested in hearing what the community has to say. Thank you.

[LabDevUser]

Does anyone have any other comments on this subject? Since my original question I have narrowed the scope a bit. There will be twelve (12) 2.5" HDD bays available. My questions are:

Should I put all of these on one big RAID10 array or two smaller RAID10 arrays? Will there be any drastic performance issues between a 12-spindle RAID10 array and a 6-spindle RAID10 array?

Given the aforementioned, would it be reasonable to purchase a 16-port RAID card and put all 12 drives on the same RAID controller/PCI slot --or should I put the arrays on two separate RAID controllers as well as two PCI buses?

Technically on this machine I have two 5.25" bays and four 3.5" bays. Because of adapters and an adequate PSU, that could mean any number of configurations up to a maximum of twenty (20) 2.5" bays available. Presently I am thinking I will configure the machines with 16x 2.5" bays and 2x 3.5" bays. The 16 2.5" bays will be broken out as follows:

Bays 0-11 will be either 1x 12-drive RAID10 array or 2x 6-drive RAID10 arrays.

Bay #12 will be a hot spare

Bay #13 will be reserved for a 128gb SSD performing RAID Controller Cache functions

Bays #14 & #15 will be reserved for ESXi Cache functions and fitted with 128gb SSDs

Will that system perform reasonably well?