VMware Cloud Community
goppi
Enthusiast
Enthusiast

Which RAID setup to choose

Hi.

We are setting up a new ESXi 4.0 on a Proliant DL380 server.

We have 6 SAS 146 GB Disks as DAS which should be configured for VMware datastore.

The question is which RAID configuration to use.

Generally I see the following possibilities

1.) 1 RAID 5 with 6 Disks

2.) 2 RAID 5 with 3 Disks each and spread the VMs between the two arrays.

3) 1. RAID 10 wih 6 Disks.

I would expect option 3 to be the fastest one but a lot of HD space would be wasted.

What about option 1 and 2. I would expect option 2 to be faster than option 1 with regards to write speed.

What do you think?

BTW: Of cource the controller has BBWC and 512 MB.

I'm interrested in what you think.

Regards.

Reply
0 Kudos
11 Replies
RParker
Immortal
Immortal

What about option 1 and 2. I would expect option 2 to be faster than option 1 with regards to write speed.

Nope. RAID 5 with 6 spindles vs RAID 5 on 3 spindles definately will NOT be faster, and probably more significantly slower. I would go with RAID 5 on all 6 spindles. having 10 VM's across 6 spindles is WAY better than 5 VM's on 3 spindles any day of the week.

And 'slower' is relative, it depends on what you are doing. RAID controllers are not really designed for heavy IO on a local machine ANY WAY, so RAID 10 may be faster but is it worth the 50% disk space loss? I don't think so.

When you want performance you don't throw your VM's on local storage in the FIRST place, so anyone concerned with speed, local disks are not in the equation. SAS is better than SATA, so at least you are on the right track.

ESX 4.0 is better than 3.5 was at Disk IO, so I think they made up the difference, and you won't need RAID 10.

Reply
0 Kudos
brainslice
Enthusiast
Enthusiast

Not sure if this is in your budget, but I had a similar configuration.

We were trying out ESX to see how things would work out in our lab environment.

It was installed on a single DL380G5 with 8x 73GB 15k SAS drives and a P800 controller.

The local storage was set up as disk 1+2 in a mirror for the system and disk 3-8 in RAID5 for the VM datastore.

The problem with this is that it's relatively small so doesn't fit many VMs, doesn't perform particularly great when you load it up and makes things very difficult to deal with if you have more than one ESX host or decide to use vCenter.

We now have two DL380G5 hosts (with more to come) managed by vCenter and have moved the datastore onto a 4G FibreChannel SAN. This is far better performing and very simple to move around VMs and use vCenter. hth.

As far as the local storage, I would still run the RAID5 across 6 disks.

Reply
0 Kudos
goppi
Enthusiast
Enthusiast

Nope. RAID 5 with 6 spindles vs RAID 5 on 3 spindles definately will NOT be faster, and probably more significantly slower.

But that's what I was saying?

I would go with RAID 5 on all 6 spindles. Having 10 VM's across 6 spindles is WAY better than 5 VM's on 3 spindles any day of the week.

But why if the assumption RAID5 with 6 disks is slower (write speed) than RAID5 with 3 disks? If you have 2 independend RAID5 arrays, operations on array 1 would not have any impact on operations on array2. So overall perfomance would be better with two independant RAID5.

so anyone concerned with speed, local disks are not in the equation.

So what is the technical reson for this?

Supposed you have similar parameters like drive speed, controller cache size, number of spindles, RAID level I would expect similar speed for SAN and DAS. Are the controllers within SAN arrays so far superior than controllers for DAS? They share the same technology. Or is the reason ESXi handles SAN HBAs better than DAS controllers on the virtualization layer?

Regards.

Reply
0 Kudos
goppi
Enthusiast
Enthusiast

Hi brainslice.

Thanks for sharing your findings.

It's clear that your current setup is more flexible and scalable than your previous setup.

But what about diskspeed? How was your previous setup performing with regards to diskspeed?

Is the SAN storage you are currently running comparable (disk speed, number of spindles, controller cache, RAID level) with the DAS setup you had previously?

I'm interrested in finding out if you could expect improved performance from SAN storage if you run it with similar parameters like DAS.

Regards.

Reply
0 Kudos
brainslice
Enthusiast
Enthusiast

The original DAS datastore was doing ok with 10 vms. It would start to feel sluggish when booting multiple vms or performing install / reboots etc. I don't have any actual benchmarks to compare, however. Since things were going well with ESX I knew that we needed to look at SAN storage for expansion and flexibility.

One thing we can still do with the local storage is to manually migrate VMs to the local storage if there was any major work or reconfiguration needed on the SAN.

The SAN is on some bigger hardware than the local controller and disks. It's multipath connected via 2x QLogic QLE2560 HBA to a SANBOX 5802 and the array is a Winchester SX2394R with redundant controllers 2G cache and 12x 146G 15k SAS disk in a RAID6. This can be expanded with additional disk shelves and have the controllers put in active/active when the need arises.

I don't think it would be worth the cost to implement a SAN solution with similar specs to the DAS array on the P800.

goppi
Enthusiast
Enthusiast

@brainslice:

Thanks again for sharing you findings.

Regards,

goppi

Reply
0 Kudos
I-Like-VmWare-S
Contributor
Contributor

Hello,

In my test enviroment i'm using a USB Card with ESXI installed on it, to boot.

I've made it, using de .DD file which is 'zipped' inside the IMAGE.TGZ file in the ISO file, and have used WINIMAGE to make the 'bootable' USB card (are also using an CF Card, made within the same way).

In that case, i'm saving at least 2 expensive SAS disks. In case the USB card is failing, i've got another stick with the same ESXI files.

So, 8 disk's to use !

Reply
0 Kudos
I-Like-VmWare-S
Contributor
Contributor

Recently i discovered (of course somewhere in this forum), that when using an equal number of spindles (disks), in combination with an calculated Stripe Size will give an performance boost for disk access when using VMFS volumes for storing your VM's and/or Data disks. This in combination with the Block Size (1 t/m 8 mb) choosen when creating the VMFS volume.

Example:

A VMFS volume with a Block Size of 1 mb is 1024 kb. Dividing with a Stripe Size of 256 kb, would mean 4 (data) disks. (4 * 256 = 1024 kb). In total you wil need a fifth disk for parity. So a RAID5 volume with 5 disks in total, where 4 disks are actualy being used for data. Each I/O command will be accomplised in 1 time. When using a Stripe Set of 128 kb with 4 (data) disks, 2 I/O command's would be needed.

If you would have 8 (data) disks, a 128 kb Stripe Size would be enough for 1 I/O command.

In my experience, the more disks are being used, the better (not only for reading).

Booting with an USB disk, would spare 2 disks.

So, i have choosen to use 5 disks (41) for my Guest VMDK's and have made an additional array of 21 (the rest of the harddisks) for extra storage space for ISO files etc.

Reply
0 Kudos
Lord_Micron
Contributor
Contributor

The first question I would ask is: What functions are the guest VMs going to be performing on this host?

I would only consider options 1 & 3.

Choose RAID 5 if you're not going to be doing a ton of disk I/O, as RAID 5 does take a hit on write performance from the generation and recording of parity... You could offset this some by adjusting the utilization of the controller cache, but the effect would be marginal. (IE: What portion it uses for reads and what it uses for writes.)

If the roles of the guests will be I/O intensive, you should stick to RAID 10. Regardless of whether it's SAN or DAS, you don't want it eating your cache... I've seen Clariions heavily impacted by poor configuration.

Reply
0 Kudos
joergriether
Hot Shot
Hot Shot

since money nowadays is not the huge factor anymore i always would use raid 10 for important datastores, locally and also in my san. on the other hand, if the data store is for my lab machines, it is raid 50.

regards

Joerg

Reply
0 Kudos
afidel
Contributor
Contributor

Using a DL380 G5 with 6x146GB 10k drives in a RAID10 for the VM store I get 2,800 IOPS doing 4k random reads and writes using IOMeter which is better than I can do under Windows native on the same configuration (2,100). Since traditional metrics say a 10k drive is good for ~150IOPS under load that means the cache is extremely effective if you aren't overrunning it. I think you'd have to have a fairly IO intensive load to really feel it on this setup.

Reply
0 Kudos