San configuration - datastore have more than 1 LUN

[tdiroff]

Hi

We are looking at our San configuration for improvements. Our San has about 10TB in total.

Currently we have 6 luns approx 1.5TB each and all being joined into 1 datastore of 10TB

It has been suggested that we should NOT combine multiple LUN's into one datastore for the reason if 1 LUN goes bad then the whole datastore is un-recoverable.

While our SAN LUN's are created on top of raid 6, and thus somewhat 'protected' the 1 datastore per LUN advice is interesting

Does anyone have any thoughts to share?

Thank you

Tom D

[AndreTheGiant]

Which kind of storage do you have?

On "traditional" storage the best solution is have LUNs on different RAID groups on different disks.

But this does not apply on some "new" storage (for example Equallogic that usual has a single RAID group for all the disks).

Andre

[tdiroff]

Hi Andre!

We have a 16 drive Infortrends Fibre Channel array. 15 drives are configured for raid 6 and we have 1 hot spare.

Tom D

[azn2kew]

Not familiar with Infotrend storage systems, but see if you can create multiple RAID groups for different purposes such as Windows, VMware, Database systems and choose the right RAID level and disk spindles accordingly. I wouldn't use 1.5TB for the LUNs but rather carve LUNs between 400-600GB in size so to limit the number of SCSI Reservation conflicts due to many vmdk metadata activities involved and eventually lead to I/O thrashing. You should have multipaths to multiple storage controllers and SP for extra redundancies and performance since all paths are spread evenly. On our EMC Clariion, we create different RAID groups of different purpose and carve the LUNs accordingly with RAID levels and spindles required and all connected to redundant storage processors/controllers through multipaths.

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen

VMware vExpert 2009

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant

[tdiroff]

Hi Stephen

Yes, we have also been considering smaller LUNs for the reason that you noted. However, the big question is whether to have 1 LUN per datastore or combine multiple LUN's into a larger datastore..... what did you do?

Take care

Tom D

[JeffDrury · Accepted solution]

I would say that having a single large datastore, no matter the underlying storage architecture, is probably not a good idea. While this configuration is allowed you can paint yourself into a corner fairly quickly.

1 - There is some question on the maximum number of VMDK's that you should have per datastore. http://communities.vmware.com/message/944120 Basically the more VMDK's on the datastore the more SCSI locking activity that will take place on the disk, causing performance concerns. Every VMware engineer I have talked to seems to stay around the 20 VMDK per datastore range, however this may be changing with vSphere.

2 - How do you backup this datastore and how long does it take? What happens during a restore scenario? 10TB is a lot of space and could take quite a bit of time to backup. If you are pushing this to tape and you ever have to restore the entire 10TB it could take weeks or months. Breaking this space into smaller datastores gives you smaller chunks of data that you can restore in the event of a disaster.

3 - The number of datastores that you have is very important if you are ever looking at implementing Site Recovery Manager. SRM relies on your storage device to replicate data between sites. There could be serious implications on data change rates for a single 10TB LUN. If you move to multiple datastores it will make it easier to manipulate your recovery plan.

Aside from the above VMware considerations. I would imagine performance of 7 LUN's at RAID-6 is not that great. You have a good deal of protection but you are paying for that with performance and capacity. If you went to 10 datastores and 10 LUN's you could change RAID settings at each datastore to maximize performance, protection, and capacity. For example 4 LUN's at RAID-10 for high performing DB's and apps, and 6 LUN's at RAID-5 for capacity. With smaller datastores and LUN's a disk or LUN failure will only affect a smaller set of services rather than your entire datacenter.

In short unless there is a really compelling reason to go with a single 10TB datastore, I would recommend changing your architecture.

[azn2kew]

The rule of thumb is 1 VMFS per LUNs and you can have multiple LUNs per datastore, and it all depends your design due to different usage and performances.

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen

VMware vExpert 2009

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant

[AndreTheGiant]

We have a 16 drive Infortrends Fibre Channel array. 15 drives are configured for raid 6 and we have 1 hot spare.

I do not know Infortrends storage.

But check your manual to be sure that 15 disk in a RAID 6 configuration is a good value.

For a lot of storage best practice suggest to use max 6-8 disks. With more you can have a slower system on write I/O operations.

Andre

[tdiroff]

Hi Jeff

Great input!!!

Performance has not been an issue for us to date and we have 70 vmdks operating on the datastore. However, it would be terribly short sighted of me to not want to address that in a new design, especially when trying to match application to raid type.

We use esxpress for backup (to disk via FTP) and the backups take about 8hrs. For restores we restore VM by VM and certainly for all of VM's it would take quite some time (days).

Tom D