SAN allocation; split or keep one large LUN?

[GMarsh]

we are designing a new cluster and the conversation came around to the SAN connections. the idea suggested is to have one LUN strictly for VM OS's and another LUN for Data ie; File share data, SQL database and log files, etc.

There is an arguement from another engineer that we should have it all on one LUN, because splitting them will make one LUN very busy with Data writing and the other seem wasted.

Which do you think is the better model or best practice? Split the LUN's and designate one for OS and the other Data? OR one large LUN and just create the .vmdk files for disk?

[arkturas]

It all depends on how big your infrastructure is going to be, are we talking GB or TB

you need to make allocation for backups too.

we run about 5vms per LUN to keep path thrashing & SCSI reservations conflicts down, LUNS are +/- 300GB in size, you do end up wasting space as you need to account for VM snapshots/vmotion tasks. +/- in our case about 3-5% of the LUN

How many VM's do you intend to run, what size are the VM.s (total vmdk)

[jguzmanr]

what's large? large is a relative term. I do tend to agree with your engineer that one lun for the OS's and one for data is not the right way to go. You should decide if you need one or two luns depending on the number of vm's and the amount of I/O that they will have. I personally do not virtualize SQL as we really do hammer them (we average about 2K IOPS, per instance), so we have a cluster for this. I'm not saying to not virtualize SQL, it just wasn't a fit for us.

[weinstein5]

The questions the other posters asked are important - I will also through one importan fact is ESX/ESXi only support LUNs 2 TB or smaller -

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[GMarsh]

on this particular ESX host there is only going to be 10 VM's (required by the project) however in planning we figure that there will be more. the LUN sizes we are also trying to come to an agreement on a typical standard size; knowing that this will not always be the case but hopefully a normal standard can be realized. that size we are thinking is in the 200-500GB range.

The size of the standard VM is a 14-20GB OS partition and then a data partition varying in size according to the requirement of the application.

[arkturas]

agree, we have a number of SQL VM's fairly small though (70GB) the disk I/O is very high, I haven't noticed any performance degradation disk queue length is pretty low.

[arkturas]

another point to make is that you dont want to be adding LUN extents if you find you made the LUNS too small! whilst not a problem it can be difficult to manage.

I think a 400GB per LUN, is your best bet, with plenty of room for expansion.

you also need to allow for OS expansion, (easily done using disk part)

how is your SAN infrastructure configured, do your storage processor support Active\Active ? or are they Active\Passive

[azn2kew]

Our standard SAN LUN size is between 400-600GB and store between 10-15 VMs per LUN and that varies in types of applications and servers. With less I/O intensive we tend to have more but high I/O like Domino server, Exchange and SQL than that would be less. Only time I use large RDM/VMFS LUNs are 1TB for my file servers. You can limit the iSCSI reservation conflicts by reducing the amount of VMDKs in a LUN and prevent from path thrashing as mentioned. I would look at best practice guide how to design SAN storage for VMware. www.vmware.com ->resources page.

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant

[arkturas]

Housing OS, Database on seperate LUNS could cause issues (unless using RDMs) when upgrading the VMFS lun from 2 to 3 it was advised that the VMDK's were on the same LUN.

this may help: my VM/s have seperate VMDKs for OS, Logs, Database all sitting on the same LUN

[Ken_Cline]

As most have said - it really depends. Typically, I don't like to split my VMs across LUNs. There are exceptions, but the "norm" is they stay together. General rule of thumb is 10 - 15 VMs per VMFS volume - more for low activity VMs, fewer for high activity VMs.

Another thing to keep in mind - Microsoft's 'required' system drive sizes for W2K8 are pretty obscene. This will (potentially) impact your LUN sizing calculations.

Ken Cline

VMware vExpert 2009

Technical Director, Virtualization

Wells Landers

TVAR Solutions, A Wells Landers Group Company

VMware Communities User Moderator

[GMarsh]

so just to make sure I understand the concensus from all of you that answered. the better idea would be to have one LUN say 400GB and put only 10-15 VM's on it with standard C:\ and then use the same LUN remaining space for the application data specific information and NOT split them apart.

[msemon1]

So If I have 8 VMFS datastores, each about 1.5TB and about 20 VM's, are my chances of SCSI reservation errors and path thrashing increased by having this many VM's per datastore. They all have ample free space.

Thanks,

Mike

[Ken_Cline]

So If I have 8 VMFS datastores, each about 1.5TB and about 20 VM's, are my chances of SCSI reservation errors and path thrashing increased by having this many VM's per datastore. They all have ample free space.

Chances are increased - yes. Does that mean that you're going to have problems? Maybe, maybe not. There are a lot of factors that go into this whole "how many LUNs per VM" discussion. Here are a few:

- How many IOPS does the storage backend support (i.e. are we looking at 14 * 144GB @ 15K FC or 2 * 500GB @ 7200 SATA?)

- What is the storage interconnect (i.e. Fibre Channel, iSCSI, DAS)

- How many hosts are concurrently accessing the VMFS volume?

- How volatile is the metadata (i.e. file size, file name, etc.) on the volume?

- How many VMs will have active snapshots?

- For how long will snapshots be active?

- How active are the VMs on the volume?

All of these factors (and more) need to be considered before you begin to get a clearer understanding of whether the LUN is sized correctly.

Ken Cline

VMware vExpert 2009

VMware Communities User Moderator

[RParker]

There is an arguement from another engineer that we should have it all on one LUN, because splitting them will make one LUN very busy with Data writing and the other seem wasted.

He is absolutely correct. That's it exactly. If you look at it logically, you split your VM's, one OS and one DATA. you still have the same problem, ALL OS are on the SAME LUN, so if the data is intensive you are still sharing IO from other VM's.

Also you are eventually playing a shell game with your VM's. If ALL your VM's are sharing ALL the same LUNs (one for OS, one for Data, one for Logs, one for DB, etc..). What happens when one LUN goes down? Guess what -- ALL your VM's are TOAST.

Not a good idea, the more VM's spread out over more LUN's the more chance you have that a single glitch can affect ALL those VM's at once. So isolate the events, keep the VM's confined to ONLY one LUN. In the long run, that's the best course, minimize collateral damage!

The engineer is correct, and that's the way it should be done, avoid Crossing / Linking LUN's.

[RParker]

what's large? large is a relative term. I do tend to agree with your engineer that one lun for the OS's and one for data is not the right way to go.

Everyone keeps talking SIZE. SIZE does NOT matter for LUN's. DATA is your key factor, not storage. LUN size is relative.

What's large, that's a good question. 400Gb is that a big LUN? What if your VM's are 200GB each? Yeah, umm.. no, not very big at all.

You will get SCSI reservations regardless... I found that out. The idea is to keep those reservations to a minimum.. by keep the number of VM's on a single LUN down, but SIZE of the LUN shouldn't be your first consideration, in fact throw it out. It's pointless.

Only the number of VM's is your concern, because that's what determines your LUN deployment. And keep the number somewhere between 15 and 25 VM's... but even those numbers are relative, because SCSI reservations happen when a file is opened and closed, but once it's open, I don't think it needs to 'reserve' that handle, it's already in open... the reservation is a conflict manager to keep other operations from stepping on other close open operations at the same time.

So these are just guidelines, and you will see over time that these 'ideas and considerations' are general principles not absolute rules. They will change over time. And so will your environment.

[msemon1]

Here is our setup

• All drives are SCSI 10K

• FC with dual port HBA's. Brocade switches, and EMC SAN

• All hosts have 6 network cards so that SC, VMotion, and Virtual machine networks have teamed pNics

• All 16 ESX 3.5 U3 hosts access 8 VMFS datastores

• Have removed all old snapshots and try not to retain for very long new ones created.

• The VM's are active since they are a mixture of DC's, file servers, Citrix servers, SQL servers etc.

Looking for best way to reduce SCSI reservation issues and optimize performance. So it sounds like I need to look most at operations like VMotion and backups to ensure we don't have too many opertions going on at the same time

on each LUN. I have affinity rules setup so DC's, SQL servers, and other critical servers don't run on the same hosts. Have been guilty of trying to run too many Vmotions at same time

Mike

[Ken_Cline]

The "worst" thing I see in your config is 16 hosts in the cluster. In "Virtualization according to Ken (™)" the optimal cluster size is eight nodes. I've found this to be a good tradeoff between storage contention and capacity flexibility.

The other thing that I've found is that snapshots tend to grow most rapidly just after creation. There's typically a fairly small area of a disk that is highly volatile, so when you create a new snapshot, the first thing that happens is that area of volatility gets hit and the snapshot grows to a relatively stable size. Once you reach this plateau, snapshot growth tends to slow down. Obviously, there are situations where this is not the case, but I find it to be the "norm".

So...with that said - look at how your VMs are distributed across your VMFS volumes and your hosts. Based on that distribution, you can figure out how many hosts are actively accessing each VMFS volume (vCenter Server maps can help here). A smaller number of hosts is preferred (and this is not something that DRS looks at when making migration decisions). I would recommend that you go with more VMFS volumes of a smaller size with a limited number of VMs per volume.

Ken Cline

VMware vExpert 2009

VMware Communities User Moderator

[Roanry]

We have 8 node clusters that have over 60TB connected to them via 4 x 4GB qlogic HBA's. We never break apart our VM's vmdk files and spread them accross many LUN's. I did this back in ESX 2 for our SharePoint environment and it not only became a pain to manage which VM was where but when it came down to moving the VM's to other LUN's to increase the size of the VMDK's it became even a bigger managment nightmare.

Each LUN sizes is different and we try to keep as few of VM's on each LUN as possible. The main issue you will have if you fill up your LUNs with a lot of VM's is when you start using snapshots. If the snapshots grow to fast and you forget about them you will have issues where the other VM's will not power on.

Storage management of your environment is a key here.

[arkturas]

Just out of curiosity, I don’t want to lead this thread in a different direction, but how do you backup 60TB of VM data - you have to be utilising some sort of SAN snapshot ?

Do you back that up every night? Or is it just incremental?

[francisbandi]

having separate luns to System disk and data disk has pros and cons,

1 - if you plan to implement VCB - implement all volume on same lun

2- if you plan to implement data replciation / SDRF other disater recovery type solution, you may need to organize these disks in separate luns.

regarding lun size - 300-500 is optimimu size as per some recomendation.

we do system disk as 20 GB and this may change with windows 2008.

and other disk varies.. we see average of 4-6 VMs for disk depending on data disk requirements.

hope this info from the field helps..