VMware Cloud Community
dp_tn
Contributor
Contributor

Best Practice LUN Size

We are planning out our VI3 deployment. I have seen lots of postings/responses about Max LUN size, etc.. but am more interested in the "best" or optimal LUN size in an ESX VI3 environment. I know there are a lot of "well, it depends on..." discussions that could be had, but wanted to see what folks are seeing out in the field.

We are looking at HA clusters of 8 - 16 hosts. One of our concerns is the administrative overhead involved in managing a large number of smaller LUNs. Engineers from our storage vendor swear that a single VMFS file system of more than 400GB (=LUN size of 400GB) is crazy, etc... but in many cases a 400GB LUN does not efficently meet our needs. We are considering 1TB VMFS LUNs.

The majority of our guest workload is not disk-intensive. We do not place Exchange, SQL Server, or like workloads in guest machines at this time. We also have the benefit of stripping very wide across numerous spindles on an array dedicated to our virtual infrastructure.

What experiences are out there with regards to LUN sizes, RAID types? Again, there is a huge "it depends" factor here, but want to see what people are doing.

Thanks

Tags (4)
Reply
0 Kudos
17 Replies
Gabrie1
Commander
Commander

You're right there is no best practise in this, but the storage vendors statement "more than 400Gb is crazy" is certainly untrue.

For my environment I've done the following. Inventory of all existing physical servers that needed to virtualized. Collected their disk sizes and came to a X terrabyte needed over Y vm's. Which averages at X/Y Gb per VM in my case 25Gb per VM.

Reading all the forum posts and a number of white papers, I decided that I would never put more then 16 vmdks together on 1 LUN. Which would give me a need for 16 x 25Gb per LUN = 400Gb. Then per VM I would assign 1Gb memory normally, which would also cost 1GB swap space on the lun, so I got to 416Gb, lets say 425GB per LUN. Then I took a 10% margin for snapshots and this got me to around 500Gb per LUN. And this is what I'm using now Smiley Happy

Hope this helps.

Gabrie

http://www.GabesVirtualWorld.com
Reply
0 Kudos
chrisbuzby
Enthusiast
Enthusiast

Per RAID, 0+1 if you have the drive's and 5 as a secondary if you do not have the drive count / need more space...

Another concideration is the most efficient use of your SAN's capabilities, if it has 2 storage processors / controllers, I would suggest splitting the load accross both, obviously if you have a failure then you would fail to a single and have performance impact.

Sizeis a bit more tricky, One thing to keep in mind is you can really "hurt yourself" by filling up a drive, and because of snapshots you can fill-up a drive relativly easily... So don't give up the keys aka alow users / server admins to snap!

Guess the above really applies to any app, not sure how much VM specific best practices there are out there, I heard some thing about 1TB, being a good rule of thumb, but never got any logic behind it... Ohh and as always the more spindels the better

- Chris

Reply
0 Kudos
esiebert7625
Immortal
Immortal

500GB-600GB seems to be the recommended max size around here. Here are some good threads that talk about this....

LUNS - http://www.vmware.com/community/thread.jspa?messageID=333672

LUNS Size - http://www.vmware.com/community/thread.jspa?threadID=36725

Sizing LUN for enough free space after VM - http://www.vmware.com/community/thread.jspa?messageID=649338

Smaller LUNS or Larger LUNS - http://www.vmware.com/community/thread.jspa?threadID=90203

Larger LUNS = More Disks = More Performance - http://www.vmware.com/community/thread.jspa?threadID=84843&tstart=0

RAID Recommendations for local disk - http://www.vmware.com/community/thread.jspa?messageID=622456

RAID 5/6 Scaling - http://www.vmware.com/community/thread.jspa?messageID=718641

Which is better, more or fewer disks - http://www.vmware.com/community/thread.jspa?messageID=599611

SAN Configuration Guide - http://www.vmware.com/pdf/vi3_esx_san_cfg.pdf

SAN Conceptual and Design Basics - http://www.vmware.com/pdf/esx_san_cfg_technote.pdf

SAN System Design and Deployment Guide - http://www.vmware.com/pdf/vi3_san_design_deploy.pdf

Fyi...if you find this post helpful, please award points using the Helpful/Correct buttons.

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

Thanks, Eric

Visit my website: http://vmware-land.com

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

Reply
0 Kudos
depping
Leadership
Leadership

I usually don't go over 500 for a multiple vm VMFS. If it's for 1 vm only with not a lot of I/O I create it depending on the size the customer want, if it will generate a lot of I/O and needs more flexibility I use RDM's.

Duncan

Reply
0 Kudos
Gabrie1
Commander
Commander

Hi

According to a session on VMworld this year, the need for RDMs is getting

less and less. They said that the speed difference between vmdk and RDM is

next to none. Also the VMFS overhead is very little. Maybe its worth doing

some testing with the performance differences between them?

Gabrie

http://www.GabesVirtualWorld.com
Reply
0 Kudos
depping
Leadership
Leadership

the performance gain is little. but enlarging a rdm / lun on the fly is much easier than enlarging a vmfs + vmdk on the fly. or better said, you can't enlarge a vmfs / vmdk on the fly. so this is the little extra flexibility you gain.

Duncan

-


If you find this post helpful, please award points using the Helpful/Correct buttons.

Reply
0 Kudos
jkumhar
Enthusiast
Enthusiast

For deciding the LUN size of for ESX server,there are several factors

1.The number of virtuals machines which will be hosted on the ESX server.

2.The type of application which will be running on the virtual machines...like Exchange,MS SQL,Oracle etc.Since these applications are I/O intesive application,so it will be better that we should choose the LUN size of not more than 400GB for the optimal performance of the virtual machines.If we are having more space requirment then we can connect more LUN's of small size.

3.The type of software used for the backup of ESX server and the SAN replication.If the size of the LUN's are large then it will take time for the replication of the SAN.

Reply
0 Kudos
jlauro
Expert
Expert

The size of the LUNs is partly dependant on the speed of your SAN. I do mostly 2TB LUNs. One of my VMs has a 800GB second drive, and several are at least 300GB (a few are 4GB)... RAID 10 works much better for holding a larger number of virtual machines than RAID 5 (better random I/O performance with RAID 10, and more VMs is more random). It really doesn't help performance that much if you have one 2TB vmfs, or 5 400GB ones. Earlier version of ESX it made a bigger difference, but even then the main difference is the number of virtual disks per VMFS is more important than the size of each VMFS partition.

You can grow vmdk on the fly (have to shutdown guest, and quick grow). It depends on your SAN if LUN growing is easy or not, but VMDK is always easy to grow... and even if you SAN allows it, it still generally requires the VM to be rebooted to take effect.

Reply
0 Kudos
Preetam_Zare
Contributor
Contributor

Thanks I found this information quite useful, in deciding the LUN size.

http://vmzare.wordpress.com

- Preetam Zare
Reply
0 Kudos
BigHug
Enthusiast
Enthusiast

How about NFS datastore. VMware recommends not to put too many VMs on the VMFS due to I/O or metadata contention. Does the same guideline apply to NFS datastore? BTW, I know VMware doesn't recommend NFS on the production. But it seems to be working pretty well.

Reply
0 Kudos
Preetam_Zare
Contributor
Contributor

it all depends upon workload characteristic. At least no one would recommend to use Production machines on NFS you might want to save cost on for some temp machines.

- Preetam Zare
Reply
0 Kudos
Sangokan
Enthusiast
Enthusiast

We just has implemented esx and had to make a lun design. I spend some time with a vmware implementer and we design some small LUNs for the C: partitions and some other for Data partitions. Now when the Dell SAN engineer came to create the LUNs while installing the SAN he stated that the only way to spread disk IO was between RAID groups within the SAN, saying so he implicitely said that no matter what size were the LUN you have to spread over RAID groups and not LUNs.

Now from an esx point of view i am still wondering if it is relevant to create multiple LUNs.

Reply
0 Kudos
jlauro
Expert
Expert

There are times when the hosts has to lock the disk, and in those situations the number of VMs per LUN make more of a difference than the disks behind those LUNs. This is more of an issue with ESX 2.x than 3.x.

Reply
0 Kudos
NYSDHCR
Enthusiast
Enthusiast

What is the downside (besides management) in having many small single Volumes containing single vms? Is this not a good idea from a performance standpoint?

Reply
0 Kudos
Si118d
Contributor
Contributor

Hi,

There is no performance benefit to be had putting each VM on its own VMFS volume, unless you spread each VMFS volume across enough discs and not share it with anything else. Beside this the management overhead would be a nightmare especially regarding future expansion or DR.

Si

Life, The Universe, & Everything

Life, The Universe, & Everything
Reply
0 Kudos
Ken_Cline
Champion
Champion

What is the downside (besides management) in having many small single Volumes containing single vms? Is this not a good idea from a performance standpoint?

There is a maximum of 256 LUNs per server. If you want to have the ability to migrate any VM to any host in the cluster, then all hosts must have all LUNs presented. That means that if you create one LUN per VM, you are effectively limiting your cluster to running NO MORE THAN 256 VMs. Depending on the size of your environment, that could be a significant limitation. Given that a single dual-socket, quad-core host can fairly comfortably support 30-40 VMs, you would max out at a cluster size of 8 or 9 hosts (allowing for a single host failure). If you deploy larger hosts or more hosts per cluster, you could hit the ceiling very quickly.

As for the performance aspect - there could be a small boost by creating multiple LUNs (more command queues in operation and fewer entities impacted by SCSI reservations during metadata updates), but I think you'll find that, in most environments, you're not going to be hitting an I/O bottleneck that would be significantly addressed by using many LUNs. You'll probably also be creating LUNs across fewer spindles, which would decrease performance. I'd suggest starting out with a plan to put a conservative number of VMs (say 10 or so) per VMFS. If you don't see a bottleneck, then you can look to increase the density. If you do have specific VMs that you feel may benefit by some isolation, work with them on a case-by-case exception-only basis. Your best bang for the buck will come from putting lots of cache in your storage subsystem and running as many spindles as you can.

Ken Cline

Technical Director, Virtualization

Wells Landers

VMware Communities User Moderator

Ken Cline VMware vExpert 2009 VMware Communities User Moderator Blogging at: http://KensVirtualReality.wordpress.com/
Reply
0 Kudos
williambishop
Expert
Expert

The downside of 1 to 1 vm to lun is management overhead. In my environment, I'm betting that over 2000 vm's would equate to a WHOLE lot of pain. Even small shops however, would make this impractical. Not only do you have the limit on the number of luns a server can see, almost all arrays have a limit of how many hosts/initiators that can be addressed. Even on a large frame like the dmx series, you could only put 64 initiators on a port....In addition, you'll have to think about presenting those luns...Think about the sheer number of entries per host/host group!

--"Non Temetis Messor."
Reply
0 Kudos