LUNs and ESX Server - Basic Understanding

[theone99]

Hi There...

I'm trying to clarify the relationship that LUN's play in the VI Infrastructure:

1) Is it true that you can only have 1 VMFS volume associated with an ESX Server at any given time?? Or can you have multiple VMFS volumes running on different LUN's all associated with the same ESX Server?

2) I know you have the have "shared storage" between the ESX Servers in a vmotion cluster, but does that "shared storage" refer to

a) the same VMFS volume (regardless of how many LUNs the VMFS volume spans) or

b) or can it be different multiple VMFS volumes (b/c multiple VMFS volumes can be associated with an ESX Server at any given time)

3) If you can have multiple VMFS volumes associated with 1 ESX Server, do the VM's in a vmotion cluster all have to live in the same VMFS volume?

Thanks!

[mcowger]

1) False - you can have many LUNs/VMFS volumes per host - the limit is something like 256

2) both - it refers to any LUN or LUNS that can be sen/accessed by multiple ESX hosts at the same time.

3) no, but every host that you want to be able to vmotion a guest to needs to be able to see the LUN the guest lives on.

--Matt

VCP, vExpert, Unix Geek

[theone99]

Thank you for your response - that leads me to one more question then....

I've read that in order to preserve performance for VM's that do high I/O disk activity you should isolate them to their own LUN - so....

1) Does that mean the relevant VM's should be in a separate LUN which is part of a larger VMFS volume (a VMFS volume that spans many LUNs and those particular VM's happen to be in one specific LUN within the larger volume)?

In other words, when you have a VMFS volume that spans multiple LUN's, do you have the ability to place VM's on a specific LUN within the larger VMFS volume - and isolate those VM's to that one specific LUN - or since it's one large logical volume (VMFS) you don't have control over which LUN within the volume a VM goes to..... the VM just ends up on the next LUN in the volume as LUNs fill up...

2) OR If you want to isolate high I/O disk activity VM's then do you HAVE to put them on their own VMFS volume...

Thanks (again)!!!!

[bulletprooffool]

I believe you'll get some debate on this - some will argue maximising the number of spindles is the best way to get high performance, some will say LUN isolation.

I'd imagin that different environments could justify different answers (sequential / non sequential, high read, high write etc)

We always run multiple VMs per LUN and somemachines are quite hardworking - In fact YellowBricks also do not really head down the 1 VM 1 Lun route . . .

See : http://www.yellow-bricks.com/2009/06/23/vmfslun-size/

[JoJoGabor]

This depends on your underlying storage. I would never use more than one LUN for a VMFS volume (you are referring to extents here) just due to the management of having multiple LUNs. If one LUN is lost the entire VMFS volume is lost.

If you are using NetApp, HP EVA or some storage where you have large groups of disks then split that into LUNs then this will give better performance than say 2 disks per LUN. Isolating a VM to a specific LUN in this case may not get the benefits that you imagine. It depends what type of activity the OS will be performing on that LUN. FOr sequential writes such as a database logs partition a dedicated 2-disk LUN would probably give great performance.

[theone99]

So, does that mean, that you do have the ability to put a specific VM in a specific LUN that's part of a larger VMFS volume??? In other words you can pick and choose between LUNs for VM placement even though those LUNs all belong to some larger logical VMFS volume??

Btw, I am going to be using Exchange 2007 in this situation...

[JoJoGabor]

I'm not exactly sure what you mean here. Assuming you take my advice and dont use extents to merge two LUNs together and format the combined LUNs with VMFS. Each LUN will be formatted with VMFS. You can have many of these VMFS datastores which can be used to store VMDK disk files.

However for Exchange, you may want to take advantage of storage level snapshots or there may be a high performance requirement. Here is what I would generally recommend:

Store the c: system VMDK on a shared VMFS datastore

Store data and logs separately on VIrtual RDMs presented to the VM. These are raw LUNs presented to the VM for NTFS formatting. Depending on your storage system, will affect how you would carve out these LUNs on the SAN.

[theone99]

When you say store data and logs separately on RDM's - do you mean the data will have one RDM and the logs another RDM?? I understand the reason for wanting to put them on separate LUNs, but do they have to be RDMs? Is there a performance advantage for using RDMs? I don't remember reading about a performance advantage for RDM's...

And also, what I was talking about before - being able to pick and choose between LUNs (that have been joined together via extents and formatted as one large VMFS volume) the more I talk about that it seems the answer is more obvoius - you would NOT be able to choose which LUN a VM is on if the LUNs are all joined together via extents and they are all formatted as 1 large VMFS volume.

This question came from the idea that I thought there would be a performance benefit for keeping VMs on separate LUNs (but within the same VMFS volume) - am I just smoking something???? I guess I'm just confused between the advantage of joining LUNs together as 1 larger VMFS volume instead of just keeping the LUNs separate with a VMFS volume in each... Why would I ever join LUNs together in 1 larger VMFS volume - can vmdk files span across LUNs (extents) in a larger VMFS volume?? Perhaps space is the only advantage for joining LUNs in 1 VMFS volume..?

Thanks...

[JoJoGabor]

You may be smoking something, yes

Only use an extent if you need extra space in an emergency. In the long run it is a nightmare. If you need extra performance just add all the physical disks into one array and then make one or more LUNs in that array.

There is a slight (maybe 5%) performance increase when using RDMs, but the main advantage is you can use tools such as NetApp SNapmanager to ensure that when you take a SAN level snapshot the snapmanager tool tells Exchange to commit anything in memory to disk to ensure a consistent database is on disk.

You can store different VMDKs of a VM in different VMFS datastores but in a VMFS datastore made up of 2 LUNs in an extent you wouldn't be able to specify which LUN your VMDK is in.

Typical use of a VMFS datastore will contain maybe 10 or 15 VMDKs from different VMs. For HIgh-IO VMs or important Exchange or database VMs a bit more consideration should be made for disk configuration.

[theone99]

Thank you very much for your reply - very helpful indeed!!!!!

Just one last question - I promise. I understand everything you are saying - but I notice one thing you said that others have said - "Typical use of a VMFS datastore will contain maybe 10 or 15 VMDKs from different VMs"...

Why is this? Performance reason?

I was actually planning on having 1 LUN (1 VMFS volume) which would hold all our VMs - about 35 of them (except for the Exchange server - that would be separate). I thought that since all of these VMDK files are sitting on a RAID'ed group of drives it was unnecessary to divide them up into different LUNs - is this not the case?

Thanks!!!!

[bulletprooffool]

Yup,

Once you go beyond this there will be a large amount of Disk read / write, accross a spread of VMDKs.

You can run it beyond this limit, but I would keep it around that for production environments.

If you're just running test machines or something similar on that, you can ramp the number right up.

You can use ESXTOP to monitor performance to see if you are getting any issues.

[JoJoGabor · Accepted solution]

Also remember that a VMFS datastore cannot be larger than 2TB.

ALso consider what would happen if an administrator accidentally deleted a LUN

VMware will also create SCSI reservations on a LUN basis whenever it creates a new VM amongst other tasks. If these get too high, the LUN is locked drastically reducing Disk performance. SPlitting between LUNs will help this,

[theone99]

Thank you so much for your help. You've been very helpful!!! :smileygrin: