Managing LUN extents and dealing with the 2TB datastore/LUN limit

[skcc_sysadmin]

Hello,

Our organization is planning on using ESX 3.5 (VI 3 installable, not the full ESX server) for hosting some of our file servers.  The problem is that we have a very large amount of data (~16TB), and ESX 3.5 seems to have a 2TB limitation on both LUNs, datastores, and VMFS volumes.  The VMWare engineer I spoke to mentioned that if we were planning on using VM's for file servers that we'd have to essentially take our large amount of data and essentially reorganize it so that we could fit it into ESX datastores of 2TB or less.  

All of our lab's data currently resides on a large (6TB+) NTFS and Linux EXT3 LUNs on our SAN, and we'd have to break it up and allocate it to several datastores.  Such a way of doing things leaves me feeling reasonably uneasy.  I also notice that I'm losing quite a bit of space since on our RAID systems that hook into the SAN switch (Apple XServe RAIDs) can only create equal size LUNs (up to 6 per array).  So what I'm dealing with is using six 582.19GB LUNs.  Obviously I can take 3 full 582GB LUNs and a little under half of a 4th LUN to have a full 2TB.  Of course I can't use the last half of the 4th LUN's extents for anything else due to the 2TB VMFS and LUN limitation on ESX 3.5.  The engineer I spoke to also voiced concerns about striping data over so many LUNs, especially of that size.  I can understand that concern, especially when there is the added layer of a VMFS corruption on top of the VM's filesystem (NTFS and EXT3 in our case) potential for corruption, as well.

In your professional opinions, what would a seasoned VMWare admin do with this sort of situation?  I like the benefits that VMWare provides, but I'm wondering if sticking to physical file servers might be our best option.   

[jhanekom]

Tough one... unless you stripe the data over multiple LUNs (as you've suggested), there's no easy way to get around this.

Let me point out that you probably should not place the data on VMFS in this case - you don't really gain anything. The real power of VMFS lies in the facts that it is a multiple-access file system and that it allows you to subdivide a large LUN into lots of smaller virtual machines. In your case, you won't be doing either of these.

If you do end up virtualising the machine, raw device mappings (RDM) would probably be more appropriate.

[TomHowarth]

I would stear clear of extents and would investigate the use of RDMs rather than VMFS for the larger file stores

Tom Howarth

VMware Communities User Moderator

[skcc_sysadmin]

This is a good point, however it sounds like from the ESX 3.5 Configuration Maximums documentation that the maximum RDM size is also 2TB. If I was to use pRDM or vRDM, can I combine multiple LUNs to have more usable space for a single VM? It would seem as if I couldn't, from what I've been reading here: http://www.vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_3_server_config.pdf (pp. 143-150). If that assumption is correct, the only benefit that using VMFS would be the ability to stripe several LUNs together. With our current SAN (Which is really very basic. No real SAN management software to speak of), it doesn't have any mechanisms or software to allow this at the device level. I'm sure many of the EMC SAN's have that capability. Unfortunately, we do not have an "EMC" budget, much like we don't have a "Cisco" budget either.

[jhanekom · Accepted solution]

The best you can achieve with VMFS is concatenation, not striping (nitpicking, I know...)

Even so, you would still not be able to present a virtual disk larger than 2TB to a VM, so VMFS would still be of no real use.

You're quite correct in that RDMs are limited to 2TB as well; I believe this is due to the virtual SCSI adapter in the VM (though I could be wrong.) The only way to circumvent this limit is to concatenate various LUNs inside the guest operating system.

[dascott]

My employers installed a new DAE Tray and created a single LUN storage unit of 5Tb - which unfortunately, ESX3.5 hosts cannot detect.

We then contacted VMWARE to ask if there was any workaround (there isn't) or if the 2Tb limit would be removed in a future upgrade (e.g. ESX4.0).

The response was that VMware have 'no plans to increase the 2Tb limit' and it was suggested we complete a feature request form.

Surely VMware cannot be blind to ever increasing storage trends and will have plans to support larger capacities? If not, then customer could be forced to look elsewhere...