RDM 2TB limitation on ESXi 5.5

[MKruer201110141]

I just purchased a 6TB drive for my home storage/media server. But have a question on the RDM mounting logic. ESXi 5.5 see the full drive. However just like this post, and several others,  (ESXi 5.5 rdm local storage 2tb limit ?) I am seeing the same issue when mounting the RDM because the apparent limitations of 2TB for the max size of a vmdk file.

I am reading a lot of conflicting information. According to the tech doc http://www.vmware.com/pdf/vsphere5/r55/vsphere-55-configuration-maximums.pdf ESXi 5.5 should be able to present a single vmdk up to 64TB. However all the instructions for creating the RDM seem to be limited to the 2TB limit.

Below was the command I used to create the RDM

vmkfstools -z /vmfs/devices/disks/t10.ATA_____WDC_WD60EFRX2D68MYMN0_________________________WD2DWX11D4447554 /vmfs/volumes/datastore1/Raw_Disks/RDM_WDC_WD60EFRX_WX11D4447554.vmdk -a lsilogic

Can someone clarify the issue and offer a solution around this limitation?

[JPM300]

This is probably due to the fact that your new drive looks to be locally attached.  You can't do RDM drives with directly attached drives, well not natively anyhow.  however I think I remeber seeing some blog posts on how people found ways to do it.  With that said RDM is reservered for iSCSI / Fibre connections.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=101770...

Hope this has helped.

[MKruer201110141]

Yeah, I forgot to mention that, yes this would be a locally attached SATA drive. I recall seeing the Raw Device Mappings as an option and it is greyed out.  Soon as i get done with my burn in test, I will mount the disk back into the server and then see if i can get it to mount correctly.  Thanks for the link. I will try this out as soon as I can

[JPM300]

Np, I hope it works out for ya

[MKruer201110141]

So I add the drive back, and that still did not work for me. The Raw Device Mapping is still greyed out. I can mount the drive data store, but the max file size of the VMDK is still only allowing 2TB images. Reading more about this,

VMFS5 uses 1 MB blocks throughout (with block suballocation for small files), and has a file size limit of 62 TB, though the VMDK size is restricted to 2 TB - 512 B in ESXi versions earlier than 5.5 due to a limitation in the version of SCSI emulated.

My current Setup is.

ESXI 5.5.0 1623387

ASRock c2750d4i CPU/MB Combo

So Yeah, whats the real story? Is this one of those "you have to use vcenter to enable this features" deals?

[jlong]

If you're trying to create a vmdk larger than 2tb on esxi 5.5 you can connect to your host using vmware workstation and create a larger than 2tb vmdk without the need for vcenter. Just use the latest version of vmware workstation. I've done it and it works. One limitation I noticed is if you create a new vmdk with vmw workstation it's thin provisioned and wants to create multiple vmdk's. I created a 2 tb vmdk in the vic that was thick/eager 0'd and then used vmw workstation to expand the vmdk. That at least kept the vmdk as 1 file and I would assume that it's still thick/eager 0.

I have no experience with RDM so I can't comment on that. Hope this is what you were looking for.

[admin]

MKruer201110141 wrote:

I am reading a lot of conflicting information. According to the tech doc http://www.vmware.com/pdf/vsphere5/r55/vsphere-55-configuration-maximums.pdf ESXi 5.5 should be able to present a single vmdk up to 64TB. However all the instructions for creating the RDM seem to be limited to the 2TB limit.

Can someone clarify the issue and offer a solution around this limitation?

Do not use an RDM. 62TB vmdks are created on VMFS volumes, not RDMs. RMDs are passthrough devices as bare storage for VMs and have nothing to do with a vmdk.

Create a VMFS volume and then use the Web Client to create a vmdk of your choice on that disk, as this is a new feature of 5.5 it is only available in the Web Client (or VMware Workstation as mentioned above).

[MKruer201110141]

Frank, thanks for the info, and clearing up my suspicions.

A few things, I am running a personal edition of ESXi5.5 at home, and unless the licensing has changed I cannot use web client because the web client will not register a personal ESXi with the  current keys.

Second, the reason why I am trying to use RDM is because this drive will end up being a media drive not running nor saving any vms to it. Also not to say I don’t trust VMFS5.x but if I use another format, like ext# or ntfs, in case something go’s wrong with the media, there are a lot more "tools" to recover data should I choose too.

That being said. It looks like I am able to access the full drive as a Virtual RDM, not a Physical RDM. For those that may come across this thread later ("vmkfstools -r" not "vmkfstools -z" when creating the link)

I am still testing things out, but so far so good, I am verifying that indeed the VM can see, read and write to the vRDM. The only downside to the Virtual RDM that I see right now is that it doesn’t expose the SMART Data to the VM system attached, unlike Physical RDM which does.

I hope to see this resolved in the future by exposing the SMART data on the vRDM or by allowing for large local SATA storage pRDM

[MKruer201110141 · Accepted solution]

After testing this out I can confirm that this will work on an ESXi 5.5.0 Build 1623387,  it also works for a vmx-08 Virtual Machines.

So for those having issue with mounting > then 2TB for local storage to a vm creat the RDM Mount, but use the "-r" not the "-z"

i.e. vmkfstools -r /vmfs/devices/disks/<device> /vmfs/volumes/datastore1/<VM Name>/<Virtual Raw Disk>.vmdk -a lsilogic

Additional information on finding your raw disk How to create an RDM mapping file via the CLI

You will loose the ability to monitor the SMART data on the drive, but hey its works.
The performance delta is within the margin of error for RAW drive

[Dropnet]

been looking for hours for this issue so might as well confirm as of today (build 2302651) its still not possible to use rdm +2 TB with  pass thru

[ErikCarlseen]

OK, finally got this working as a proper >2TB VMDK file, without the web client. It's not brutally difficult, but not for the faint of heart either.

1) I was able to use vmkfstools from the ESX shell (should work from the console or SSH) using ESXi 5.5U2 (Build 2068190):

vmkfstools -c 8192g "/vmfs/volumes/54e8c719-145dc484-8f04-002590fcf4c0/My_Server/My_New_Volume.vmdk"

2) I had to manually add these lines to the vmx file associated with the virtual machine, while it was powered down:

scsi1:1.present = "true"           

scsi1:1.fileName = "/vmfs/volumes/54e8c719-145dc484-8f04-002590fcf4c0/My_Server/My_New_Volume.vmdk"

scsi1:1.deviceType = "scsi-hardDisk"  

scsi1:1.redo = ""

Obviously this is well outside of the realm of supported behavior, so Your Milage May Vary, don't blame me if it nukes your server or makes your cat suicidal or whatever. But it did work for me, and I have a nice happy 8TB virtual disk attached to a VM on my lab ESXi box without having to use the web console.

Editorial comment: This split between the C# client and the web client and the lack of full management functionality in the ESXi hypervisor is ridiculous. We always use vCenter in production, but we don't license it for the lab and even in production there are times when you need to take care of things without vCenter.