XFS file system on RAW mapping or VMFS partition?

[etieseler]

I am working on setting up a file system to handle millions of small files that will be randomly accessed. They will be served up via web servers using either NFS or SMB to access them. I like what I have been reading about XFS and some of its features such as parallelism. I want to have this be very high performance and that made me think about some things: are there any penalties or benefits of using either RAW mapping or putting the XFS partition on a VMDK stored on a VMFS partition? The back end storage is a Compellent SAN with tiered storage. Basically, does anyone know if it will make any differences in the way XFS performs if its built directly on the disk, or built onto of a VMFS partition?

I would be happy to accept any suggestions as well, I am in the designing phase now, so any thoughts or input would be helpful.

Regards,

Eduard Teiseler

[Finikiez]

>if it will make any differences in the way XFS performs if its built directly on the disk, or built onto of a VMFS partition

It will make difference when there are other VMs on the same VMFS datastore. If you have single vmdk on dedicated VMFS I wouldn't expect any difference compare to RDM. Or they will be very-very small.

Other question - how much data will you have? do you really want to virtualize it or it will be easier just add RDM with data to a VM?

[jnielsen7]

@Finikiez

@etieseler

I have the same question. In my case I will be using at least 90TB of space. Even with the advancements in VMFS-5 in terms of addressable space and possibly even performance I'm wondering if a 62TB vmdk (the max addressable size in VMFS-5) + 28TB vmdk that are combined into one 90TB logical volume (using LVM in RHEL) is going to be majorly cumbersome in comparison to a single 90TB RDM mapping to a LUN. I also wanted to use XFS as the file system since I've heard that it should play nice with RHEL environments.

I am wondering about a migration scenario though. I've heard you can convert RDM from physical to virtual to migrate files (to another SAN, for example), but I know little about how that actually works.

Can anyone take a shot at answering this question?

Best regards,

Josh

[Finikiez]

Hello!

in comparison to a single 90TB RDM mapping to a LUN

It doesn't matter if you use RDM or vmdk maximum size for a single disk can't be more than 90Tb

https://configmax.vmware.com/guest

Or to say more - 64Tb for physical RDM and 62Tb for virual RDM.

You can switch between physical RDM and virtual RDM by readding it to VM, read KB article VMware Knowledge Base

Also you can convert virtual RDM into vmdk . Read this article Migrating RDMs, and a question for RDM Users. - VMware vSphere Blog

[jnielsen7]

Hi Finikiez! Thanks for the reply.

It doesn't matter if you use RDM or vmdk maximum size for a single disk can't be more than 90Tb

https://configmax.vmware.com/guest

Or to say more - 64Tb for physical RDM and 62Tb for virual RDM.

Oh, bummer. Good to know though!

Several thoughts & questions in the following.

I've actually never used RDM before so I still have a question of whether you can treat it like a normal virtual disk entity, meaning (say for a Linux guest OS) it shows up as a discrete device in /dev and each mapping can be managed in LVM as a physical volume which I can map a virtual volume to, just like with a vmdk disk.

But my foremost question is simply the pros & cons of RDM vs. VMDK. Do you think I will get better performance, mobility, and scalability with RDM? Mostly I have to a support a project which will possibly double the storage requirements over the course of a year and it needs to be scalable. When I thought of a migration scenario where I needed to move several 62TB vmdks I chafed at the idea. Even with RDM I imagine migration from one SAN's LUN to another is quite a task, and won't be fast.

Snapshotting is not a requirement since typical VM backup solutions (such as Veeam) are not going to work well for the volume of data we have anyway, so we will probably buy an identical SAN appliance and set them up for block-level or lun-level replication.

Are there any potential gains to using RDM in that scenario?

P.S. I'm also curious what happens if an RDM mapping gets deleted. Does the data get wiped out or does it remain on the SAN to where you could make a new mapping to the same LUN (maybe even in a completely different VM) and that data would automatically appear in the RDM volume? If the latter is the case then it would sound safer than having the data on vmdks which could be irreparably lost if the VM was accidentally deleted or corrupted.

Sorry for all the questions. Thanks again for engaging!

-Josh

[continuum]

Here is a list of the options you have to store a VMDK:

1. thin provisioned vmdk + snapshots / VMFS

2. thin provisioned / VMFS

3. thick provisioned - mapping table unknown / VMFS
4. NFS

5. thick provisioned - mapping table is known / VMFS

6. RDM

In my biased view ( I offer VMware recovery support as a freelancer )
I have listed the options to the parameter Longterm-Reliabilty) from 1 = poor to 6 = best
My clients usually have problems with #1 - #3..
With RDMs the VMFS-layer is not involved at all - so the probabilty to survive power failures without issues is way better.
So I would recommend to use RDMs when ever you can live with the drawbacks ( less portable, snapshots should be avoided)

About XFS vs EXT .... I prefer EXT - simply because the tools that can help with recovery/ repair tasks seem to be have better results with EXT.
But to be honest - I dont have a lot of XFS-recovery experience.

[jnielsen7]

With RDMs the VMFS-layer is not involved at all - so the probabilty to survive power failures without issues is way better.

So I would recommend to use RDMs when ever you can live with the drawbacks ( less portable, snapshots should be avoided)

Good information. Thank you. Is there a noticeable I/O performance difference with RDM or would your say normal vmdks on top of VMFS-5 perform comparably?

[continuum]

I have not seen any benchmark tests recently. I would expect that there is no relevant difference.
So the decision you have to make is between portabilty and ease of use for regular vmdks and reliabilty for RDMs.