It all depends on how big you make your vmdk for a virtual machine. Usually the default is recommend but again it depends on the size of the virtual machine disk. Usually anything over a 100GB should be a RDM file.
It really depends on the size of your LUNs and the size you want your VM hard drives to be. We had a couple LUN's around the 1TB with large vmdk files so we went 4MB but that was what worked for us, on smaller LUNs with smaller hard drives we used smaller block sizes. I can't remember exact numbers but I believe the sizes are 1MB = 256GB vmdk files, 2MB = 512GB, 4MB=1TB
In addition to what the above posters have mentioned, I would throw in the notion that SVmotion makes that decision an easier one to live with than it was before, where if you ran up against the max file size on a LUN it was more of a hassle to clear off the LUN (cold migrations with corresponding downtime) to reformat it if necessary. So, if your infrastructure is such that you can make use of SVmotion, then make the best educated guess of your largest potential .vmdk file and pick your block size accordingly. I tend to go with larger block sizes as I've already been burned with a VM that had to have a larger file than my VMFS volume would accomodate, and I have some room to spare on my LUNs. However, if you're more space constrained or have mostly small VMs, then go for a smaller block size. Either way, SVmotion is your friend when the unimaginable happens!
Don't forget 8M = 2048GB
VMware Communities User Moderator
Why should everything above 100GB be a RDM?
Care to explain?
In a recent VMware user group meeting I asked the exact same question and the general consensus was that the block size you choose when creating the VMFS has no bearing on performance etc. The only limitation is the maximum size of a single vmdk. If you're going for large vmdks, increase the block size.
Re: RDMs for vmdks over 100GB, I think it depends what the vmdk is being used for. If it's a large vmdk, using RDMs gives you flexibility to move between virtual and physical (that's why I've used RDMs).