iSCSI/VMFS
+ Better performance than NFS
Yes, although this is dependent on many factors. But, everything being equal, iSCSI will be faster
+ Ability to implement clustered Linux VMs in one ESX box
Not sure about this one. I don't see why you even need shared storage to do this, I have done several "Cluster in a Box" setups with Windows with no shared storage at all. I will allow others to comment, I do not see a specific plus/minus here.
+ Ability to share VMFS between multiple ESX servers (stop a VM on ESX1 and start on ESX2)
Yes (and also VMotion between the machines).
+ Possibility to use hardware initiator with TOE
Yes, make sure you get a supported adapter, the list is rather short.
NFS
+ Easier image backup of virtual disks
This is really a "kind of" answer. Yes, you can take SAN snapshots if you use NFS. However, you still need to ensure consistency before snapping. This is a very long conversation, but basically, the answer is, don't think that you can just pop a snapshot of your SQL server running on an NFS datastore and have no problems. Databases will come up suspect, and Windows will want to do disk checks (fsck for Linux).
\- No clustering support
I think this is correct (no cluster between boxes). Please make sure to read the newest documentation; but to the best of my knowledge, this is still the case. Also, it is worth noting that this is MSFT clustering, NOT VMware clustering. That should still work without issue.
\- NFS storage cannot be shared between multiple ESX servers
This is incorrect. NFS can be shared between ESX servers. It functions just like a typical datastore, and will be accessable to mutiple hosts at the same time.
\- TOE's not available
At this time, I think this is still a limitiation. Also, remember, NFS has more overhead higher up in the TCP/IP stack, so TOE is going to be less effective then it is for iSCSI.
Now, what's my opinion? NFS is a file level protocol. Your VMDKs are block level devices (as are RDMs). Using a file level protocol to encapsulate a block level device does not seem like a good idea to me. The only advantage that I see by implementing via NFS is that some array controllers are "smarter" about NFS mount points then they are about iSCSI LUNs.
However, it is very important to note that running on NFS does not change the nature of the snapshot/backup operations! In a typical NFS/CIFS mount, you can just take a SAN snapshot whenever you feel like it, and you will not have problems with consistency. However, that is not the case with VMware NFS mounts (again, because it is somehow encapsulating the block level commands in a file protocol; although the SAN is not aware of this, you have to treat it just like a block device).
My advice would be to use FC or iSCSI. Not because NFS does not work (it does; it seems fine in my experience, although I have never implemented it in production), but because you are accessing the device in block mode, and iSCSI and FC are block mode protocols. Using NFS just introduces a "wierdness" factor that I don't need when trying to architect a site to site failover or consistent data snapshots.
Both of the block protocols are "owned" by the operating system, rather then being owned by the SAN. This is an important distinction; and, IMHO, is the way it should be (as the VMs believe they own the filesystem).
Hope that helps!
