We use NFS extensively in production and are very happy with the performance/scalibility, especially so because we use it in a NetApp MetroCluster configuration - synchronous replication between DC's that are ~70km apart. We have the datastores presented across site (PROD/DR) for flexibility to over 30 hosts (although only use site local storage) and we haven't seen it affect host performance at all - it is fit for purpose.
With regards to TOE, I only have experience of it in a Windows environment and we generally disable it on all systems with high network utilisation for stability. Not sure how this would translate to in virtualisation, but thought I would throw it out there anyway.
So back to NFS ... it would make sense to plan your network infrastucture to cater for your requirements and test performance to get a benchmark of what your infrastucture is capable of (also ensure that you use aligned IO - this is a must for performance). We segregate NFS traffic on a dedicated layer 2 VLAN which is only exported to the ESXI hosts IP storage interfaces. Obviously, you will get better performance from 10GB than 1GB so this would be good to plan / cater for budget permitting.
Hope this helps.
Cheers,
Jon
vExpert 2014 - 2022 | VCP6-DCV | http://www.jonmunday.net | @JonMunday77