ESXi 5.1 and Slow NFS speeds

[stu01]

I can't seem to come to any sort of conclusion on what's going on here.  I have successfully connected two separate ESXi 5.1 boxes with an NFS share on a FreeNAS box. The connection is easily established and I can browse, create folders, and upload/download to the datastore via NFS, but the speeds are horrendously slow (1KB/sec?!).  This behavior is consistent between both ESXi boxes.  However, there are other NFS shares connected to Ubuntu instances that experience normal/fast speeds.

I did some digging and logged some of the NFS errors on the ESXi boxes:

2013-03-02T10:08:02.571Z cpu0:2899)WARNING: NFS: 1990: Failed to get attributes 
(I/O error)
2013-03-02T10:08:02.571Z cpu0:2899)NFS: 2053: Failed to get object 36 7ec0861b b
c711aa4 75e5ba38 b77930a 4000a 0 15ded1 0 0 0 0 0 :I/O error

2013-03-02T11:09:14.224Z cpu0:2050)NFS: 4544: Received nfsstat3 code 791 for the
 procedure 3 client 0x410015601d20 sock 0x4100180b1100 server's universal addres
s 10.0.0.10.0.111

2013-03-02T11:09:14.340Z cpu0:2050)NFS: 4544: Received nfsstat3 code 2049 for th
e procedure 3 client 0x410015601d20 sock 0x4100180b1100 server's universal addre
ss 10.0.0.10.0.111

2013-03-02T11:09:14.356Z cpu0:2050)NFS: 4544: Received nfsstat3 code 2 for the p
rocedure 3 client 0x410015602090 sock 0x410018036cd0 server's universal address 
10.0.0.10.8.1

I also did a nfsstat on the FreeNAS box and found bunches entries in getattr:

Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0         0
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
        0         0         0         0         0         0         0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
        0         0         0         0         0         0         0         0

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
     3781        60       163         0      4909     65837         8         6
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0        43      1682
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0        68         4         0         0
Server Ret-Failed
              119
Server Faults
            0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses
        0         0         0         0
Server Write Gathering:
 WriteOps  WriteRPC   Opsaved
    65821     65837        16

Again, connecting to the NFS share is fine...I know NFS is working properly on the FreeNAS box becuase other Ubuntu linux machines are able to transfer without speed issues at all.  I'm trying to figure out what exactly to tweak and where - are settings in ESXi or in the NFS server wrong? It also seems that, since the behavior is consistent between the two separate ESXi boxes, that the NFS on the FreeNAS box just isn't tweaked right for ESXi to play nice.

[SG1234]

interesting - from esxcfg-nics -l --> have you verified that the NICs are linked fine?

are there physical switches connecting to the freeNAS box? are the switch ports showing any errors --  is jump frames enabled on the vmknic?

whats the latency from vmkping like??

HTH

~Sai Garimella

[ramkrishna1]

Hi

Welcome to the communities.

Please check physical port or host NIC configuration.

"concentrate the mind on the present moment."

[montanabob]

Have you checked this isn't just the sync write issue w/ESXi and NFS?  i.e. everything the esxi box issues is a sync write.  or are you noticing the issue on reads as well?

[Scissor]

*edit* removed my rambling reply with mistakes and replaced with below *edit*

More info about this behavior can be found at this link:

Sync writes, or: Why is my ESXi NFS so slow, and why is iSCSI faster? | FreeNAS Community

[cforger]

Hello,

I assume you're writing to a ZFS pool? If so, I have a fix for you. If it's UFS, this may still work, but it's untested.

FreeNAS is using FreeBSD, and the way I fixed slow NFS speeds in ESXi was by patching the NFS code to ignore O_SYNC commands from ESXi. Basically I'm making it more like iSCSI for speed, but with the ease of NFS.

Check out : Technical Musings: Speeding up FreeBSD's NFS on ZFS for ESX clients

That patch still applies for FreeBSD 10. I'm currently running through benchmarks in FreeBSD 10, and I'm eying cutting out the NFS cache subsystem as it's unnecessary duplicated overhead when you're working with a ZFS pool. Not sure how many people use FreeBSD/FreeNAS as their ESXi SAN technology, but there is a lot of performance to be had with the right code.

I've run 10+ machines in production for years with the patch, and have yet to experience any corruption issues.