We are using some Linux Virtual Machines as NFS servers, sharing a few small file systems to 3 - 4 clients. I know this is not a common use for a VM, but we often have need to share files, and NFS is quite convenient. (The file systems I am sharing are on SAN storage, and their performance is great. This is not a disk I/O issue.)
Problem is that the NFS performance is highly variable. I cannot determine what is causing the high degree of variance in the NFS performance; it is nothing obvious like network or I/O load on the ESXi host or the NFS client or server. I've used NFS often on physical servers and I know how to tune it. On VMware I'm just not seeing what is causing the slow down. A batch job can go from taking 20 seconds to over 5 minutes. A recursive listing of the NFS file system takes anywhere between 4 seconds and 18 seconds. I do not encounter this degree of variance when using physical NFS servers.
The problem seems specific to NFS rather than network. I use netcat to test TCP performance between NFS clients and server, and it is good, without much deviation.
Any ideas on what I can look at? Thanks.