It seems that your mind is made up. But try changing the scheduler on the NFS box to noop.
As I mentioned above, but the partiton needs to be created aligned properly on the NFS server, but much more importantly the guest partition also needs to be 4K aligned.
There are some other tweaks too but I'm saving those for my book 
Please award points to any useful answer.
I'm willing to try--but probably not enough to buy your book!
Your changes are tweaks. They certainly might be able to get back some of the 15% or so performance drop I'm seeing with the right hardware, but it's not going to help the 50% to 75% performance drop I'm seeing on commodity hardware!
To quote Michael Abrash, "Profile before you optimize." I'm not trying to wring out the last few percent of performance. I'm trying to figure out why performance with NFS datastores on commodity hardware falls into an abyss. noop ain't gonna fix that...
And for the record, here's the result with elevator=noop added to the kernel command line:
root@iridium /# sync; sync; sync; time { dd if=/dev/zero of=test.txt bs=1M count=256; sync; sync; sync; }
256+0 records in
256+0 records out
268435456 bytes (268 MB) copied, 23.0501 seconds, 11.6 MB/s
real 0m37.009s
user 0m0.000s
sys 0m0.781s
root@iridium /# mount 172.28.19.16:/mnt/InternalRAID1/shares/VirtualMachines /mnt
root@iridium /# cd /mnt
root@iridium mnt# sync; sync; sync; time { dd if=/dev/zero of=test.txt bs=1M count=256; sync; sync; sync; }
256+0 records in
256+0 records out
268435456 bytes (268 MB) copied, 8.87204 seconds, 30.3 MB/s
real 0m9.116s
user 0m0.000s
sys 0m0.349s
root@iridium mnt#
No meaningful difference.
I never said the book would cost
BTW, how are you running dd? Is this from ESX console, or within a VM?
I just added a SATA drive to a Debian VM providing NFS shares to ESXi. Using IO-Meter in the Windows VM I get read or write at about 40 MB/s. 8K random with 2GB test file and 8 IOs outstanding gives about 115 IOPS at 66ms (70:30 read:write).
Deliberately misaligning the guest read or write is about 27 MB/s and 80 IOPS at 100ms on the same basis.
All on quite a full WD Caviar-green so numbers look OK to me.
Re command scheduler, switching it from cfq to noop for a device with IO-Meter ultimately running against provided by a SATA RAID-10 volume on a Perc-5 doubles the sequential throughput on my test box...
Please award points to any useful answer.
I have a feeling that the poor network performance is from the NAS (a VIA-based system) on the other end of the iperf test, not theESXi system. So, I reran the iperf test, with the ESXi system against the IBM x236. Better results:
-
Client connecting to 172.28.19.17, TCP port 5001
TCP window size: 16.0 KByte (default)
-
local 172.28.16.100 port 41688 connected with 172.28.19.17 port 5001
0.0-10.0 sec 1.09 GBytes 935 Mbits/sec
Another strike against commodity (or in this case, embedded) hardware!
Next I'll try putting OpenFiler on an Intel-based machine with either an Intel or Broadcom NIC and see what the numbers are. I have a strong suspicion that we will see similarly-improved iperf numbers, but still see terrible NFS performance. If I do, I think we can put the nail in the coffin of any other excuse besides terrible NFS performance...
Unless someone else has any other bright ideas? I've come this far, and I've got the lab set up. If it isn't too difficult, I'll test it.
Frankly, at this point, I've virtually written NFS off. The only reason I was considering it was for simple full-VM backup. I've found that I can achieve the exact same thing with iSCSI using the (unsupported) ESXi console in the lab (and for production I can use the ESX console or VCB). Looks like iSCSI is the clear winner on commodity hardware.
So we all agree: Battery Backed Cache is important for ESXi in general, but essential for NFS in particular.
Would have been more useful if you had said that instead of asking me to change I/O schedulers and worry about alignment! I was very clear that my testing was on commodity hardware (and specified as SATA/software RAID1).
As long as you are using NFS without a BBC, performance will be a small fraction of what the hardware is capable of delivering. (Can I mark my own answer as correct / helpful? )
However, if there is anything else that someone wants to suggest regarding other ways BESIDES BBC to improve NFS performance with ESXi I am all ears--and my test machines stand ready to go!
Sorry but something is a miss here; BBWC does exactly that (speed up writes) since from the NFS servers perspective the storage hardware completes the write with near zero latency, the controllers caching and coallescing converting individual block writes (read-update-write) to full-stripe writes. This should be faster than an SSD since it's completed at DRAM speeds. To get the write performance from ESX the export needs to be async too.
NFS tuning for a particular server is vital, but retesting this in some depth actually I can't get much improvment in writes. Reads and IOPS are a different storey - that Debian install runs at 60MB/s and 380 IOPS 'out the box', rising to 115MB/s and 480 IOPS with some tuning
Please award points to any useful answer.
By the way, re alignment, it really is important and I can't emphasise this enough. Consider what's happening with a mixed workload (70:30 I use for testing read:write) running from a guest with misaligned partition relative to NFS blocks.
As everyone agrees, ESX will complete writes serially. The guest asks to update an 8K block that staddles three 4K NFS blocks. ESX reads the three 4K blocks, updates with the new 8K worth and then writes back the three blocks. This obviously adds a massive penalty to writes (physical disk seek+latency regardless of BBWC since the read needs to be physically delivered before the write can commence) but also completely blocks IO in the process. This effect can be seen on OpenFiler, for example, running iostat from SSH and looking at avgqu-sz which will be close to 1 during such testing. With a queue depth of only 1 the array will perform pretty much like a single disk running read-update-write, regardless of how many physical disks are present. Because of this the impact gets worse, relatively, as the number of disks are increased - my own benchmarking run against a 6x 15k SAS RAID-5 array (Perc-6i, 256MB BBWC, OpenFiler) showed 220 IOPS misaligned, rising to over 1200 with everything set up properly
Anyway, HTH
Please award points to any useful answer.
Remind me again what the advantage of NFS is?
Seriously, the only advantages I saw with NFS were all on the low end: not as low as pre-boxed NAS systems that you get from a retail store, but lower than a typical entry-level EMC or NetApp setup. However, by the time you build a storage unit that will make ESXi happy, AND make sure that you properly align every last guest (and that means aligning it with the NFS block size, the filesystem block size, and the RAID array block size), you've eaten up a lot of the initial savings by avoiding a "real" storage array in the first place.
The only area I see being easier with NFS than with iSCSI is backup of VM's outside of VMware. That is really nice, but not that nice! And I can mostly get around that, even with iSCSI.
So, due to the performance issues, seeing as you can't actually use a low-end NFS-based NAS system for anything but the most trivial uses, what are the advantages of NFS on anything besides a NetApp Filer?