I'm trying to optimize iSCSI performance on a small cluster of ESX 4 hosts attached to a couple OpenFiler NSAs.
We have dedicated NICs & switching for our iSCSI traffic. MTU is set to 9000 across the board. Things generally seem to run well, but I'm looking for any additional tweaks I can perform.
In advanced setting on the software iSCSI initiator, I have these options:
Header Digest Prohibited Data Digest Prohibited ErrorRecoveryLevel 0 LoginRetryMax 4 MaxOutstandingR2T 1 FirstBurstLength 262144 MaxBurstLength 262144 MaxRecvDataSegLen 131072 MaxCommands 128 DefaultTimeToWait 2 DefaultTimeToRetain 0 LoginTimeout 5 LogoutTimeout 15 NoopTimeout 10 NoopInterval 40 InitR2T No ImmediateData Yes DelayedAck Yes
In OpenFiler's target configuration, I have the following options:
HeaderDigest None DataDigest None MaxConnections 1 InitialR2T Yes ImmediateData No MaxRecvDataSegmentLength 131072 MaxXmitDataSegmentLength 131072 MaxBurstLength 262144 FirstBurstLength 262144 DefaultTime2Wait 2 DefaultTime2Retain 20 MaxOutstandingR2T 8 DataPDUInOrder Yes DataSequenceInOrder Yes ErrorRecoveryLevel 0 Wthreads 16 QueuedCommands 32
In both cases, these are the default settings. It is obvious some of them don't match up, but I'm not able to determine what the best values are for each, or if it even really matters that they don't match on each side.
Also, by changing from the default MTU of 1500 to 9000, should any of these settings be tweaked?
Openfiler didnt use to support iscsi reservations and we have had corruption due to that.
Best to use NFS IMO instead on openfiler and then mess with the nowait, noatime kind of tweaks. Plus use a fast suitable FS
little to no performance hit, plus you should get file caching on the openfiler OS to help.
Been running freenas in a similar, small scale rig, also with NFS.
So far, so good. Light workload. If I had to really put the spurs to it, I'd probably roll to openfiler or start thinking about shelling out the $$$ for some real storage.
I wouldn't say things are unraveling, been a rock ever since ESXi 3.5 came out.
As we've filled more and more capacity, there's been a decline in performance. I realize the disks will only do so much, especially with random reads, but still want to make sure the iSCSI settings are optimal so we don't take a performance hit at that layer.
BTW: We did test caching - with the size of our datastores and the data access patterns, we couldn't get enough RAM in the server to actually hit the cache with enough frequency to be worth it. Plus, cache misses seem to be about 30% slower than just disabling the cache and doing direct IO all the time.
Regardless of the specifics of the storage backend we're using, is there anybody with enough iSCSI knowledge to help me fine-tune these settings?
Can you help with some NFS tweaks?
OK, grabbed a spare box and set up OpenFiler on it. Fully patched OF.
Set up an iSCSI target, attached to an ESX server, migrated a test VM, ran IOMeter: 449 IOPS, 6.4 MBps, 2.89 MS latency
Wiped out the target, set up NFS on the same disks. Attached to ESX, migrated the same test VM onto it, ran IOMeter: 117 IOPS, 1.51 MBps, 8.51 MS latency
All settings default on OF with regard to NFS, using XFS for the filesystem.
Any pointers on where to look first? I would think on the same hardware performance numbers should not be that far apart.
As already, was told, OpenFiler is good only if you are using it as NFS - tweaks you can find here - https://forums.openfiler.com/viewtopic.php?pid=7285
And if you are looking for good iSCSI solution- i would offer you to try StarWind, Datacore or Lefthand.