Fine tuning iSCSI performance - ESXi4 + OpenFiler

[ebowser]

Hello!

I'm trying to optimize iSCSI performance on a small cluster of ESX 4 hosts attached to a couple OpenFiler NSAs.

We have dedicated NICs & switching for our iSCSI traffic. MTU is set to 9000 across the board. Things generally seem to run well, but I'm looking for any additional tweaks I can perform.

In advanced setting on the software iSCSI initiator, I have these options:

Header Digest		Prohibited
Data Digest		Prohibited
ErrorRecoveryLevel	0
LoginRetryMax		4
MaxOutstandingR2T	1
FirstBurstLength	262144
MaxBurstLength		262144
MaxRecvDataSegLen	131072
MaxCommands		128
DefaultTimeToWait	2
DefaultTimeToRetain	0
LoginTimeout		5
LogoutTimeout		15
NoopTimeout		10
NoopInterval		40
InitR2T			No
ImmediateData		Yes
DelayedAck		Yes

In OpenFiler's target configuration, I have the following options:

HeaderDigest			None
DataDigest			None
MaxConnections			1
InitialR2T			Yes
ImmediateData			No
MaxRecvDataSegmentLength	131072
MaxXmitDataSegmentLength	131072
MaxBurstLength			262144
FirstBurstLength		262144
DefaultTime2Wait		2
DefaultTime2Retain		20
MaxOutstandingR2T		8
DataPDUInOrder			Yes
DataSequenceInOrder		Yes
ErrorRecoveryLevel		0
Wthreads			16
QueuedCommands			32

In both cases, these are the default settings. It is obvious some of them don't match up, but I'm not able to determine what the best values are for each, or if it even really matters that they don't match on each side.

Also, by changing from the default MTU of 1500 to 9000, should any of these settings be tweaked?

Thanks!

[Smeagol]

Openfiler didnt use to support iscsi reservations and we have had corruption due to that.

Best to use NFS IMO instead on openfiler and then mess with the nowait, noatime kind of tweaks. Plus use a fast suitable FS

little to no performance hit, plus you should get file caching on the openfiler OS to help.

[manfriday]

I agree. I tried to use openFiler for a while too, and while it looked like it worked for a while, eventually things unraveled. If you need to use OpenFiler, I'd go with NFS.

[awliste]

Been running freenas in a similar, small scale rig, also with NFS.

So far, so good. Light workload. If I had to really put the spurs to it, I'd probably roll to openfiler or start thinking about shelling out the $$$ for some real storage.

My two.

- abe

[ebowser]

I wouldn't say things are unraveling, been a rock ever since ESXi 3.5 came out.

As we've filled more and more capacity, there's been a decline in performance. I realize the disks will only do so much, especially with random reads, but still want to make sure the iSCSI settings are optimal so we don't take a performance hit at that layer.

BTW: We did test caching - with the size of our datastores and the data access patterns, we couldn't get enough RAM in the server to actually hit the cache with enough frequency to be worth it. Plus, cache misses seem to be about 30% slower than just disabling the cache and doing direct IO all the time.

--

Regardless of the specifics of the storage backend we're using, is there anybody with enough iSCSI knowledge to help me fine-tune these settings?

[ebowser]

Can you help with some NFS tweaks?

OK, grabbed a spare box and set up OpenFiler on it. Fully patched OF.

Set up an iSCSI target, attached to an ESX server, migrated a test VM, ran IOMeter: 449 IOPS, 6.4 MBps, 2.89 MS latency

Wiped out the target, set up NFS on the same disks. Attached to ESX, migrated the same test VM onto it, ran IOMeter: 117 IOPS, 1.51 MBps, 8.51 MS latency

All settings default on OF with regard to NFS, using XFS for the filesystem.

Any pointers on where to look first? I would think on the same hardware performance numbers should not be that far apart.

Thanks!

[TimPhillips]

As already, was told, OpenFiler is good only if you are using it as NFS - tweaks you can find here - https://forums.openfiler.com/viewtopic.php?pid=7285

And if you are looking for good iSCSI solution- i would offer you to try StarWind, Datacore or Lefthand.