Tips for improving ISCSI performance?

[dailowai]

I followed all of the instructions from the great post below, but I seem to only be averaging 30mb/s when doing sequential reads. I'm not sure what else I can change, but I know each NIC is capable of going up to 60 mb/s so I'd imagine I should be able to go at least that fast. It's attached to an OpenFiler that can go that fast so I'm not sure where the bottleneck is at.

I've played around with the round robin IOPS settings and it doesn't make much of a difference. Looking at the averages for both my NIC's through the performance tab it seems each is averaging around 15 mb/s. Are there any other tests or settings that I can try to see where the bottleneck is?

http://virtualgeek.typepad.com/virtual_geek/2009/09/a-multivendor-post-on-using-iscsi-with-vmware-vs...

[J1mbo]

How are you testing this?

Could you give some detail of the setup?

Please award points to any useful answer.

[dailowai]

I just created a VM on the iscsi datastore and have been running tools like "HD Tach" and "Iometer" to generate bemchmarks to see how fast the sequential reads are. On a Linux guest I also just did a simple dd test and I've was consistently getting 30mb/s across the board. I just made some changes on the Openfiler side and bumped it up to 60mb/s! Not bad although it'd be nice i fit was a little more faster

Before I didn't have any additional settings in my configuration file, but I added the parameters below from a thread I saw on the openfiler forums and it seemed to bumped up the average speed.

HeaderDigest None

DataDigest None

MaxConnections 1

InitialR2T Yes

ImmediateData Yes

MaxRecvDataSegmentLength 262144

MaxXmitDataSegmentLength 262144

MaxBurstLength 262144

FirstBurstLength 262144

DefaultTime2Wait 10

DefaultTime2Retain 20

MaxOutstandingR2T 8

DataPDUInOrder Yes

DataSequenceInOrder Yes

ErrorRecoveryLevel 0

[J1mbo]

Using IOMeter with 32K sequential read or write you'll need a fair queue depth specified (outstanding IOs) - say 32, to get good throughput, and a large test space (4GB+) to reduce the impact of caching and get meaningful results.

Try to enable flow control on your switch, if it is supported, and leave all connections as auto/auto (speed/duplex).

Please award points to any useful answer.

[Josh26]

When you say "openfiler can go that fast" have you confirmed this in some sort of testing, or is this based on figures from the guys that produced OpenFiler?

[dailowai]

I ran Iometer on the Openfiler array and was able to get 400 mb/s sequential reads on the array I have setup on there. Running iPerf from a guest VM on the ESX machine to the array I was able to get 60 mb/s per NIC. I figured with two NIC's I should be able to achieve around 100 mb/s of sequential read throughput. Looks like maybe I need to tweak the ISCSI settings some more as when I added those parameters it seemed to double the performance.