local scsi vs soft iscsi adapter, lun and world queue lengths

[herraa1]

Hi,

esxtop reports the following queue lengths in my ESX 3.0.1 servers:

\- local scsi

ADAPTR = vmhba0

AQLEN = 976

LQLEN = 128

WQLEN = 16

\- software iscsi

ADAPTR = vmhba40

AQLEN = 64

LQLEN = 12

WQLEN = 16

The LUN queue length (LQLEN) used by the ESX servers on the software iscsi adapter seems to be "strange" at least. I would have expected it to find there a large value. It probably makes no sense to allow a world to place 16 requests on a LUN whose esx-related queue is limited by 12 requests.

I'm not 100% sure, but those values could be a limiting factor for the software iscsi adapter performance.

I'm testing software iscsi using both a NetApp FAS940 (2 x 14x72g FCAL RAID DP) and a NetApp FAS3020 (2 x 13x250g AT-FCX RAID DP). All network connections are gigabit ethernet.

None of the filers seem to be a bottleneck as they are able to perform linearly better when used from I/O intensive VMs executed from different ESX servers using the same LUN of the same filer. On the contrary, if the VMs are vmotioned and run on the same ESX server the performance does not increase in the same way and they always perform below a certain I/O limit.

Is there any way to change those queue length values to be able to make some more performance tests?

Your tips are welcome,

Albert

[christianZ]

In the past we discussed it here,

e.g. see the thread:

http://www.vmware.com/community/message.jspa?messageID=509652#509652

So it seems you can't make it faster.

[Paul_Lalonde]

I've been peeking into the iscsi_mod.o source code for a while, so I'll take a closer look to see if anything can be done with the Q-lengths.

Paul

[herraa1]

Hi,

Any findings yet?

It is frustrating to see how the software iSCSI initiator performs in ESX 3.0.1.

This is a sample esxtop disk output, focused at the LUN level:

9:15:44am up 3 days, 20:55, 215 worlds; CPU load average: 0.71, 0.76, 0.70

ADAPTR CID TID LID WID NCHNS NTGTS NLUNS NVMS AQLEN LQLEN[/b] WQLEN ACTV[/b] QUED %USD[/b] LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/

vmhba0 - - - - 1 2 2 1 976 0 0 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 1 - 1 1 1 1 64 12 0 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 2 - 1 1 1 1 64 12 0 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 10 - 1 1 1 22 64 12[/b] 0 12[/b] 245 100[/b] 21.42 178.62 103.67 74.95 0.31 0.1

vmhba40 0 0 12 - 1 1 1 2 64 12 0 0 0 0 0.00 0.45 0.00 0.45 0.00 0.0

vmhba40 0 1 - - 1 1 1 1 64 0 0 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

You can see there how LQLEN=12 is limiting performance.

This is another esxtop disk output focused at the LUN/VM level:

9:22:03am up 3 days, 21:02, 215 worlds; CPU load average: 0.70, 0.70, 0.72

ADAPTR CID TID LID WID NCHNS NTGTS NLUNS NVMS AQLEN LQLEN WQLEN ACTV QUED %USD LOAD CMDS/s[/b] READS/s WRITES/s MBREAD/s MBWRTN/

vmhba0 - - - - 1 2 2 1 976 0 0 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 1 - 1 1 1 1 64 12 0 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 2 - 1 1 1 1 64 12 0 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 10 1024 0 0 0 1 64 12 32 0 2 0 0.06 6.83 6.38 0.46 0.40 0.0

vmhba40 0 0 10 1079 0 0 0 1 64 12 32 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 10 1084 0 0 0 1 64 12 32 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 10 1096 0 0 0 1 64 12 32 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 10 1128 0 0 0 1 64 12 32 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 10 1145 0 0 0 1 64 12 32 1 0 3 0.03 26.87[/b] 16.40 10.48 0.03 0.0

vmhba40 0 0 10 1153 0 0 0 1 64 12 32 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 10 1176 0 0 0 1 64 12 32 0 0 0 0.00 1.82 0.00 1.82 0.00 0.0

vmhba40 0 0 10 1184 0 0 0 1 64 12 32 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 10 1192 0 0 0 1 64 12 32 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 10 1200 0 0 0 1 64 12 32 0 0 0 0.00 1.37 0.00 1.37 0.00 0.0

vmhba40 0 0 10 1208 0 0 0 1 64 12 32 2 61 6 1.97 82.90[/b] 52.84 30.06 0.10 0.0

vmhba40 0 0 10 1224 0 0 0 1 64 12 32 2 61 6 1.97 84.72[/b] 61.95 22.77 0.12 0.0

vmhba40 0 0 10 1232 0 0 0 1 64 12 32 4 59 12 1.97 85.18[/b] 55.57 29.61 0.11 0.0

vmhba40 0 0 10 1240 0 0 0 1 64 12 32 3 60 9 1.97 82.44[/b] 56.94 25.51 0.11 0.0

vmhba40 0 0 10 1248 0 0 0 1 64 12 32 0 0 0 0.00 3.64 0.00 3.64 0.00 0.0

vmhba40 0 0 10 1256 0 0 0 1 64 12 32 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 10 1264 0 0 0 1 64 12 32 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 10 1280 0 0 0 1 64 12 32 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 10 1288 0 0 0 1 64 12 32 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 10 1296 0 0 0 1 64 12 32 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 10 1320 0 0 0 1 64 12 32 0 0 0 0.00 0.91 0.00 0.91 0.00 0.0

vmhba40 0 0 12 - 1 1 1 2 64 12 0 0 0 0 0.00 9.57 0.00 9.57 0.00 0.5

vmhba40 0 1 - - 1 1 1 1 64 0 0 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

You can see how VMs running IOMETER (500000 sectors, 64 IOs per target, default access specification) are only doing tenths of IOs per seconds again because of the LQLEN=12 limitation.

Using NFS instead of iSCSI leads to orders of thousands of IOs per second.

We have found that it is faster to install the Microsoft software iSCSI initiator on every VM than relying on the ESX 3.0.1 software iSCSI initiator. Yes, this has a side effect (higher host CPU usage), but having such a poor performance seems not an option for anything doing moderate IO.

I hope that VMware really works on improving this absolutely serious inconvenience.

Albert

[Paul_Lalonde]

Yes, the limits defined in the iscsi-limits.h file seem awfully low.

Can you try this latest compile (includes the SCSI RESERVE patch) that increases the queue length and depth limits?

http://home.cogeco.ca/~plalonde2/iscsi_q.zip

Let me know if you see a performance increase.

Regards,

Paul

[Paul_Lalonde]

Just in case you haven't been following the iscsi patch threads and such...

\- Rename the existing /usr/lib/vmware/vmkmod/iscsi_mod.o to iscsi_mod.o.BAK.

\- Copy this file to /usr/lib/vmware/vmkmod/iscsi_mod.o

\- chmod 444 iscsi_mod.o

\- Reboot and test iSCSI.

Paul

[herraa1]

Hi Paul,

The change \_made_ a difference. So we are on the right track.

Now LQLEN=32 allows for more IOs per second.

This test talks by itself:

Test Scenario:

4 VMs (w2k3 sp1) on the same vmfs LUN (hosted on a NetApp FAS3020 28x250g disks, 2 AT-FCX shelves) running iometer concurrently (each iometer instance doing 500000 records, 12 outstanding IOs, default access specification)

Results for default VMware settings (LQLEN=12, Disk.SchedNumReqOutstanding=16)

11:30:20am up 17:39, 82 worlds; CPU load average: 0.06, 0.16, 0.21

ADAPTR CID TID LID WID NCHNS NTGTS NLUNS NVMS AQLEN LQLEN[/b] WQLEN ACTV[/b] QUED %USD LOAD CMDS/s[/b] READS/s WRITES/s MBREAD/s MBWRTN/

vmhba0 - - - - 1 2 2 1 976 0 0 0 0 0 0.00 4.37 0.00 4.37 0.00 0.0

vmhba40 0 0 1 - 1 1 1 6 64 12[/b] 0 12[/b] 38 100 4.17 723.69[/b] 483.91 239.77 1.02 0.4

vmhba40 0 0 2 - 1 1 1 3 64 12 0 0 0 0 0.00 0.49 0.00 0.49 0.00 0.0

Results using Paul's iSCSI patch (which changes LQLEN to 32) and Disk.SchedNumReqOutstanding=32

11:58:44am up 18 min, 82 worlds; CPU load average: 0.44, 0.27, 0.16

ADAPTR CID TID LID WID NCHNS NTGTS NLUNS NVMS AQLEN LQLEN[/b] WQLEN ACTV[/b] QUED %USD LOAD CMDS/s[/b] READS/s WRITES/s MBREAD/s MBWRTN/

vmhba0 - - - - 1 2 2 1 976 0 0 0 0 0 0.00 0.00 0.00 0.00 0.00 0.0

vmhba40 0 0 1 - 1 1 1 6 8192 32[/b] 0 31[/b] 14 96 1.41 13929.54[/b] 9291.50 4638.04 18.05 9.0

vmhba40 0 0 2 - 1 1 1 3 8192 32 0 0 0 0 0.00 12.05 0.00 12.05 0.00 0.0

I think the AQLEN (8192) in your patch is excessively large. And the LQLEN could be also tweaked a bit. But the whole stuff looks pretty good.

Having these queue-related parameters configurable at the module command line would be nice to get anyone's desired configuration.

Thanks a lot!

Cheers,

Albert

PS: VMware? Are you listening?

[Paul_Lalonde]

Hi Albert,

Actually, I pulled the value of 8192 of AQLEN from the iscsi-limits.h file for the Linux 2.6 iSCSI initiator, but yes, I do think it it set a little high.

I would be happy to insert some code into the iscsi_mod.o vmkernel module to allow for configurable parameters. I could either have the values pulled from an /etc/configfile or specified on the command line, ie:

vmkload_mod iscsi_mod.o aqlen=256 lqlen=32

etc.

Let me get to work on this, I'll let you know when it's done!

Regards,

Paul

[yossih]

Hi Paul,

I'm wondering is you had a chance to implement the option to modify the AQLEN or the LQLEN using a command line. Also, can you please let me know if your iSCSI patch is included in ESX 3.01 build 32039.

Thanks!

Yossi

[Paul_Lalonde]

I meant to get this done months ago... thanks for the reminder. I will put some time into it in the next couple of weeks.

Paul

[Oscar_Olsson]

We're running ESX 3.0.1, and we see the exact same issue. As I understand it, the patch mentioned in this thread is a homebrew modification, so thus it wouldn't be suitable for a production environment from a VMware support perspective? Since this thread is somewhat dated by now, I wonder if there have been any other developments with this issue? For instance, has a patch been released that sets more sane default values, or allows the user to modify them in some way? Also, out of curiosity, does anyone have any insight on why ESX server doesn't use the openiscsi driver these days? The initiator used starts to feel old and underdeveloped..

//Oscar