iSCSI MPIO throughput limited to 1G

[gamania]

I have followed the VMware's "iSCSI SAN Configuration Guide" (filename vsp_40_iscsi_san_cfg.pdf) to set up vSphere 4 with software iSCSI MPIO. Round robin is also enabled on the storage to utilize both paths. I can see both paths are active from my Equallogic PS5000X, and are working with active traffic when doing sqoio test. However, the throughput in SQLIO test are maxed out to 1G. Where could the problem be? I imagine the MPIO will result in greater throughput just like what will happen in native Windows set up, but it doesn't seem so.

[Photubias]

Hello,

I have the same problem as Gamania, but it can't be a switch problem as I am using Peer-2-Peer cabling. I have 3 Peer-2-Peer cables and created 3 VMkernel Ports, with 3 vmnics, 3 vswitches and 3 IP-addresses.

This is what happens when I use fixed Paths.

You clearly see that it uses only 1 of the 1Gb links, wich is normal if you take a look at this screenshot:

When set to Round Robin, it turns to this (attached round-robin-esx.jpg) and you would expect that all 3 links will use each 1Gb so I get 3 times the througput.

However each Gigabit link uses 33% wich soms up to once again 1Gb (round-robin.jpg), is there some setting to push it past that boundary?

I do use IOMeter and when I use the EXACT same setup and add 3 VNics to my Virtual Machine, and I use Microsoft MPIO then it gets to 330MBps just fine...

Greetings and thanks in advance

[paithal]

Could you tell what type of storage are you using?.

Try setting the RR params like below:

esxcli nmp roundrobin setconfig --device <device UID> --iops 1 --type iops

Unfortunately there is no UI support for above. Let us know what do you get.

[Photubias]

Brilliant! This worked like a charm, I get 309.38MBps now. Check screenshot for Network Usage at the iSCSI Target side (Windows 2008 Storage Server with WinTarget iSCSI Target Software).

The storage array was an (experimental) 5 Disk RAID0 of 5 times Intel SSD X25-E (32GB). When I perform a DAS run I get over 600MBps throughput, so I'm pretty sure my bottleneck will be at the network and the ESX4 hypervisor overhead.

I'm researching Software iSCSI inside ESX4 over iSCSI in the Guest OS as they both support MPIO & jumbo frames right now. Howerver Jumbo Frames were possible in ESX3.5 (through cli) I did not found to get MPIO working in ESX3.5 update 4...

Can you tell me what was the problem with the first setup, what did the command you described do?

It looks like it sets the I/O Operation Limit to 1 and sets the Limit Type to Iops (as I can tell from the esxcli nmp roundrobin getconfig --device <ID> command).

Appearantly it switches over to the next path after only 1 I/O operation, but I do not see how this helps with the 1Gb limit?

Thanks in advance...

[paithal]

ESX RR policy does round-robin across multile active paths based on usage. There is only one active IO path at a given time and it sends down certain number of IO's before switching to another active path. The default path switching is based on "number of IOs" and default value for that is 1000. The above command changed that value to 1. Now, it worked great for your config but may yield different result for different storage. Switching paths for every other IO may totally trash the cache on the storage processor side.

[asp24]

This setting helped me to finally being able to get more than 1 Gbps storage "speed" in one VM. I have tested with IOMeter, and I get 215 MB/s in the 100% read test now (ca 7000 cmds/s). I see that the random test is affected in a negative way, but not by much (from ca 5000 cmds/s and 38-40 MB/s to 4300 cmds/s and 33 MB/s)

I will test this on one or two hosts and monitor the performance hit/gain.

I have also tried with 2, 5 and 10 io's between the "switching of paths", but the results was not changing (at least not much).

My settings (two different iscsi datastores, two vmkernels):

esxcli swiscsi nic add -n vmk0 -d vmhba35

esxcli swiscsi nic add -n vmk1 -d vmhba35

esxcli nmp device setpolicy --device naa.600d0230ffffffff03555077c8fc4700 --psp VMW_PSP_RR

esxcli nmp device setpolicy --device naa.600d0230ffffffff03555077c8fc4701 --psp VMW_PSP_RR

esxcli nmp roundrobin setconfig --device naa.600d0230ffffffff03555077c8fc4700 --iops 1 --type iops

esxcli nmp roundrobin setconfig --device naa.600d0230ffffffff03555077c8fc4701 --iops 1 --type iops

[asp24]

I have to do some more testing I think. During some benchmarking the VM lost its test disk. After a while all the host VMs hang. It seems like the host and the SAN lost connection. The SAN problably did not like the RR.

[glynnd1]

asp24, what firmware do you have on the array?

[asp24]

It's an Infortrend S16E-G1130 firmware 3.64J.09

[glynnd1]

Ah! I though we were still discussing the EqualLogic array.

[sakacc]

For what it's worth - will be presenting a LOT of data of various MPIO NMP RR configurations (LUN counts, VM configs, guest workloads, differing RR "number of IOs before path change" on EMC CLARiiON and EMC Celerra iSCSI configs - though should be generally applicable to many platforms).

Quick early observations is that there is a lot of difference between the default (which results in pretty unbalanced configs with low amounts of threads, and pretty balanced with lots of VMs, and lots of datastores/LUNs), and the "1" configuration (which is pretty balanced even with a single VM). We've also compared with PowerPath/VE which is a MPP (Multipathing Plugin).

It's at VMworld, session TA2467. In spite of the stuff above (which makes it sound like an EMC fest), there's a lot of stuff that applies to multivendor configs, and we cover FC/FCoE, iSCSI and NFS configs - just putting it out there - it will be best practices, not a commerical.

Not trying to promote the session, but too much stuff to do in a VMTN post....

Chad (aka virtualgeek).

[glynnd1]

Chad,

For those of us not attending VMworld will your presentation be freely available from your blog?

David

[ymmt2005]

Hi,

I have configured MPIO with Dell MD3000i and am having a problem.

In any of path selection policy, only one path is active and the others are shown as standby.

Are these standby paths will be activated? If so, when?

Any suggestions are appreciated. Thanks.

[ConstantinV]

Alternative ways will be activated if main way will break down.

StarWind Software Developer

[ymmt2005]

Thanks.

I got to know that MD3000i's dual controller is active-passive for each LUN,

therefore standby paths are there. It has two ethernet ports for each controller,

which I can use as active-active load balancing paths by setting path selection

policy to "Roundrobin".