iSCSI Software Adapter Round Robin limited to 1gb of throughput with 4x 1gb Nics

[VSprague]

I have 4 Intel nics that I'm using for iSCSI traffic on an ESXi host. The NAS also has 4 nics. I have a Standard vSwitch setup with 4 separate VMKernel Ports, each VMKernel port has 1 active NIC. I added each of the VMKernel ports into the iSCSI Software Adapter. I have the NAS configured with 4 individual IP addresses so I can properly do MPIO. When I scan the iSCSI target it comes up with 16 paths. I also have jumbo frames enable on everything. I have also experimented with changing the IOPS count in the Round Robin policy, I set it down to 1 which had some negative side effects so I bumped it up to 100 which seems to be better for the guest VM's but none of it resolved the core issue. Whenever I put an iSCSI load though the best throughput I can get is about 120MBps, it uses about 25-30KBps per nic max. I know it's not a NAS issue because I setup a Hyper-V host with the same 4-ports for iSCSI with MPIO and was able to get almost 400MBps of throughput and 6000IOPS of write performance. I tired setting up each of the different iSCSI ports on different subnets, like 10.10.1.0 and 10.10.2.0 etc and that did change my number of path's down to 4 but I was still seeing only about 1gb of throughput. If I set the iSCSI policy to Fixed instead of RR it will fully utilize 1gb nic and give me the same performance as Round Robin does with 4 nics. I also tried running multiple migrations at once to see if it would use more than the 1gb of throughput and it doesn't, the two jobs just slowdown. I would appreciate it if someone could explain to me what is going on here so I don't need to ditch ESXi and resort back to using Hyper-V.

[VSprague]

So the best I can figure is that Round Robin is only using 1 path at any given time. So no matter what I tweak or change Round Robin is just not going to give me more than 1gb of throughput. I would think that I would be able to setup multiple fixed paths that can all run at the same time. But I have not found a way of doing that. When I set the path policy to fixed I can only set 1 NIC at a time to preferred. So at this point I guess I'm stuck unless someone can tell me how to enable multiple fixed paths or get Round Robin to use all the paths at the same time instead of one at a time.

[a_p_]

When I scan the iSCSI target it comes up with 16 paths.

I don't know whether the number of paths causes issues. However, ESXi only supports up to 8 paths to a LUN for iSCSI (see https://www.vmware.com/pdf/vsphere6/r60/vsphere-60-configuration-maximums.pdf). Maybe you can run a test with a maximum of 8 paths to see whether this helps.

André

[VSprague]

I tried reducing the number of paths by putting each HOST/NAS NIC pair on a different subnet. So NIC1 was on 10.10.6, NIC2 was on 10.10.7 etc. This reduced the number of paths down to 4 but my performance was still limited to about 1gb overall. As a test I left the IOPS count for Round Robin high and I can watch the network monitor on the NAS and in ESXi and it shows one NIC is used, then when it hits the count it switches to the next NIC, so on and so forth. It doesn't use all the NICs at once.

[hussainbte]

One thing I know for sure is that you cannot get RR to use multiple paths at any given point of time. It will switch to the next active path after the No of IO Count is done.

Also, If you could tell us what exactly are the issues you are facing while setting the No of IO in RR to 1. that is a VMware recommendation unless specified by array vendor.

please share the Array make and model and failover mode the array is running in (ALUA OR Active Active)

[VSprague]

I'm just not seeing satisfactory throughput to the NAS. This is a lab environment, I'm running a Synology DS1815+ with 8x 5TB WD Red Pro in RAID 10. I have all the 4 NICs on the NAS Active and individually IP'd. My issue is that I was expecting 4x 1gb NICs to equal 4gb of throughput when using MPIO but I'm only seeing 1gb of throughput. I set the IOPS count down to 1 and it still only leaves with me 1gb of throughput.