Problems with iSCSI Multipathing

ChristianHanzal · ‎06-14-2010

Hi Forum,

on Friday we updated our ESXi Server with more RAM and used the downtime for some reconfiguration.

First of all, we updated to ESXi U2 (261974) and then reconfigured the iSCSI Connection.

The Goal:

We wanted to achieve an Multipath Connection between our iSCSI Storage and the VMWare Server for redundancy and higher bandwidth over two GBIt/s Connections.

The Problem:

After setting all up (as described below) i recognized no Traffic on one of the VMWare NICs.

On the Storage Side, the Traffic is shared between the NICs, on the VMWare Side only one NIC seems to be active.

The Rig:

Dell R900 with Customized ESXi 4.0 Updated to U2 (261974)

Dell PowerConnect 5424

Infortrend S12E-G1133-2 iSCSI Storage

Before the Setup begun, we read several Documentations, Forums and Blogs.

Those were IMHO the most relevant:

http://www.vmware.com/pdf/vsphere4/r40_u1/vsp_40_u1_iscsi_san_cfg.pdf

http://www.infortrend.com/doc/appNote/APP_iSCSI_bottom_up_v1.1.pdf

http://communities.vmware.com/thread/211361

The Setup:

On the Switch we created a dedicated VLAN.

ESXi and Infortrend are connected to this VLAN with 2 NICs, the corresponding Ports are untagged.

On the Storage-Side, we created a Channel Group with CH0 and CH1 connected and assigned the relevant IP Configuration

This was done according to the PDF linked above.

On the VMWare-Side we created a VSwitch and bound vmnic2 & vmnic3 to it.

Then we created to VMkernel Ports and overwrote the failover order to assign one physical NIC to one VMKernel Port.

See esxi_network.png

Then we bound the VMkernel Ports to the Software initiator to activate Multipathing using

esxcli swiscsi nic add -n <port_name> -d <vmhba>

for each VMKernel Port

The result is

-

C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>esxcli --server xp-esxi1 swiscsi nic list -d vmhba34

vmk1

pNic name: vmnic2

ipv4 address: 10.128.3.2

ipv4 net mask: 255.255.255.0

ipv6 addresses:

mac address: 00:22:19:c9:f7:cf

mtu: 1500

toe: false

tso: true

tcp checksum: false

vlan: true

link connected: true

ethernet speed: 1000

packets received: 302467793

packets sent: 381745364

NIC driver: bnx2

driver version: 1.6.9

firmware version: 4.6.0

vmk2

pNic name: vmnic3

ipv4 address: 10.128.3.3

ipv4 net mask: 255.255.255.0

ipv6 addresses:

mac address: 00:22:19:c9:f7:d1

mtu: 1500

toe: false

tso: true

tcp checksum: false

vlan: true

link connected: true

ethernet speed: 1000

packets received: 171907

packets sent: 187

NIC driver: bnx2

driver version: 1.6.9

firmware version: 4.6.0

-

As you can se here, the amount of Packages received and sent is very low on vmk2, as described above.

We used one of the IPs from the Storage for Dynamic Discovery and got both IPs back in static Discovery - as expected, the Storage reports any Path that is available back to VMWare.

See esxi_vmhba34_discovery.png

All LUNs are available and accessible.

When i looked to my physical switch while creating I/O load on one of the Machines that reside on the iSCSI Storage, i recognized, that only 3 Ports are involved.

See switch_loadleveling_high_traffic.png

Ports 13 & 24 are connected to ESXi, 17 & 18 are connected to the Storage.

This is the Device List:

-

C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>esxcli --server xp-esxi1 nmp device list

naa.600d02310007373400000000353c24b2

Device Display Name: IFT iSCSI Disk (naa.600d02310007373400000000353c24b2)

Storage Array Type: VMW_SATP_DEFAULT_AA

Storage Array Type Device Config:

Path Selection Policy: VMW_PSP_RR

Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=1: NumIOsPending=0,numBytesPending=0}

Working Paths: vmhba34:C1:T0:L0, vmhba34:C3:T0:L0

naa.600d02310007373400000002353c24b2

Device Display Name: IFT iSCSI Disk (naa.600d02310007373400000002353c24b2)

Storage Array Type: VMW_SATP_DEFAULT_AA

Storage Array Type Device Config:

Path Selection Policy: VMW_PSP_RR

Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=1: NumIOsPending=0,numBytesPending=0}

Working Paths: vmhba34:C1:T0:L2, vmhba34:C3:T0:L2

mpx.vmhba32:C0:T0:L0

Device Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)

Storage Array Type: VMW_SATP_LOCAL

Storage Array Type Device Config:

Path Selection Policy: VMW_PSP_FIXED

Path Selection Policy Device Config: {preferred=vmhba32:C0:T0:L0;current=vmhba32:C0:T0:L0}

Working Paths: vmhba32:C0:T0:L0

mpx.vmhba0:C0:T0:L0

Device Display Name: Local HL-DT-ST CD-ROM (mpx.vmhba0:C0:T0:L0)

Storage Array Type: VMW_SATP_LOCAL

Storage Array Type Device Config:

Path Selection Policy: VMW_PSP_FIXED

Path Selection Policy Device Config: {preferred=vmhba0:C0:T0:L0;current=vmhba0:C0:T0:L0}

Working Paths: vmhba0:C0:T0:L0

naa.600d02310007373400000001353c24b2

Device Display Name: IFT iSCSI Disk (naa.600d02310007373400000001353c24b2)

Storage Array Type: VMW_SATP_DEFAULT_AA

Storage Array Type Device Config:

Path Selection Policy: VMW_PSP_RR

Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=1: NumIOsPending=0,numBytesPending=0}

Working Paths: vmhba34:C1:T0:L1, vmhba34:C3:T0:L1

naa.60024e805cd8480011b4054e04cfbe79

Device Display Name: Local DELL Disk (naa.60024e805cd8480011b4054e04cfbe79)

Storage Array Type: VMW_SATP_LOCAL

Storage Array Type Device Config:

Path Selection Policy: VMW_PSP_RR

Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=0: NumIOsPending=0,numBytesPending=0}

Working Paths: vmhba1:C0:T0:L0

-

Can you give me a hint, why my other VMWare NIC is not involved in the Communication?

greetings from Germany

Christian Hanzal

ChristianHanzal · ‎06-16-2010

I'll move this Thread up again.

It is not ueber-urgent but solving this Problem is relevant for me.

Is there anything i can try, any hint, any more Information to provide?

I would really appreciate any help.

Thank you in advance.

Best wishes

Christian

ChristianHanzal · ‎06-16-2010

I have to appologize: There was a malconfiguration on our Switch.

Problem solved - Thanks anyway.

Christian