6 Replies Latest reply on Mar 10, 2011 1:52 PM by KOByte

    vSphere/ESXi 4.1 Upgrade and iSCSI Multipathing - Paths missing?

    KOByte Lurker

      I am in the process of testing 4.1 upgrades in our 3 host cluster.  We are set up connected via software initiator to an MSA2324i.  The MSA has dual controllers, each with 2 host ports, for 4 paths.  There are 4 LUNs, including - for some reason - a LUN assigned to the enclosure. We also have one LUN on a windows server target for backups and whatnot.  When configuring the hosts previoiusly (on 4.0u1/u2), this configuration showed 2 targets and 17 paths, which kind of makes sense, with 4 paths to each of 4 LUNs and 1 to the windows target.  The host iSCSI adapters are currently configured with Send Targets, which properly reflects all 5 paths in static targets.

       

       

      After upgrading one host to 4.1, Send Targets appears to work, with all paths showing in static targets, but the result is different - I see 3 targets and 9 paths, which indicates that 2 of the host ports on the MSA are not being reflected correctly:

       

       

      At one point, though my experimenting, I added all 4 IPs to Send Targets and it configured correctly.  I adjust them back and it was retained, but later reverted.  The other two hosts, still on 4.0u2, are showing all paths correctly still.

       

      These hosts are all BL460c blades, each with 2 pnics attached to the iSCSI adapter as laid out in a few best practices documents I used way back when:

       

       

      Why is this happening, and how can I rectify it?

       

      Thanks!

        • 1. Re: vSphere/ESXi 4.1 Upgrade and iSCSI Multipathing - Paths missing?
          GeorgeMahesh Novice

          Could you please remove all of the IPs from dynamic discovery and add all of those 4IPs in the static target tab and rescan. Please let us know if it made a difference.

          • 2. Re: vSphere/ESXi 4.1 Upgrade and iSCSI Multipathing - Paths missing?
            KOByte Lurker

            If I remove the ips from dynamic discovery and add them to static, it shows all 161 paths.  Similarly, if I add at least 3 of the 4 to dynamic discovery, it shows all 161 paths.  In both of these scenarios it shows 5 targets, which is still different than the other pre-4.1 hosts.  I'd be really interested to know:

             

            1.  When adding a single IP to dynamic discovery, why does SendTargets not get all paths?

            2. Why does 4.1 show 5 targets when seeing all paths, less when not, when pre-4.1 sees only 2 targets?

             

            Thanks!

             

             

            • 3. Re: vSphere/ESXi 4.1 Upgrade and iSCSI Multipathing - Paths missing?
              GeorgeMahesh Novice

              Looks like the enclosures are also getting counted as targets in 4.1 and that is why we are seeing 5 targets. I have to confirm the same again.

               

              Regarding the number of paths. I think we are seeing 16+1 paths only when we are adding more than 2 target IPs. This is expected because the 1st and 2nd IP that we give are both there on the same controller and discovers the luns only through those ports and may be the 3rd IP and 4th IP are on the second controller and they are not discovering the targets. So, when we give 3 or 4 target IPs for discovery both the controllers are asked to show the luns and they show the luns from all the ports on both the controllers.

               

              For more investigations, It will be nice if you could provide us the esxcfg-mpath output and let us know if you have configured port binding on those 2 VMkernel interfaces.

              • 4. Re: vSphere/ESXi 4.1 Upgrade and iSCSI Multipathing - Paths missing?
                KOByte Lurker

                I'm not positive which configuration part you are asking about port binding for, but they were set up as per the iSCSI SAN Configuration guide for active multipathing, with the nics bound to vmhba33 and the teaming settings adjusted so there are no standby nics for each of them.  Also, this config hasn't changed on my end with the upgrade, and the other 2 hosts that are still on 4.0u2 don't have this symptom.

                 

                I've pasted the esxcfg-mpath -l output below.  This is with the configuration of 41 static target ips and 161 paths.

                 

                 

                C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>esxcfg-mpath.pl<http://esxcfg-mpath.pl> --server es

                x1.krechojard.local --username root --password KreCH0j@RD -l

                iqn.1998-01.com.vmware:esx1-78d9ed09-00023d000008,iqn.1986-03.com.hp:storage.msa

                2324i.0944da4f56,t,3-naa.600c0ff000da9c130000000000000000

                   Runtime Name: vmhba33:C7:T1:L0

                   Device: naa.600c0ff000da9c130000000000000000

                   Device Display Name: HP iSCSI Enclosure Svc Dev (naa.600c0ff000da9c1300000000

                00000000)

                   Adapter: vmhba33 Channel: 7 Target: 1 LUN: 0

                   Adapter Identifier: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Identifier: 00023d000008,iqn.1986-03.com.hp:storage.msa2324i.0944da4f5

                6,t,3

                   Plugin: NMP

                   State: active

                   Transport: iscsi

                   Adapter Transport Details: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Transport Details: IQN=iqn.1986-03.com.hp:storage.msa2324i.0944da4f56

                Alias= Session=00023d000008 PortalTag=3

                 

                iqn.1998-01.com.vmware:esx1-78d9ed09-00023d000008,iqn.1986-03.com.hp:storage.msa

                2324i.0944da4f56,t,3-naa.600c0ff000da9c134fe6bc4b01000000

                   Runtime Name: vmhba33:C7:T1:L5

                   Device: naa.600c0ff000da9c134fe6bc4b01000000

                   Device Display Name: HP iSCSI Disk (naa.600c0ff000da9c134fe6bc4b01000000)

                   Adapter: vmhba33 Channel: 7 Target: 1 LUN: 5

                   Adapter Identifier: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Identifier: 00023d000008,iqn.1986-03.com.hp:storage.msa2324i.0944da4f5

                6,t,3

                   Plugin: NMP

                   State: active

                   Transport: iscsi

                   Adapter Transport Details: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Transport Details: IQN=iqn.1986-03.com.hp:storage.msa2324i.0944da4f56

                Alias= Session=00023d000008 PortalTag=3

                 

                iqn.1998-01.com.vmware:esx1-78d9ed09-00023d000006,iqn.1986-03.com.hp:storage.msa

                2324i.0944da4f56,t,4-naa.600c0ff000da9e320000000000000000

                   Runtime Name: vmhba33:C5:T1:L0

                   Device: naa.600c0ff000da9e320000000000000000

                   Device Display Name: HP iSCSI Enclosure Svc Dev (naa.600c0ff000da9e3200000000

                00000000)

                   Adapter: vmhba33 Channel: 5 Target: 1 LUN: 0

                   Adapter Identifier: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Identifier: 00023d000006,iqn.1986-03.com.hp:storage.msa2324i.0944da4f5

                6,t,4

                   Plugin: NMP

                   State: active

                   Transport: iscsi

                   Adapter Transport Details: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Transport Details: IQN=iqn.1986-03.com.hp:storage.msa2324i.0944da4f56

                Alias= Session=00023d000006 PortalTag=4

                 

                iqn.1998-01.com.vmware:esx1-78d9ed09-00023d000006,iqn.1986-03.com.hp:storage.msa

                2324i.0944da4f56,t,4-naa.600c0ff000da9c134fe6bc4b01000000

                   Runtime Name: vmhba33:C5:T1:L5

                   Device: naa.600c0ff000da9c134fe6bc4b01000000

                   Device Display Name: HP iSCSI Disk (naa.600c0ff000da9c134fe6bc4b01000000)

                   Adapter: vmhba33 Channel: 5 Target: 1 LUN: 5

                   Adapter Identifier: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Identifier: 00023d000006,iqn.1986-03.com.hp:storage.msa2324i.0944da4f5

                6,t,4

                   Plugin: NMP

                   State: active

                   Transport: iscsi

                   Adapter Transport Details: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Transport Details: IQN=iqn.1986-03.com.hp:storage.msa2324i.0944da4f56

                Alias= Session=00023d000006 PortalTag=4

                 

                iqn.1998-01.com.vmware:esx1-78d9ed09-00023d000001,iqn.1986-03.com.hp:storage.msa

                2324i.0944da4f56,t,2-naa.600c0ff000da9e3277e5bc4b01000000

                   Runtime Name: vmhba33:C0:T1:L30

                   Device: naa.600c0ff000da9e3277e5bc4b01000000

                   Device Display Name: HP iSCSI Disk (naa.600c0ff000da9e3277e5bc4b01000000)

                   Adapter: vmhba33 Channel: 0 Target: 1 LUN: 30

                   Adapter Identifier: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Identifier: 00023d000001,iqn.1986-03.com.hp:storage.msa2324i.0944da4f5

                6,t,2

                   Plugin: NMP

                   State: active

                   Transport: iscsi

                   Adapter Transport Details: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Transport Details: IQN=iqn.1986-03.com.hp:storage.msa2324i.0944da4f56

                Alias= Session=00023d000001 PortalTag=2

                 

                iqn.1998-01.com.vmware:esx1-78d9ed09-00023d000003,iqn.1986-03.com.hp:storage.msa

                2324i.0944da4f56,t,1-naa.600c0ff000da9e3277e5bc4b01000000

                   Runtime Name: vmhba33:C2:T1:L30

                   Device: naa.600c0ff000da9e3277e5bc4b01000000

                   Device Display Name: HP iSCSI Disk (naa.600c0ff000da9e3277e5bc4b01000000)

                   Adapter: vmhba33 Channel: 2 Target: 1 LUN: 30

                   Adapter Identifier: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Identifier: 00023d000003,iqn.1986-03.com.hp:storage.msa2324i.0944da4f5

                6,t,1

                   Plugin: NMP

                   State: active

                   Transport: iscsi

                   Adapter Transport Details: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Transport Details: IQN=iqn.1986-03.com.hp:storage.msa2324i.0944da4f56

                Alias= Session=00023d000003 PortalTag=1

                 

                iqn.1998-01.com.vmware:esx1-78d9ed09-00023d000006,iqn.1986-03.com.hp:storage.msa

                2324i.0944da4f56,t,4-naa.600c0ff000da9e3277e5bc4b01000000

                   Runtime Name: vmhba33:C5:T1:L30

                   Device: naa.600c0ff000da9e3277e5bc4b01000000

                   Device Display Name: HP iSCSI Disk (naa.600c0ff000da9e3277e5bc4b01000000)

                   Adapter: vmhba33 Channel: 5 Target: 1 LUN: 30

                   Adapter Identifier: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Identifier: 00023d000006,iqn.1986-03.com.hp:storage.msa2324i.0944da4f5

                6,t,4

                   Plugin: NMP

                   State: active

                   Transport: iscsi

                   Adapter Transport Details: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Transport Details: IQN=iqn.1986-03.com.hp:storage.msa2324i.0944da4f56

                Alias= Session=00023d000006 PortalTag=4

                 

                iqn.1998-01.com.vmware:esx1-78d9ed09-00023d000008,iqn.1986-03.com.hp:storage.msa

                2324i.0944da4f56,t,3-naa.600c0ff000da9e3277e5bc4b01000000

                   Runtime Name: vmhba33:C7:T1:L30

                   Device: naa.600c0ff000da9e3277e5bc4b01000000

                   Device Display Name: HP iSCSI Disk (naa.600c0ff000da9e3277e5bc4b01000000)

                   Adapter: vmhba33 Channel: 7 Target: 1 LUN: 30

                   Adapter Identifier: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Identifier: 00023d000008,iqn.1986-03.com.hp:storage.msa2324i.0944da4f5

                6,t,3

                   Plugin: NMP

                   State: active

                   Transport: iscsi

                   Adapter Transport Details: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Transport Details: IQN=iqn.1986-03.com.hp:storage.msa2324i.0944da4f56

                Alias= Session=00023d000008 PortalTag=3

                 

                iqn.1998-01.com.vmware:esx1-78d9ed09-00023d000003,iqn.1986-03.com.hp:storage.msa

                2324i.0944da4f56,t,1-naa.600c0ff000da9c130000000000000000

                   Runtime Name: vmhba33:C2:T1:L0

                   Device: naa.600c0ff000da9c130000000000000000

                   Device Display Name: HP iSCSI Enclosure Svc Dev (naa.600c0ff000da9c1300000000

                00000000)

                   Adapter: vmhba33 Channel: 2 Target: 1 LUN: 0

                   Adapter Identifier: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Identifier: 00023d000003,iqn.1986-03.com.hp:storage.msa2324i.0944da4f5

                6,t,1

                   Plugin: NMP

                   State: active

                   Transport: iscsi

                   Adapter Transport Details: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Transport Details: IQN=iqn.1986-03.com.hp:storage.msa2324i.0944da4f56

                Alias= Session=00023d000003 PortalTag=1

                 

                iqn.1998-01.com.vmware:esx1-78d9ed09-00023d000003,iqn.1986-03.com.hp:storage.msa

                2324i.0944da4f56,t,1-naa.600c0ff000da9c134fe6bc4b01000000

                   Runtime Name: vmhba33:C2:T1:L5

                   Device: naa.600c0ff000da9c134fe6bc4b01000000

                   Device Display Name: HP iSCSI Disk (naa.600c0ff000da9c134fe6bc4b01000000)

                   Adapter: vmhba33 Channel: 2 Target: 1 LUN: 5

                   Adapter Identifier: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Identifier: 00023d000003,iqn.1986-03.com.hp:storage.msa2324i.0944da4f5

                6,t,1

                   Plugin: NMP

                   State: active

                   Transport: iscsi

                   Adapter Transport Details: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Transport Details: IQN=iqn.1986-03.com.hp:storage.msa2324i.0944da4f56

                Alias= Session=00023d000003 PortalTag=1

                 

                iqn.1998-01.com.vmware:esx1-78d9ed09-00023d000001,iqn.1986-03.com.hp:storage.msa

                2324i.0944da4f56,t,2-naa.600c0ff000da9e320000000000000000

                   Runtime Name: vmhba33:C0:T1:L0

                   Device: naa.600c0ff000da9e320000000000000000

                   Device Display Name: HP iSCSI Enclosure Svc Dev (naa.600c0ff000da9e3200000000

                00000000)

                   Adapter: vmhba33 Channel: 0 Target: 1 LUN: 0

                   Adapter Identifier: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Identifier: 00023d000001,iqn.1986-03.com.hp:storage.msa2324i.0944da4f5

                6,t,2

                   Plugin: NMP

                   State: active

                   Transport: iscsi

                   Adapter Transport Details: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Transport Details: IQN=iqn.1986-03.com.hp:storage.msa2324i.0944da4f56

                Alias= Session=00023d000001 PortalTag=2

                 

                iqn.1998-01.com.vmware:esx1-78d9ed09-00023d000005,iqn.1991-05.com.microsoft:mira

                -vsphere-target,t,1-naa.60003ffc4adfae8984a4f6d3b32ebc2a

                   Runtime Name: vmhba33:C1:T0:L50

                   Device: naa.60003ffc4adfae8984a4f6d3b32ebc2a

                   Device Display Name: MSFT iSCSI Disk (naa.60003ffc4adfae8984a4f6d3b32ebc2a)

                   Adapter: vmhba33 Channel: 1 Target: 0 LUN: 50

                   Adapter Identifier: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Identifier: 00023d000005,iqn.1991-05.com.microsoft:mira-vsphere-target

                ,t,1

                   Plugin: NMP

                   State: active

                   Transport: iscsi

                   Adapter Transport Details: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Transport Details: IQN=iqn.1991-05.com.microsoft:mira-vsphere-target A

                lias= Session=00023d000005 PortalTag=1

                 

                iqn.1998-01.com.vmware:esx1-78d9ed09-00023d000001,iqn.1986-03.com.hp:storage.msa

                2324i.0944da4f56,t,2-naa.600c0ff000da9c134fe6bc4b01000000

                   Runtime Name: vmhba33:C0:T1:L5

                   Device: naa.600c0ff000da9c134fe6bc4b01000000

                   Device Display Name: HP iSCSI Disk (naa.600c0ff000da9c134fe6bc4b01000000)

                   Adapter: vmhba33 Channel: 0 Target: 1 LUN: 5

                   Adapter Identifier: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Identifier: 00023d000001,iqn.1986-03.com.hp:storage.msa2324i.0944da4f5

                6,t,2

                   Plugin: NMP

                   State: active

                   Transport: iscsi

                   Adapter Transport Details: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Transport Details: IQN=iqn.1986-03.com.hp:storage.msa2324i.0944da4f56

                Alias= Session=00023d000001 PortalTag=2

                 

                iqn.1998-01.com.vmware:esx1-78d9ed09-00023d000001,iqn.1986-03.com.hp:storage.msa

                2324i.0944da4f56,t,2-naa.600c0ff000da9c13dd8cbb4b01000000

                   Runtime Name: vmhba33:C0:T1:L15

                   Device: naa.600c0ff000da9c13dd8cbb4b01000000

                   Device Display Name: HP iSCSI Disk (naa.600c0ff000da9c13dd8cbb4b01000000)

                   Adapter: vmhba33 Channel: 0 Target: 1 LUN: 15

                   Adapter Identifier: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Identifier: 00023d000001,iqn.1986-03.com.hp:storage.msa2324i.0944da4f5

                6,t,2

                   Plugin: NMP

                   State: active

                   Transport: iscsi

                   Adapter Transport Details: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Transport Details: IQN=iqn.1986-03.com.hp:storage.msa2324i.0944da4f56

                Alias= Session=00023d000001 PortalTag=2

                 

                iqn.1998-01.com.vmware:esx1-78d9ed09-00023d000003,iqn.1986-03.com.hp:storage.msa

                2324i.0944da4f56,t,1-naa.600c0ff000da9c13dd8cbb4b01000000

                   Runtime Name: vmhba33:C2:T1:L15

                   Device: naa.600c0ff000da9c13dd8cbb4b01000000

                   Device Display Name: HP iSCSI Disk (naa.600c0ff000da9c13dd8cbb4b01000000)

                   Adapter: vmhba33 Channel: 2 Target: 1 LUN: 15

                   Adapter Identifier: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Identifier: 00023d000003,iqn.1986-03.com.hp:storage.msa2324i.0944da4f5

                6,t,1

                   Plugin: NMP

                   State: active

                   Transport: iscsi

                   Adapter Transport Details: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Transport Details: IQN=iqn.1986-03.com.hp:storage.msa2324i.0944da4f56

                Alias= Session=00023d000003 PortalTag=1

                 

                iqn.1998-01.com.vmware:esx1-78d9ed09-00023d000006,iqn.1986-03.com.hp:storage.msa

                2324i.0944da4f56,t,4-naa.600c0ff000da9c13dd8cbb4b01000000

                   Runtime Name: vmhba33:C5:T1:L15

                   Device: naa.600c0ff000da9c13dd8cbb4b01000000

                   Device Display Name: HP iSCSI Disk (naa.600c0ff000da9c13dd8cbb4b01000000)

                   Adapter: vmhba33 Channel: 5 Target: 1 LUN: 15

                   Adapter Identifier: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Identifier: 00023d000006,iqn.1986-03.com.hp:storage.msa2324i.0944da4f5

                6,t,4

                   Plugin: NMP

                   State: active

                   Transport: iscsi

                   Adapter Transport Details: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Transport Details: IQN=iqn.1986-03.com.hp:storage.msa2324i.0944da4f56

                Alias= Session=00023d000006 PortalTag=4

                 

                iqn.1998-01.com.vmware:esx1-78d9ed09-00023d000008,iqn.1986-03.com.hp:storage.msa

                2324i.0944da4f56,t,3-naa.600c0ff000da9c13dd8cbb4b01000000

                   Runtime Name: vmhba33:C7:T1:L15

                   Device: naa.600c0ff000da9c13dd8cbb4b01000000

                   Device Display Name: HP iSCSI Disk (naa.600c0ff000da9c13dd8cbb4b01000000)

                   Adapter: vmhba33 Channel: 7 Target: 1 LUN: 15

                   Adapter Identifier: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Identifier: 00023d000008,iqn.1986-03.com.hp:storage.msa2324i.0944da4f5

                6,t,3

                   Plugin: NMP

                   State: active

                   Transport: iscsi

                   Adapter Transport Details: iqn.1998-01.com.vmware:esx1-78d9ed09

                   Target Transport Details: IQN=iqn.1986-03.com.hp:storage.msa2324i.0944da4f56

                Alias= Session=00023d000008 PortalTag=3

                 

                 

                C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>

                • 5. Re: vSphere/ESXi 4.1 Upgrade and iSCSI Multipathing - Paths missing?
                  DaIceMan Enthusiast

                  KOByte,

                    did you resolve the issue? We are having a similar problem with our dual controller MSA2312i and 4.1. We have 4 hosts with dual port QLogic QMH4062 Indipendent hardware adapters. The first port of the QMH (vmhba0) shows all 12 paths correctly (including LUN 0 on the hosts which do not boot from LUN0 which it sees as the MSA Enclosure so in reality it hows 16LUNs) but the second port (vmhba1) shows only 6. This goes for all 4 hosts. We tried adding dynamic discovery IPs but nothing changed.

                  Paths1.jpg

                  Paths2.jpg

                  • 6. Re: vSphere/ESXi 4.1 Upgrade and iSCSI Multipathing - Paths missing?
                    KOByte Lurker

                    Yes, our issue turned out to be two-fold:

                     

                    1.  All of our ports and adapters were on the same subnet, even though Controller A1/B1 were on Switch1 and A2/B2 on Switch 2.  I thought the switches were interconnected by default in the blade enclosure but they weren't.

                     

                    2.  Some kind of bug in vmkping makes it not see the paths if they cannot be pinged from vmk1 (in this case vmk1 couldn't ping A2/B2).

                     

                    To fix this I interconnected the switches, and eventually reconfigured the subnets for all of the san and vmk ports connected to switch 2, which is probably what should have been done initially.