ESX 4.1 and vCenter only allows use of 1 Fibre Channel LUN

[eforbus]

Hi,

I have 8 ESX 4.1 (build 348481) blades in a Cisco UCS 5108 chassis. Each blade has 2 x HBAs available. I also have a Fiber channel storage array that is configured to provide 2 1.8 TB LUNs (for now and more in the future).

When I go into vCenter and select any of the ESX servers under "Configuration" -> "Storage Adapters" -> "Rescan All..." each server detects the array (and multiple paths to it) but only shows a single LUN available (LUN 0) when 2 LUNs should actually be available (LUN 0 and LUN 1). See image below.

Now, the thing that is confusing me is that when I get on the ESX server command line via SSH and run "esxcfg-mpath -b" I see a list of all paths including paths to LUN 0 and LUN 1 (see output below). Any ideas on why ESX and vCenter won't allow me to get at LUN 1??

esxcfg-mpath -b :

naa.600508e0000000001e10a036bc3f5601 : LSILOGIC Serial Attached SCSI Disk (naa.600508e0000000001e10a036bc3f5601)
   vmhba0:C1:T0:L0 LUN:0 state:active sas Adapter: 5c84c75ed0910000  Target: 1563fbc36a0101e

mpx.vmhba32:C0:T0:L0 : Local USB CD-ROM (mpx.vmhba32:C0:T0:L0)
   vmhba32:C0:T0:L0 LUN:0 state:active Local HBA vmhba32 channel 0 target 0

eui.706f636c6162766f : SCST_FIO Fibre Channel Disk (eui.706f636c6162766f)
   vmhba2:C0:T1:L1 LUN:1 state:active fc Adapter: WWNN: 20:00:00:25:b5:00:00:ae WWPN: 20:00:00:25:b5:00:00:1e  Target: WWNN: 20:00:00:24:ff:32:10:f2 WWPN: 21:00:00:24:ff:32:10:f2
   vmhba2:C0:T1:L0 LUN:0 state:active fc Adapter: WWNN: 20:00:00:25:b5:00:00:ae WWPN: 20:00:00:25:b5:00:00:1e  Target: WWNN: 20:00:00:24:ff:32:10:f2 WWPN: 21:00:00:24:ff:32:10:f2
   vmhba1:C0:T0:L1 LUN:1 state:active fc Adapter: WWNN: 20:00:00:25:b5:00:00:ae WWPN: 20:00:00:25:b5:00:00:be  Target: WWNN: 20:00:00:24:ff:32:45:c2 WWPN: 21:00:00:24:ff:32:45:c2
   vmhba1:C0:T0:L0 LUN:0 state:active fc Adapter: WWNN: 20:00:00:25:b5:00:00:ae WWPN: 20:00:00:25:b5:00:00:be  Target: WWNN: 20:00:00:24:ff:32:45:c2 WWPN: 21:00:00:24:ff:32:45:c2
   vmhba1:C0:T1:L1 LUN:1 state:active fc Adapter: WWNN: 20:00:00:25:b5:00:00:ae WWPN: 20:00:00:25:b5:00:00:be  Target: WWNN: 20:00:00:24:ff:32:45:c3 WWPN: 21:00:00:24:ff:32:45:c3
   vmhba1:C0:T1:L0 LUN:0 state:active fc Adapter: WWNN: 20:00:00:25:b5:00:00:ae WWPN: 20:00:00:25:b5:00:00:be  Target: WWNN: 20:00:00:24:ff:32:45:c3 WWPN: 21:00:00:24:ff:32:45:c3
   vmhba2:C0:T0:L1 LUN:1 state:active fc Adapter: WWNN: 20:00:00:25:b5:00:00:ae WWPN: 20:00:00:25:b5:00:00:1e  Target: WWNN: 20:00:00:24:ff:32:10:f3 WWPN: 21:00:00:24:ff:32:10:f3
   vmhba2:C0:T0:L0 LUN:0 state:active fc Adapter: WWNN: 20:00:00:25:b5:00:00:ae WWPN: 20:00:00:25:b5:00:00:1e  Target: WWNN: 20:00:00:24:ff:32:10:f3 WWPN: 21:00:00:24:ff:32:10:f3

mpx.vmhba32:C0:T0:L1 : Local USB Direct-Access (mpx.vmhba32:C0:T0:L1)
   vmhba32:C0:T0:L1 LUN:1 state:active Local HBA vmhba32 channel 0 target 0

mpx.vmhba32:C0:T0:L2 : Local USB Direct-Access (mpx.vmhba32:C0:T0:L2)
   vmhba32:C0:T0:L2 LUN:2 state:active Local HBA vmhba32 channel 0 target 0

[mcowger · Accepted solution]

Its because the array is reporting both LUNs with the same Page 83 ID value, so ESX thinks you have 2 paths to the same LUN.  ESX's multipath driver uses the Page 83 ID value to determine if 2 LUNs are the same device.

This was a problem with certain early Solaris based arrays, and other arrays that used Solaris-based pre-COMSTAR code, as well as arrays that use SCST-based code, which it looks like yours does.

There's really nothing you can do about it, because this is a bug in the array's software (it shouldn't be exporting 2 different LUNs with the same Page 83 ID value - its a violation of the standard).

[eforbus]

That seems reasonable. Is there any way to check and see what page 83 id was that ESX received for each LUN?

[mcowger]

Its there in your mpath log above: 706f636c6162766f

[eforbus]

It looks like it was definitely the page 83 id that ESX did not like here. I checked with the storage array admin and the array does use scst but a newer version (2.0.0.1). Apparently you can set the page 83 id in the config and it was previously set to:

LUN 0 - 0x83 id = "poclabvol1 e7260bac"

LUN 1 - 0x83 id = "poclabvol2 f5a22ed5"

I read on another forum for scst that some windows machines only read the first 8 characters of the id to check for uniqueness ("poclabvo" in our case) but wasn't sure what ESX did. I just had the 0x83 id's changed to the following and it works!!

LUN 0 - 0x83 id = "e7260bac"

LUN 1 - 0x83 id = "f5a22ed5"

vCenter now shows both LUNs and esxcfg-mpath -b shows 2 devices! Thanks for your help in pointing me in the right direction

esxcfg-mpath -b
naa.600508e0000000001e10a036bc3f5601 : LSILOGIC Serial Attached SCSI Disk (naa.600508e0000000001e10a036bc3f5601)
   vmhba0:C1:T0:L0 LUN:0 state:active sas Adapter: 5c84c75ed0910000  Target: 1563fbc36a0101e

eui.6537323630626163 : SCST_FIO Fibre Channel Disk (eui.706f636c6162766f)
   vmhba2:C0:T0:L0 LUN:0 state:active fc Adapter: WWNN: 20:00:00:25:b5:00:00:ae WWPN: 20:00:00:25:b5:00:00:1e  Target: WWNN: 20:00:00:24:ff:32:10:f3 WWPN: 21:00:00:24:ff:32:10:f3
   vmhba2:C0:T1:L0 LUN:0 state:active fc Adapter: WWNN: 20:00:00:25:b5:00:00:ae WWPN: 20:00:00:25:b5:00:00:1e  Target: WWNN: 20:00:00:24:ff:32:10:f2 WWPN: 21:00:00:24:ff:32:10:f2
   vmhba1:C0:T0:L0 LUN:0 state:active fc Adapter: WWNN: 20:00:00:25:b5:00:00:ae WWPN: 20:00:00:25:b5:00:00:be  Target: WWNN: 20:00:00:24:ff:32:45:c2 WWPN: 21:00:00:24:ff:32:45:c2
   vmhba1:C0:T1:L0 LUN:0 state:active fc Adapter: WWNN: 20:00:00:25:b5:00:00:ae WWPN: 20:00:00:25:b5:00:00:be  Target: WWNN: 20:00:00:24:ff:32:45:c3 WWPN: 21:00:00:24:ff:32:45:c3

mpx.vmhba32:C0:T0:L0 : Local USB CD-ROM (mpx.vmhba32:C0:T0:L0)
   vmhba32:C0:T0:L0 LUN:0 state:active Local HBA vmhba32 channel 0 target 0

mpx.vmhba32:C0:T0:L1 : Local USB Direct-Access (mpx.vmhba32:C0:T0:L1)
   vmhba32:C0:T0:L1 LUN:1 state:active Local HBA vmhba32 channel 0 target 0

mpx.vmhba32:C0:T0:L2 : Local USB Direct-Access (mpx.vmhba32:C0:T0:L2)
   vmhba32:C0:T0:L2 LUN:2 state:active Local HBA vmhba32 channel 0 target 0

eui.6635613232656435 : SCST_FIO Fibre Channel Disk (eui.706f636c6162766f)
   vmhba2:C0:T0:L1 LUN:1 state:active fc Adapter: WWNN: 20:00:00:25:b5:00:00:ae WWPN: 20:00:00:25:b5:00:00:1e  Target: WWNN: 20:00:00:24:ff:32:10:f3 WWPN: 21:00:00:24:ff:32:10:f3
   vmhba2:C0:T1:L1 LUN:1 state:active fc Adapter: WWNN: 20:00:00:25:b5:00:00:ae WWPN: 20:00:00:25:b5:00:00:1e  Target: WWNN: 20:00:00:24:ff:32:10:f2 WWPN: 21:00:00:24:ff:32:10:f2
   vmhba1:C0:T0:L1 LUN:1 state:active fc Adapter: WWNN: 20:00:00:25:b5:00:00:ae WWPN: 20:00:00:25:b5:00:00:be  Target: WWNN: 20:00:00:24:ff:32:45:c2 WWPN: 21:00:00:24:ff:32:45:c2
   vmhba1:C0:T1:L1 LUN:1 state:active fc Adapter: WWNN: 20:00:00:25:b5:00:00:ae WWPN: 20:00:00:25:b5:00:00:be  Target: WWNN: 20:00:00:24:ff:32:45:c3 WWPN: 21:00:00:24:ff:32:45:c3

[mcowger]

Great news!  Glad to help.  And you gave me an idea for a blog post