iSCSI LUNs Connected but Datastore Not Visible/Available

[tdubb123]

i got some nimble storage iscsi volumes that are showing as mounted under the iscsi hba but it is not showing up as a datastore .

any idea?

[vfk]

Can you see the lun via the cli? esxcfg-scsidevs -l

[tdubb123]

yes i see it there

[vfk]

I suspect then you are not seeing the lun when you are connected to vCenter, in this case, if the lun is presented to all hosts in the cluster, then connect to one of the esxi hosts directly and provision that lun.   Once provisioned, rescan the hosts in vCenter.

[tdubb123]

i did connect to that host. I remove the scsi targets rescanned and readded but still does not show up as datastore. Its only showing up as a mounted volume under the iscsi hba. I have 2 other hosts running the same build and they are able to see the datastore.

The datastore is already added int the cluster so it should be visiable among all hosts. I also verified all the iqns are correct from the san side since I am seeing a mounted volume under the iscsi hba.

[vfk]

Is this a clone / snapshot LUN?  can you check the vmkernal in /var/logs/vmkernel.log;

[tdubb123]

not a clone. brand new volume.

I see these in vmkernel

/var/log # cat vmkernel.log | grep -i nimble

2014-11-10T16:44:36.627Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000001 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:44:59.294Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000004 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:45:02.321Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000004 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:45:15.921Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000002 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:45:18.948Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000002 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:45:41.074Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:45:44.102Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000001 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:46:06.769Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000003 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:46:09.799Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000003 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:46:23.401Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000002 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:46:26.431Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000002 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:46:49.100Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000004 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:46:52.130Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000004 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:47:04.977Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000003 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:47:08.005Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000003 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:47:31.436Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000002 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:47:34.581Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000002 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:47:47.420Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:47:50.445Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000001 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:48:13.115Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000003 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:48:16.141Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000003 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:48:28.992Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000002 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

2014-11-10T16:48:32.022Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000002 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

/var/log # cat vmkernel.log | grep -i nimble | more

2014-11-10T16:44:36.627Z cpu23:33619)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000001 TARGET: iqn.2007-11.com.nimblestorage:nim-ucs-vol2-v53e2c6e4ade1c307.00000002.fe0305e3 TPGT: 99c TSIH: 0]

[weblee]

Take the problem host out of the cluster if you can....then rejoin the cluster.  If the other members see the storage there is a problem with arbitration within the cluster specific to this host.

[tdubb123]

tried that same issue

[tdubb123]

seeing more in the logs

~ # cat /var/log/vmkernel.log | grep -i eui.4ce09c97d9bf27126c9ce900e30503fe | more

2014-11-10T16:44:59.294Z cpu4:33080)NMP: nmp_ThrottleLogForDevice:2322: Cmd 0x28 (0x413647829d40, 0) to dev "eui.4ce09c97d9bf27126c9ce900e30503fe" on path "vmhba32:C3:T2:L0" Failed: H:0x5 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL

2014-11-10T16:44:59.294Z cpu4:33080)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "eui.4ce09c97d9bf27126c9ce900e30503fe" state in doubt; requested fast path state update...

2014-11-10T16:44:59.295Z cpu4:33080)ScsiDeviceIO: 2338: Cmd(0x413647829d40) 0x28, CmdSN 0x14d4a from world 0 to dev "eui.4ce09c97d9bf27126c9ce900e30503fe" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0.

2014-11-10T16:44:59.295Z cpu19:159396)Partition: 541: Read of GPT entries (hdrlba=1) failed on "eui.4ce09c97d9bf27126c9ce900e30503fe": I/O error

2014-11-10T16:44:59.295Z cpu19:159396)Partition: 644: Read from primary gpt table failed on "eui.4ce09c97d9bf27126c9ce900e30503fe".

2014-11-10T16:45:15.921Z cpu14:33621)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "eui.4ce09c97d9bf27126c9ce900e30503fe" state in doubt; requested fast path state update...

2014-11-10T16:45:15.921Z cpu14:33621)ScsiDeviceIO: 2325: Cmd(0x413647829d40) 0x28, CmdSN 0x14d7c from world 0 to dev "eui.4ce09c97d9bf27126c9ce900e30503fe" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0.

2014-11-10T16:45:41.074Z cpu22:33621)NMP: nmp_ThrottleLogForDevice:2322: Cmd 0x28 (0x413647829d40, 0) to dev "eui.4ce09c97d9bf27126c9ce900e30503fe" on path "vmhba32:C0:T2:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x6 0x29 0x0. Act:EVAL

2014-11-10T16:45:41.074Z cpu22:33621)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "eui.4ce09c97d9bf27126c9ce900e30503fe" state in doubt; requested fast path state update...

2014-11-10T16:45:41.074Z cpu19:159396)Partition: 541: Read of GPT entries (hdrlba=4294967295) failed on "eui.4ce09c97d9bf27126c9ce900e30503fe": I/O error

2014-11-10T16:45:41.074Z cpu19:159396)Partition: 649: Read from both primary and secondary gpt table failed on "eui.4ce09c97d9bf27126c9ce900e30503fe".

2014-11-10T16:45:41.074Z cpu19:159396)WARNING: Partition: 1112: Partition table read from device eui.4ce09c97d9bf27126c9ce900e30503fe failed: I/O error

2014-11-10T16:45:41.075Z cpu19:159396)ScsiDeviceIO: 2325: Cmd(0x413647829c00) 0x28, CmdSN 0x4 from world 159396 to dev "eui.4ce09c97d9bf27126c9ce900e30503fe" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-11-10T16:45:41.091Z cpu19:159396)ScsiDeviceIO: 2325: Cmd(0x41364782a240) 0x28, CmdSN 0x7 from world 159396 to dev "eui.4ce09c97d9bf27126c9ce900e30503fe" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-11-10T16:45:41.106Z cpu19:159396)ScsiDeviceIO: 2325: Cmd(0x41364782e840) 0x28, CmdSN 0xa from world 159396 to dev "eui.4ce09c97d9bf27126c9ce900e30503fe" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-11-10T16:45:41.121Z cpu19:159396)ScsiDeviceIO: 2325: Cmd(0x41364782e340) 0x28, CmdSN 0xd from world 159396 to dev "eui.4ce09c97d9bf27126c9ce900e30503fe" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-11-10T16:45:41.137Z cpu19:159396)ScsiDeviceIO: 2325: Cmd(0x413646b310c0) 0x28, CmdSN 0x10 from world 159396 to dev "eui.4ce09c97d9bf27126c9ce900e30503fe" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-11-10T16:45:41.152Z cpu19:159396)ScsiDeviceIO: 2325: Cmd(0x413647061700) 0x28, CmdSN 0x13 from world 159396 to dev "eui.4ce09c97d9bf27126c9ce900e30503fe" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-11-10T16:45:41.167Z cpu19:159396)ScsiDeviceIO: 2325: Cmd(0x4136454ea700) 0x9e, CmdSN 0x14d97 from world 0 to dev "eui.4ce09c97d9bf27126c9ce900e30503fe" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-11-10T16:45:41.182Z cpu19:159396)ScsiDeviceIO: 2325: Cmd(0x4136454ea700) 0x28, CmdSN 0x14d9a from world 0 to dev "eui.4ce09c97d9bf27126c9ce900e30503fe" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-11-10T16:46:06.769Z cpu18:33621)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "eui.4ce09c97d9bf27126c9ce900e30503fe" state in doubt; requested fast path state update...

[weblee]

Are you using jumbo frames?  if so make sure its set on the iscsi vmk nics

[timlane]

I had this same issue.  The nimble was installed in another datacenter, Jumbo Frames were not enabled between the switches connecting the data centers.  I could see the volumes in vCenter, but was unable to create the datastore.  I enabled Jumbo Frames on those switches, then I could create the datastores from the volumes in vCenter.