Remove dead path to iSCSI storage

[tcrowley1]

Version ESX 4.0.0, 236512

Storage iSCSI

Initiator Software

A LUN that was presented to my ESX host was deleted on the san without removing it from the host and now I’m getting failover errors in the vmkernel.

Mar 30 06:49:40 bl480g1-03 vmkernel: 20:09:00:10.348 cpu1:4222)WARNING: NMP: nmp_DeviceAttemptFailover: Logical device "naa.6090a01820dc7c5f9b97a440089a3483": awaiting fast path state update...

Mar 30 06:49:41 bl480g1-03 vmkernel: 20:09:00:11.348 cpu1:4222)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device "naa.6090a01820dc7c5f9b97a440089a3483" - issuing command 0x4100030b07c0

Mar 30 06:49:41 bl480g1-03 vmkernel: 20:09:00:11.348 cpu1:4222)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device "naa.6090a01820dc7c5f9b97a440089a3483" - failed to issue command due to Not found (APD), try again...

Mar 30 06:49:41 bl480g1-03 vmkernel: 20:09:00:11.348 cpu1:4222)WARNING: NMP: nmp_DeviceAttemptFailover: Logical device "naa.6090a01820dc7c5f9b97a440089a3483": awaiting fast path state update...

Mar 30 06:49:42 bl480g1-03 vmkernel: 20:09:00:12.347 cpu6:4222)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device "naa.6090a01820dc7c5f9b97a440089a3483" - issuing command 0x4100030b07c0

Mar 30 06:49:42 bl480g1-03 vmkernel: 20:09:00:12.347 cpu6:4222)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device "naa.6090a01820dc7c5f9b97a440089a3483" - failed to issue command due to Not found (APD), try again...

With esxcfg-mpath both paths are showing dead.

How do I remove these dead paths?

[binoche]

esxcfg-rescan -d --all

binoche, VMware VCP, Cisco CCNA

[tcrowley1]

I get the following error message

# esxcfg-rescan -d --all

esxcfg-rescan: unrecognized option `--all'

You may only perform one scan type at a time. Running with no arguments will do all 3 scan types

esxcfg-rescan

If I run esxcfg-rescan -d it doesn't remove the dead paths

Tim

[binoche]

esxcfg-rescan -d vmhba is your iSCSI vmhba

binoche, VMware VCP, Cisco CCNA

[tcrowley1]

When I run esxcfg-mpath -l it's still showing up as dead

iqn.1998-01.com.vmware:bl480g1-03-622b8aa8--

Runtime Name: vmhba32:C1:T14:L0

Device: naa.6090a01820dc7c5f9b97a440089a3483

Device Display Name: EQLOGIC iSCSI Disk (naa.6090a01820dc7c5f9b97a440089a3483)

Adapter: vmhba32 Channel: 1 Target: 14 LUN: 0

Adapter Identifier: iqn.1998-01.com.vmware:bl480g1-03-622b8aa8

Target Identifier:

Plugin: NMP

State: dead

Transport: iscsi

[binoche]

esxcfg-rescan -d vmhba32;

binoche, VMware VCP, Cisco CCNA

[tcrowley1]

That's the command that I ran but with no joy

[tcrowley1]

I had the same issue with another host in the same cluster and the command esxcfg-rescan -d vmhba32 worked

[ovatandas]

Hi,

Any other advice for this problem?

I had the same issue and "esxcfg-rescan -d" command did not work for me.

[AndreTheGiant]

Check if the LUN is still present in: ESX / Configuration / Storage Adapter / Properties of iSCSI interface / Static Discover.

If yes, remove it, and repeat on all your hosts.

Andre

[ovatandas]

I am sorry, I didn't say that it is FC path. It is not present, but I see a zero GB LUN in the storage adapters device list.

[binoche]

0GB, usually it is something like management lun, it is reported from your storage and harmless

binoche, VMware VCP, Cisco CCNA

[ovatandas]

But its path is dead and it gives vmkernel errors every second.

[GGTEAM]

I just getting same log in /var/log/vmkernel.

i dont remove dead ISCSI path.

How I resolve this problem ?

[BenConrad]

Try Andre's suggestion with the static targets.

To be honest, anything less than 4.1 U1 software iSCSI is not a good platform to be on, especially when needing to remove old/dead paths.  Host reboots may be in order 😞 

I've done a ton of iSCSI resiliancy testing on 4.1 U1 with the S/W initiator, it's solid.

Ben

[BalachandarL]

Hi Ben,

Our ESX is 4.0 with FC connectivity. We have 2 devices in dead state and continuously receiving warning messages. It seems, the LUNs removed already without properly cleaned up in ESX.

Shall we execute the command "esxcfg-rescan -d --all" to clear the dead path and stop the warning messages? Kindly help.

FYI,

--------------------------------------------------------------------------

WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device "naa.6006016064012a002051729ea2e0df11" - failed to issue command due to Not found (APD), try again...

WARNING: NMP: nmp_DeviceAttemptFailover: Logical device "naa.6006016064012a002051729ea2e0df11": awaiting fast path state update...

--------------------------------------------------------------------------

fc.20000025b5012800:20000025b5012802-fc.50060160c6e010e9:5006016746e010e9-naa.6006016064012a002051729ea2e0df11

   Runtime Name: vmhba2:C0:T0:L6

   Device: naa.6006016064012a002051729ea2e0df11

   Device Display Name: DGC Fibre Channel Disk (naa.6006016064012a002051729ea2e0df11)

   Adapter: vmhba2 Channel: 0 Target: 0 LUN: 6

   Adapter Identifier: fc.20000025b5012800:20000025b5012802

   Target Identifier: fc.50060160c6e010e9:5006016746e010e9

   Plugin: NMP

   State: dead

   Transport: fc

fc.20000025b5012800:20000025b5012802-fc.50060160c6e010e9:5006016746e010e9-naa.6006016064012a0068053eca4ce8df11

   Runtime Name: vmhba2:C0:T0:L7

   Device: naa.6006016064012a0068053eca4ce8df11

   Device Display Name: DGC Fibre Channel Disk (naa.6006016064012a0068053eca4ce8df11)

   Adapter: vmhba2 Channel: 0 Target: 0 LUN: 7

   Adapter Identifier: fc.20000025b5012800:20000025b5012802

   Target Identifier: fc.50060160c6e010e9:5006016746e010e9

   Plugin: NMP

   State: dead

   Transport: fc

Bala

[marchlam]

Hi,

I got the same issue before, But I did success remove it by checking all Virtual Machines.

Would you please check your virtual machine page, are there any VM left here?

If your LUN dead (on both cluster), then your VM is also dead.

So it should be remove.

Inventory > Datastore > Virtual Machine (Tab)  >

And see if there are VM still here ?

Thanks!

Marchlam

(from Hong Kong )