Need help in interpreting the error log in ESXi 5.1u1 host

[AlbertWT]

Folks,

I need your assistance in interpreting the following ESXi 5.1 Update 1 error log:

2013-09-13T00:16:36.301Z cpu0:8705)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60060160e7311b006887b1bf34a7df11": awaiting fast path state update before retrying failed command again...

2013-09-13T00:16:36.632Z cpu4:8335)ScsiDeviceIO: 2331: Cmd(0x4124403c1d40) 0x12, CmdSN 0x458c3 from world 0 to dev "naa.60060160e7311b006887b1bf34a7df11" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0.

2013-09-13T00:16:36.632Z cpu4:8335)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.60060160e7311b006887b1bf34a7df11" is blocked. Not starting I/O from device.

2013-09-13T00:16:36.632Z cpu5:8313)WARNING: VMW_VAAIP_CX: cx_claim_device:113:Inquiry to device naa.60060160e7311b006887b1bf34a7df11 failed

2013-09-13T00:16:36.634Z cpu5:8313)WARNING: NMP: nmp_SelectPathAndIssueCommand:2694:PSP selected path "vmhba1:C0:T0:L4" in a bad state (standby) on device "naa.60060160e7311b00c4fe11f33544df11".

2013-09-13T00:16:36.634Z cpu5:8313)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x12 (0x4124403c1d40, 0) to dev "naa.60060160e7311b00c4fe11f33544df11" on path "vmhba1:C0:T0:L4" Failed: H:0x1 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0. Act:FAILOVER

2013-09-13T00:16:36.634Z cpu5:8313)WARNING: NMP: nmp_DeviceRetryCommand:133:Device "naa.60060160e7311b00c4fe11f33544df11": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.

2013-09-13T00:16:36.634Z cpu5:8313)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.60060160e7311b00c4fe11f33544df11" is blocked. Not starting I/O from device.

2013-09-13T00:16:37.301Z cpu3:8704)WARNING: NMP: nmpDeviceAttemptFailover:562:Retry world restore device "naa.60060160e7311b006887b1bf34a7df11" - no more commands to retry

2013-09-13T00:16:37.428Z cpu0:8705)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60060160e7311b00c4fe11f33544df11" - issuing command 0x4124403c1d40

2013-09-13T00:16:37.428Z cpu0:8705)WARNING: NMP: nmp_SelectPathAndIssueCommand:2694:PSP selected path "vmhba1:C0:T0:L4" in a bad state (standby) on device "naa.60060160e7311b00c4fe11f33544df11".

2013-09-13T00:16:37.428Z cpu0:8705)WARNING: NMP: nmpCompleteRetryForPath:348:Retry cmd 0x12 (0x4124403c1d40) to dev "naa.60060160e7311b00c4fe11f33544df11" failed on path "vmhba1:C0:T0:L4" H:0x1 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0.

2013-09-13T00:16:37.428Z cpu0:8705)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60060160e7311b00c4fe11f33544df11": awaiting fast path state update before retrying failed command again...

2013-09-13T00:16:38.336Z cpu3:8704)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60060160e7311b00c4fe11f33544df11" - issuing command 0x4124403c1d40

2013-09-13T00:16:38.336Z cpu3:8704)WARNING: NMP: nmp_SelectPathAndIssueCommand:2694:PSP selected path "vmhba1:C0:T0:L4" in a bad state (standby) on device "naa.60060160e7311b00c4fe11f33544df11".

2013-09-13T00:16:38.336Z cpu3:8704)WARNING: NMP: nmpCompleteRetryForPath:348:Retry cmd 0x12 (0x4124403c1d40) to dev "naa.60060160e7311b00c4fe11f33544df11" failed on path "vmhba1:C0:T0:L4" H:0x1 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0.

it is repeated millions of times in the newly upgraded ESXi host with missing LUN, what could be the issue with the FC SAN storage array ?

This is happening after I have successfully upgraded the ESX host from 4.1 into ESXi 5.1 Update 1 by using the ISO and then forced the migration with Preserve the VMFS volume.

[abhilashhb]

Looks like the path to the storage that the host is using is down or offline and its trying to failover to another path.

2013-09-13T00:16:37.428Z cpu0:8705)WARNING: NMP: nmpCompleteRetryForPath:348:Retry cmd 0x12 (0x4124403c1d40) to dev "naa.60060160e7311b00c4fe11f33544df11" failed on path "vmhba1:C0:T0:L4" H:0x1 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0.

H:0x1 represents No connect.

It is ti with failover multipathing sequence. check out this KB article

[zXi_Gamer]

Looks like a lot of fight is going on who has taken the ownership of the device between VMW_VAAIP_CX and NMP.

If you are using any VAAIP, unclaim the device from NMP from esxcli storage claiming

[AlbertWT]

Thanks for the suggestion man, how can I do that ?

--

Disclaimer: This email was sent from my iPhone, apologize for any typographical error.

[zXi_Gamer]

VMware vSphere 5.1

It is a two step process. First, unclaim the device by any, then reclaim the device by the PSA VMW_VAAIP_CX. The above link is useful

[AlbertWT]

Many thanks people for the clarification, based on this article: VMware KB: Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere 5.0/5.1

There is no explanation for the error sensing code: Possible sense data: 0x5 0x24 0x0

[zXi_Gamer]

04/02 DTLPWROMAEBKVF LOGICAL UNIT NOT READY, INITIALIZING COMMAND REQUIRED

VMware KB: Interpreting SCSI sense codes in VMware ESXi / ESX