AlbertWT
Virtuoso
Virtuoso

Need help in interpreting the error log in ESXi 5.1u1 host

Folks,

I need your assistance in interpreting the following ESXi 5.1 Update 1 error log:

2013-09-13T00:16:36.301Z cpu0:8705)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60060160e7311b006887b1bf34a7df11": awaiting fast path state update before retrying failed command again...

2013-09-13T00:16:36.632Z cpu4:8335)ScsiDeviceIO: 2331: Cmd(0x4124403c1d40) 0x12, CmdSN 0x458c3 from world 0 to dev "naa.60060160e7311b006887b1bf34a7df11" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0.

2013-09-13T00:16:36.632Z cpu4:8335)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.60060160e7311b006887b1bf34a7df11" is blocked. Not starting I/O from device.

2013-09-13T00:16:36.632Z cpu5:8313)WARNING: VMW_VAAIP_CX: cx_claim_device:113:Inquiry to device naa.60060160e7311b006887b1bf34a7df11 failed

2013-09-13T00:16:36.634Z cpu5:8313)WARNING: NMP: nmp_SelectPathAndIssueCommand:2694:PSP selected path "vmhba1:C0:T0:L4" in a bad state (standby) on device "naa.60060160e7311b00c4fe11f33544df11".

2013-09-13T00:16:36.634Z cpu5:8313)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x12 (0x4124403c1d40, 0) to dev "naa.60060160e7311b00c4fe11f33544df11" on path "vmhba1:C0:T0:L4" Failed: H:0x1 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0. Act:FAILOVER

2013-09-13T00:16:36.634Z cpu5:8313)WARNING: NMP: nmp_DeviceRetryCommand:133:Device "naa.60060160e7311b00c4fe11f33544df11": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.

2013-09-13T00:16:36.634Z cpu5:8313)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.60060160e7311b00c4fe11f33544df11" is blocked. Not starting I/O from device.

2013-09-13T00:16:37.301Z cpu3:8704)WARNING: NMP: nmpDeviceAttemptFailover:562:Retry world restore device "naa.60060160e7311b006887b1bf34a7df11" - no more commands to retry

2013-09-13T00:16:37.428Z cpu0:8705)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60060160e7311b00c4fe11f33544df11" - issuing command 0x4124403c1d40

2013-09-13T00:16:37.428Z cpu0:8705)WARNING: NMP: nmp_SelectPathAndIssueCommand:2694:PSP selected path "vmhba1:C0:T0:L4" in a bad state (standby) on device "naa.60060160e7311b00c4fe11f33544df11".

2013-09-13T00:16:37.428Z cpu0:8705)WARNING: NMP: nmpCompleteRetryForPath:348:Retry cmd 0x12 (0x4124403c1d40) to dev "naa.60060160e7311b00c4fe11f33544df11" failed on path "vmhba1:C0:T0:L4" H:0x1 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0.

2013-09-13T00:16:37.428Z cpu0:8705)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60060160e7311b00c4fe11f33544df11": awaiting fast path state update before retrying failed command again...

2013-09-13T00:16:38.336Z cpu3:8704)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60060160e7311b00c4fe11f33544df11" - issuing command 0x4124403c1d40

2013-09-13T00:16:38.336Z cpu3:8704)WARNING: NMP: nmp_SelectPathAndIssueCommand:2694:PSP selected path "vmhba1:C0:T0:L4" in a bad state (standby) on device "naa.60060160e7311b00c4fe11f33544df11".

2013-09-13T00:16:38.336Z cpu3:8704)WARNING: NMP: nmpCompleteRetryForPath:348:Retry cmd 0x12 (0x4124403c1d40) to dev "naa.60060160e7311b00c4fe11f33544df11" failed on path "vmhba1:C0:T0:L4" H:0x1 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0.

it is repeated millions of times in the newly upgraded ESXi host with missing LUN, what could be the issue with the FC SAN storage array ?

This is happening after I have successfully upgraded the ESX host from 4.1 into ESXi 5.1 Update 1 by using the ISO and then forced the migration with Preserve the VMFS volume.

/* Any kind of comment or input would be greatly appreciated */
Tags (1)
6 Replies
abhilashhb
VMware Employee
VMware Employee

Looks like the path to the storage that the host is using is down or offline and its trying to failover to another path.


2013-09-13T00:16:37.428Z cpu0:8705)WARNING: NMP: nmpCompleteRetryForPath:348:Retry cmd 0x12 (0x4124403c1d40) to dev "naa.60060160e7311b00c4fe11f33544df11" failed on path "vmhba1:C0:T0:L4" H:0x1 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0.


H:0x1 represents No connect.


It is ti with failover multipathing sequence. check out this KB article


------------------------------------------------------------------------------------------------------------------------------------ If you find this or any other answer useful please mark the answer as correct or helpful. Abhilash B | Blog : http://vpirate.in | Twitter : @abhilashhb | LinkedIn : https://www.linkedin.com/in/abhilashhb/ |
zXi_Gamer
Virtuoso
Virtuoso

Looks like a lot of fight is going on who has taken the ownership of the device between VMW_VAAIP_CX and NMP.


If you are using any VAAIP, unclaim the device from NMP from esxcli storage claiming

AlbertWT
Virtuoso
Virtuoso

Thanks for the suggestion man, how can I do that ?

--

Disclaimer: This email was sent from my iPhone, apologize for any typographical error.

/* Any kind of comment or input would be greatly appreciated */
0 Kudos
zXi_Gamer
Virtuoso
Virtuoso

VMware vSphere 5.1

It is a two step process. First, unclaim the device by any, then reclaim the device by the PSA VMW_VAAIP_CX. The above link is useful

0 Kudos
AlbertWT
Virtuoso
Virtuoso

Many thanks people for the clarification, based on this article: VMware KB: Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere 5.0/5.1

There is no explanation for the error sensing code: Possible sense data: 0x5 0x24 0x0

/* Any kind of comment or input would be greatly appreciated */
0 Kudos
zXi_Gamer
Virtuoso
Virtuoso

04/02 DTLPWROMAEBKVF LOGICAL UNIT NOT READY, INITIALIZING COMMAND REQUIRED



VMware KB: Interpreting SCSI sense codes in VMware ESXi / ESX

0 Kudos