VMware Cloud Community
vchevakula
Contributor
Contributor

failed H:0x8 D:0x0 P:0x0

I m get i/o latency on from the storage

2014-05-27T06:26:44.441Z cpu22:57431)ScsiDeviceIO: 1204: Device eui.304c397836343653 performance has improved. I/O latency reduced from 468909 microseconds to 229810 microseconds.

2014-05-27T06:27:14.916Z cpu6:1103322)WARNING: ScsiDeviceIO: 1224: Device eui.304c397836343653 performance has deteriorated. I/O latency increased from average value of 116694 microseconds to 2512396 microseconds.

2014-05-27T06:27:14.925Z cpu11:8203)WARNING: ScsiDeviceIO: 1224: Device eui.304c397836343653 performance has deteriorated. I/O latency increased from average value of 116694 microseconds to 6333005 microseconds.

2014-05-27T06:27:15.355Z cpu9:8201)ScsiDeviceIO: 1204: Device eui.304c397836343653 performance has improved. I/O latency reduced from 6333005 microseconds to 1257520 microseconds.

2014-05-27T06:27:15.556Z cpu5:1103324)ScsiDeviceIO: 1204: Device eui.304c397836343653 performance has improved. I/O latency reduced from 1257520 microseconds to 244217 microseconds.

2014-05-27T06:27:15.564Z cpu7:841499)ScsiDeviceIO: 1204: Device eui.304c397836343653 performance has improved. I/O latency reduced from 244217 microseconds to 231832 microseconds.

2014-05-27T06:27:21.879Z cpu23:57431)WARNING: ScsiDeviceIO: 1224: Device eui.304c397836343653 performance has deteriorated. I/O latency increased from average value of 116697 microseconds to 2474517 microseconds.

2014-05-27T06:27:22.122Z cpu10:57156)ScsiDeviceIO: 1204: Device eui.304c397836343653 performance has improved. I/O latency reduced from 2474517 microseconds to 488562 microseconds.

2014-05-27T06:27:22.829Z cpu2:1103324)ScsiDeviceIO: 1204: Device eui.304c397836343653 performance has improved. I/O latency reduced from 488562 microseconds to 232052 microseconds.

2014-05-27T06:27:30.807Z cpu6:8198)WARNING: ScsiDeviceIO: 1224: Device eui.304c397836343653 performance has deteriorated. I/O latency increased from average value of 116700 microseconds to 2384458 microseconds.

2014-05-27T06:27:30.868Z cpu2:841501)ScsiDeviceIO: 1204: Device eui.304c397836343653 performance has improved. I/O latency reduced from 2384458 microseconds to 471383 microseconds.

2014-05-27T06:27:30.979Z cpu2:1103324)ScsiDeviceIO: 1204: Device eui.304c397836343653 performance has improved. I/O latency reduced from 471383 microseconds to 230272 microseconds.

2014-05-27T06:27:35.553Z cpu7:57430)WARNING: ScsiDeviceIO: 1224: Device eui.304c397836343653 performance has deteriorated. I/O latency increased from average value of 116699 microseconds to 2365955 microseconds.

2014-05-27T06:27:36.735Z cpu7:1103330)ScsiDeviceIO: 1204: Device eui.304c397836343653 performance has improved. I/O latency reduced from 2365955 microseconds to 459709 microseconds.

After getting these above errors …I m losing datastores …It’s on all datastore (same array) …

Below error shows how its retrying to add those datastores ……

2014-05-27T06:27:47.143Z cpu13:1265594)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x2a (0x4124011e6180, 8216) to dev "eui.61305372666a416b" on path "vmhba38:C1:T2:L0" Failed: H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

2014-05-27T06:27:47.143Z cpu13:1265594)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "eui.61305372666a416b" state in doubt; requested fast path state update...

2014-05-27T06:27:47.143Z cpu13:1265594)ScsiDeviceIO: 2318: Cmd(0x4124011e6180) 0x2a, CmdSN 0x18fb2 from world 8216 to dev "eui.61305372666a416b" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-05-27T06:27:51.597Z cpu13:1265594)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "eui.61305372666a416b" state in doubt; requested fast path state update...

2014-05-27T06:27:51.597Z cpu13:1265594)ScsiDeviceIO: 2331: Cmd(0x4124064d65c0) 0x2a, CmdSN 0x18fb3 from world 8216 to dev "eui.61305372666a416b" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-05-27T06:27:51.667Z cpu12:8227)ScsiDeviceIO: 2300: Cmd(0x412400958540) 0x2a, CmdSN 0x80000075 from world 1103291 to dev "eui.61305372666a416b" failed H:0x8 D:0x0 P:0x0

2014-05-27T06:27:51.667Z cpu12:8227)ScsiDeviceIO: 2300: Cmd(0x412400958340) 0x2a, CmdSN 0x80000015 from world 1103291 to dev "eui.61305372666a416b" failed H:0x8 D:0x0 P:0x0

2014-05-27T06:27:51.667Z cpu12:8227)ScsiDeviceIO: 2300: Cmd(0x4124039263c0) 0x2a, CmdSN 0x80000042 from world 1103291 to dev "eui.61305372666a416b" failed H:0x8 D:0x0 P:0x0

2014-05-27T06:27:51.667Z cpu12:8227)ScsiDeviceIO: 2300: Cmd(0x412401c8c780) 0x2a, CmdSN 0x8000002e from world 1103291 to dev "eui.61305372666a416b" failed H:0x8 D:0x0 P:0x0

2014-05-27T06:27:51.667Z cpu12:8227)ScsiDeviceIO: 2300: Cmd(0x412400ae8fc0) 0x2a, CmdSN 0x8000003e from world 1103291 to dev "eui.61305372666a416b" failed H:0x8 D:0x0 P:0x0

2014-05-27T06:27:51.667Z cpu12:8227)ScsiDeviceIO: 2300: Cmd(0x412403933cc0) 0x2a, CmdSN 0x80000041 from world 1103291 to dev "eui.61305372666a416b" failed H:0x8 D:0x0 P:0x0

2014-05-27T06:27:51.667Z cpu12:8227)ScsiDeviceIO: 2300: Cmd(0x412400ae80c0) 0x2a, CmdSN 0x80000033 from world 1103291 to dev "eui.61305372666a416b" failed H:0x8 D:0x0 P:0x0


My question is "failed H:0x8 D:0x0 P:0x0 " what this indicates ...is it a host issue or storage issue


one of the vmware blog tell me that its " This status is returned when the HBA driver has aborted the I/O. It can also occur if the HBA does a reset of the target."  and some other blog says it's a storage array controller issue "https://communities.vmware.com/message/1695876 "


I m running host with ESXi 5.1 update 1 and my storage array arcastor SSD iscsi san. Can anyone please help with this issue ..? let me know if you need more details


Thank you

8 Replies
DavoudTeimouri
Virtuoso
Virtuoso

Hi,

Check you SAN switches ports also.

VMware KB: Understanding SCSI host-side NMP errors/conditions in ESX 4.x and ESXi 5.x

-------------------------------------------------------------------------------------
Davoud Teimouri - https://www.teimouri.net - Twitter: @davoud_teimouri Facebook: https://www.facebook.com/teimouri.net/
KeymanP
Enthusiast
Enthusiast

Regarding I/O latency message, please check whether there is any REAL performance issue there, otherwise it would just informational.

You can refer to the following KB

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=200723...

And about the failed H:0x8 D:0x0 P:0x0 message, as you encounter this message for all ESXi hosts at the same time, you can check the followng

1. Any storage controller port issue

2. Any storage performance issue, it may caused by heavy i/o / loading made the storage in unresponsive state.

3. Check the connection between servers to storage, As you are using ISCSI, any connection dropped between the two?

4. Are you using Blade server in single/different enclosure? Different firmware level may also cause the connection issue.

Keyman | VCAP5-DCA, VCP5-DCV, VCP3
Reply
0 Kudos
vchevakula
Contributor
Contributor

davoud,

I already had reviewed this article .....based on that article ...it say's that " This status is returned when the HBA driver has aborted the I/O. It can also occur if the HBA does a reset of the target."...but at the same time different the other link says it was with storage issue. I was just trying find what exactly I was going through is it storage issue or san issue...but thanks for you reply

Reply
0 Kudos
vchevakula
Contributor
Contributor

hello keyman,

1. Any storage controller port issue

I m trying to review the logs on controller port...i will let you know once I got those logs

2. Any storage performance issue, it may caused by heavy i/o / loading made the storage in unresponsive state.

yes because of i/o latency.... storage is getting unresponsive

3. Check the connection between servers to storage, As you are using ISCSI, any connection dropped between the two?

links were good .... they were never down... I got that info from our networking team

4. Are you using Blade server in single/different enclosure? Different firmware level may also cause the connection issue.

we are using rack servers and firmware levels are same ....but I m trying to upgrade the firmware....

But your answer is helpful

KeymanP
Enthusiast
Enthusiast

hello vchevakula,

Yes check the controller would help to check the any error in SCSI layer.

You may also check the queue depth in your storage and ESXi host

VMware KB: Checking the queue depth of the storage adapter and the storage device

Change the queue depth may help to let your storage and ESXi host determine how to handle the SCSI command and won't let it aborted.

Keyman | VCAP5-DCA, VCP5-DCV, VCP3
Reply
0 Kudos
RuneSvendsen
Contributor
Contributor

Hi,

did you find a solution for this? - we are having the same problem, and sometimes a host looses access to datastores because of a esx.problem.vmfs.heartbeat.timedout

Reply
0 Kudos
ChaseHansen
Enthusiast
Enthusiast

Anyone still having this problem seeing the same error along with heartbeat timeouts as well.

Reply
0 Kudos
HariRajan
Hot Shot
Hot Shot

This issue can happen due to many reason .

are you seeing same error in all the esxi host at the same time ? , or are your running de-duplication / decompression jobs in the storage in these time slots.

finally is it a blade server ?

Thanks & Regards in Plenteous . Hari Rajan
Reply
0 Kudos