Dead Path problem (Compellent iSCSI)

[mfrycz]

Hi,

I have a problem with one of my ESX Hosts (3.5.0, 207905)

iSCSI connectivity is dropping all the time...

Apr 15 16:15:43 host vmkernel: 38:04:32:11.307 cpu5:1048)WARNING: SCSI: 1926: Could not unclaim path vmhba1:0:12. Target is busy refcount = 2.
Apr 15 16:15:44 host vmkernel: 38:04:32:12.732 cpu5:1048)WARNING: SCSI: 1506: Cannot remove path vmhba1:1:12. Target vmhba1:0:12 is active (count 1).
Apr 15 16:15:44 host vmkernel: 38:04:32:12.732 cpu5:1048)WARNING: SCSI: 1859: Path vmhba1:1:12 could not be removed during path unclaim
Apr 15 16:15:48 host vmkernel: 38:04:32:16.743 cpu5:1048)WARNING: SCSI: 1506: Cannot remove path vmhba2:0:12. Target vmhba1:0:12 is active (count 1).
Apr 15 16:15:48 host vmkernel: 38:04:32:16.743 cpu5:1048)WARNING: SCSI: 1859: Path vmhba2:0:12 could not be removed during path unclaim
Apr 15 16:15:49 host vmkernel: 38:04:32:18.172 cpu5:1048)WARNING: SCSI: 1506: Cannot remove path vmhba2:1:12. Target vmhba1:0:12 is active (count 1).
Apr 15 16:15:49 host vmkernel: 38:04:32:18.172 cpu5:1048)WARNING: SCSI: 1859: Path vmhba2:1:12 could not be removed during path unclaim
Apr 15 16:15:54 host vmkernel: 38:04:32:22.447 cpu9:1063)WARNING: SCSI: 5437: vml.02000c00006000d310000c33000000000000000a09436f6d70656c: Too many failed retries 33 (32),  Returning I/O failure. 0x25 D:0x0/H:0x1 0x0 0x0 0x0
Apr 15 16:15:54 host vmkernel: 38:04:32:22.447 cpu12:1047)WARNING: ScsiDevice: 3362: Failed for vml.02000c00006000d310000c33000000000000000a09436f6d70656c: No connection

From this output i know that I have dead path to LUN 12...

Already tried rescanning HBAs from GUI,

as well as

esxcfg-rescan -d vmhba1

esxcfg-rescan -d vmhba2

Didnt work and ESX Host still showing LUN12 as a dead path...

we are using Compellent SAN storage here and this host is connected only through iSCSI...

any ideas ?

[l33tpfy]

Has the host been recently updated? If so check that the correct ESX version is selected on the Compellent SAN. I just ran into an issue when I started upgrading to 4.1 and 4.0 was selected on the storage side. I started seeing a lot of errors and storage connectivity issues although on FC media. When I changed the OS version it resolved the problem. Also confirmed with compellent that it was the problem.

If you can take down VM operations that hit that LUN. See if the problem persists. Disable any third party storage agents.

Do you have dedicated vmkernal for the iSCSI traffic?

How many other ESX hosts access the LUN?

Have any hardware iSCSI adapters you can use?

Is snapshotting occurring when this occurs? What about VM backups?

How many LUNS is this host accessing?

[opbz]

you need to go to the properties of the swiscsi adapter and then go to the static tab.

When you first configure iscsi under the dynamic tab it populates the static tab with the paths it discoverred. Problem is that if you delete/modify paths they are not removed from this tab.

So delete the path and do a full rescan

good luck

[mfrycz]

The host was upgraded few months ago with latest update 5 from ESX 3.5 U2

OS version is set to 3.5 (which is correct)

Also none of 3rd party agents are running...

Q: Do you have dedicated VMKernel for the iSCSI traffic?

A: Yes I do

Q: How many other ESX hosts access the LUN?

A: Just this one

Q: Have any hardware iSCSI adapters you can use?

A: I got two Qlogic hardware iSCSI and on both this patch is dead

Q: Is snapshotting occurring when this occurs? What about VM backups?

A: this host is not in use apart of holding few powered off vm's (this is our DR stand-by host)

Q: How many LUNs is this host accessing?

A: 17 LUNs including this dead one

I have tried removing static paths and rescanning them but without any luck the dead path still appears

PS.From Compellent point of view there is no mapping for this Host to LUN12 (not to confuse anyone) and all mapping are done to the cluster not to individual Hosts...