Phantom storage device/path won't go away

[jimbartus]

I un-exported a a couple of lun's that my esxi host wasn't using anymore, and two of them cleaned up fine after a rescan. However one of them still shows in the list of devices with a path status of dead. I have tried lots of rescans and refreshes but it just won't disappear. It only shows up on one of the two hba's, so vmhba1 detected it missing and cleaned up, but vmhba2 has chosen to mark it as dead instead of gone.

I'm not sure if its doing any harm, it may be the reason a rescan for new vmfs volumes is so slow, but that was never really quick anyway.

Is there a way to manually prune a device from that list?

Attached are the vsphere screenshots showing the proper behavior on vmhba1 and the dead path on vmhba2.

[RParker]

I'm not sure if its doing any harm, it may be the reason a rescan for new vmfs volumes is so slow, but that was never really quick anyway.

Especially with QLogic, I noticed this many times. But did you zone the fibre? It may still think it's available since the Fibre switch may have zoning setup, and thus a path is still available. Or did you perhaps reassign the same LUN ID and it's not known to the ESX host?

[jimbartus]

The zoning is very simple, server has two hba's, each hba goes into its own switch, each switch has one link to a control node on the storage array. I'm doing single-initiatior-single-target zoning so there's one zone on each switch relevant to this server.

The LUN itself was un-exported from (or un-assigned to) that host. Nothing else is using its LUN ID (yet). Its actually been full blown deleted from the array.

[DSTAVERT]

ISOs stored on it and still referenced in a VM or something similar.

[jimbartus]

ISOs stored on it and still referenced in a VM or something similar.

Dang that got my hopes up. I went through each vm and verified it had no floppy, cdrom, or hard disks associated with the datastore that used to live on that lun, but none did. Some had links to a different lun that I recently deleted which went away just fine. Refreshed/rescanned and the phantom lun is still there.

[DSTAVERT]

Reference to empty snapshot reference files? Grep for a reference in vmx files etc?

[jimbartus]

sorry, don't know what that means exactly. I've ssh'd in and can run a 'find' that shows me all the vmx files, what should I grep for? I skimmed through a few of them and nothing stands out.

[jimbartus]

/vmfs/volumes # ls -l
drwxr-xr-x    1 root     root                  8 Jan  1  1970 4a2fe94c-db78b02f-feea-002219cc24b0
drwxr-xr-t    1 root     root                980 Jun 10 17:11 4a2fe94d-031a0a81-638c-002219cc24b0
drwxr-xr-t    1 root     root               1120 Jun 26 12:29 4a44beed-a8f53d8a-cad1-002219cc24ae
drwxr-xr-t    1 root     root               1820 Jul  8 22:07 4a547114-38dc70a4-27c8-002219cc24ae
drwxr-xr-t    1 root     root               1120 Jul  8 22:30 4a551ddb-ec6a09de-4f62-002219cc24ae
drwxr-xr-t    1 root     root               1120 Jul  8 23:30 4a552bc6-a39f0178-ef92-002219cc24ae
drwxr-xr-x    1 root     root                  8 Jan  1  1970 8eb1e47a-6300a08c-39b7-775bfe97e8b3
l---------    0 root     root               1984 Jan  1  1970 Hypervisor1 -> 8eb1e47a-6300a08c-39b7-775bfe97e8b3
l---------    0 root     root               1984 Jan  1  1970 Hypervisor2 -> cf14bf74-4861b3d7-2817-faaecfe72840
l---------    0 root     root               1984 Jan  1  1970 Hypervisor3 -> efd8efe3-03bc1cbf-15e0-080efd9e7379
drwxr-xr-x    1 root     root                  8 Jan  1  1970 cf14bf74-4861b3d7-2817-faaecfe72840
l---------    0 root     root               1984 Jan  1  1970 datastore1 -> 4a2fe94d-031a0a81-638c-002219cc24b0
drwxr-xr-x    1 root     root                  8 Jan  1  1970 efd8efe3-03bc1cbf-15e0-080efd9e7379
l---------    0 root     root               1984 Jan  1  1970 vmds-nas1 -> 4a552bc6-a39f0178-ef92-002219cc24ae
l---------    0 root     root               1984 Jan  1  1970 vmds-nas2 -> 4a44beed-a8f53d8a-cad1-002219cc24ae
l---------    0 root     root               1984 Jan  1  1970 vmds-r5-1 -> 4a551ddb-ec6a09de-4f62-002219cc24ae
l---------    0 root     root               1984 Jan  1  1970 vmds-r5-2 -> 4a547114-38dc70a4-27c8-002219cc24ae

the only one of those long-string ones that doesnt have a symlink pointing to it is 4a2fe94c-db78b02f-feea-002219cc24b0

/vmfs/volumes # for i in `find ./ -type f -name "*.vmx"`; do grep 4a2fe94c-db78b02f-feea-002219cc24b0 $i; done | wc -l
0

while we're at it, what are Hypervisor1 2 and 3?

[DSTAVERT]

grep is useful for looking for text strings within files. I was just suggesting this to look for references to your phantom luns within any of the configuration type files.

the hypervisor 1 and 2 links refer to the current bootbank instance and the previous or altbootbank instance. Upgrades are applied to the altbootbank. When the host is restarted altbootbank becomes bootbank and bootbank becomes altbootbank. In the case of a boot problem you can revert to altbootbank, the previous version. Hypervisor3 is the location for vmwaretools iso files etc.

[jimbartus]

grep is useful for looking for text strings within files. I was just suggesting this to look for references to your phantom luns within any of the configuration type files.

Yea that command I pasted above has grep in the middle of it, I know what grep is for, what I don't know is what to grep for.

the hypervisor 1 and 2 links refer to the current bootbank instance and the previous or altbootbank instance. Upgrades are applied to the altbootbank. When the host is restarted altbootbank becomes bootbank and bootbank becomes altbootbank. In the case of a boot problem you can revert to altbootbank, the previous version. Hypervisor3 is the location for vmwaretools iso files etc.

cool, thanks.