VMware Cloud Community
SDquality
Enthusiast
Enthusiast

VMFS Datastores showing inactive for some of ESXi hosts(ESXi 5.5.0 build-1746018) in vCenter. Cannot unmount the datastore through GUI or commandline

The datastore status is Mounted for some ESXi hosts but for other ESXi hosts the status is Mounted (but greyed out). I took putty session and found that partition table is fine and on executing


"~ # esxcfg-scsidevs -m | grep <Datastorename>" gives  output as:


eui.00173800ef520028:1                                            :1                                        524t6f62-dfcf62a4-73f1-00215a9c0716  0  VM126datatore


-Unable to unmount datastore and it failed with error: "The Operation not allowed in current state"

Error Stack: Call "HostStorageSystem.UnmontVmfsVolume"


Datastore is in maintenance mode part of SDRS cluster. Refresh and Rescan also did not help.


vmkernel.log  during Refresh Storage adapters.

2014-07-25T13:54:03.081Z cpu3:32816)ScsiDeviceIO: 2337: Cmd(0x412e85adaf00) 0x28, CmdSN 0x3cc0b3 from world 139461 to dev "eui.00173800ef520028" failed H:0x1 D:0x0 P:0x

0 Possible sense data: 0x5 0x20 0x0.

2014-07-25T13:54:03.081Z cpu3:139461 opID=f93369ef)Vol3: 714: Couldn't read volume header from snap-31bcf621-52cebd31-c06b34fb-61a4-00215a9c05d0: Device is permanently

unavailable

2014-07-25T13:54:03.081Z cpu3:139461 opID=f93369ef)WARNING: Vol3: 1875: Failed to refresh FS 532e6f62-dfcf62a4-71fe-00215a9c0716 descriptor: Device is permanently unava

ilable

2014-07-25T13:54:03.081Z cpu3:32816)ScsiDeviceIO: 2337: Cmd(0x412e85adaf00) 0x28, CmdSN 0x3cc0b4 from world 139461 to dev "eui.00173800ef520028" failed H:0x1 D:0x0 P:0x

0 Possible sense data: 0x5 0x20 0x0.

2014-07-25T13:54:03.081Z cpu3:139461 opID=f93369ef)Vol3: 1925: Error refreshing PB resMeta: Device is permanently unavailable

2014-07-25T13:54:03.507Z cpu3:139461 opID=f93369ef)VC: 2059: Device rescan time 822 msec (total number of devices 50)

2014-07-25T13:54:03.507Z cpu3:139461 opID=f93369ef)VC: 2062: Filesystem probe time 300 msec (devices probed 11 of 50)

2014-07-25T13:54:03.507Z cpu3:139461 opID=f93369ef)VC: 2064: Refresh open volume time 1349 msec

2014-07-25T13:54:03.513Z cpu4:139461 opID=f93369ef)ScsiDevice: 5540: eui.00173800ef520028 device :Open count > 0, cannot be brought online

2014-07-25T13:54:03.746Z cpu10:139461 opID=f93369ef)Vol3: 2182: Could not open device '3a64561b-2ba78f83-svmmirror' for probing: Not found

2014-07-25T13:54:03.746Z cpu10:139461 opID=f93369ef)Vol3: 2182: Could not open device '3a64561b-2ba78f83-svmmirror' for probing: Not found

2014-07-25T13:54:03.746Z cpu10:139461 opID=f93369ef)Vol3: 768: Could not open device '3a64561b-2ba78f83-svmmirror' for volume open: Not found

2014-07-25T13:54:03.746Z cpu10:139461 opID=f93369ef)Vol3: 768: Could not open device '3a64561b-2ba78f83-svmmirror' for volume open: Not found

2014-07-25T13:54:03.746Z cpu10:139461 opID=f93369ef)FSS: 5092: No FS driver claimed device '3a64561b-2ba78f83-svmmirror': Not supported

2014-07-25T13:54:03.746Z cpu10:139461 opID=f93369ef)Vol3: 714: Couldn't read volume header from control: Not supported

2014-07-25T13:54:03.746Z cpu10:139461 opID=f93369ef)Vol3: 714: Couldn't read volume header from control: Not supported

2014-07-25T13:54:03.746Z cpu10:139461 opID=f93369ef)FSS: 5092: No FS driver claimed device 'control': Not supported

2014-07-25T13:54:03.747Z cpu10:139461 opID=f93369ef)FSS: 5092: No FS driver claimed device '24bbde85-1c1c0c12-svmmirror': Not supported

2014-07-25T13:54:03.747Z cpu10:139461 opID=f93369ef)Vol3: 714: Couldn't read volume header from 35b962d6-421e8f05-svmmirror: Device is permanently unavailable

2014-07-25T13:54:03.747Z cpu10:139461 opID=f93369ef)Vol3: 714: Couldn't read volume header from 35b962d6-421e8f05-svmmirror: Device is permanently unavailable

2014-07-25T13:54:03.747Z cpu10:139461 opID=f93369ef)FSS: 5092: No FS driver claimed device '35b962d6-421e8f05-svmmirror': Not supported

2014-07-25T13:54:03.834Z cpu15:32828)ScsiDeviceIO: 2337: Cmd(0x412e82596940) 0x28, CmdSN 0x3cc0b5 from world 139461 to dev "eui.00173800ef520028" failed H:0x1 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-07-25T13:54:03.834Z cpu15:139461 opID=f93369ef)Vol3: 714: Couldn't read volume header from snap-31bcf621-52cebd31-c06b34fb-61a4-00215a9c05d0: Device is permanently unavailable

2014-07-25T13:54:03.834Z cpu15:139461 opID=f93369ef)WARNING: Vol3: 1875: Failed to refresh FS 532e6f62-dfcf62a4-71fe-00215a9c0716 descriptor: Device is permanently unavailable

2014-07-25T13:54:03.834Z cpu15:32828)ScsiDeviceIO: 2337: Cmd(0x412e82596940) 0x28, CmdSN 0x3cc0b6 from world 139461 to dev "eui.00173800ef520028" failed H:0x1 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-07-25T14:20:19.344Z cpu10:32823)ScsiDeviceIO: 2337: Cmd(0x412e80051f80) 0x28, CmdSN 0x3cc0bb from world 139896 to dev "eui.00173800ef520028" failed H:0x1 D:0x0 P:0

x0 Possible sense data: 0x0 0x0 0x0.

2014-07-25T14:20:19.344Z cpu10:139896 opID=bfcdcbe0)Vol3: 714: Couldn't read volume header from snap-31bcf621-52cebd31-c06b34fb-61a4-00215a9c05d0: Device is permanently

unavailable

2014-07-25T14:20:19.344Z cpu10:139896 opID=bfcdcbe0)WARNING: Vol3: 1875: Failed to refresh FS 532e6f62-dfcf62a4-71fe-00215a9c0716 descriptor: Device is permanently unavailable

2014-07-25T14:20:19.344Z cpu10:32823)ScsiDeviceIO: 2337: Cmd(0x412e80051f80) 0x28, CmdSN 0x3cc0bc from world 139896 to dev "eui.00173800ef520028" failed H:0x1 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-07-25T14:20:19.344Z cpu10:139896 opID=bfcdcbe0)Vol3: 1925: Error refreshing PB resMeta: Device is permanently unavailable

2014-07-25T14:20:19.351Z cpu10:139896 opID=bfcdcbe0)VC: 2059: Device rescan time 171 msec (total number of devices 50)

2014-07-25T14:20:19.351Z cpu10:139896 opID=bfcdcbe0)VC: 2062: Filesystem probe time 122 msec (devices probed 11 of 50)

2014-07-25T14:20:19.351Z cpu10:139896 opID=bfcdcbe0)VC: 2064: Refresh open volume time 37 msec

2014-07-25T14:20:26.401Z cpu21:139461)World: 14296: VC opID hostd-074b maps to vmkernel opID 9479153d

2014-07-25T14:20:28.163Z cpu8:34179)World: 14296: VC opID C1EF78B4-0000006D maps to vmkernel opID 33b4803

2014-07-25T14:20:28.805Z cpu4:139464 opID=557b531c)World: 14296: VC opID F6D15B37-00001176-6a-3f maps to vmkernel opID 557b531c

2014-07-25T14:20:28.805Z cpu4:139464 opID=557b531c)ScsiDevice: 5540: eui.00173800ef520028 device :Open count > 0, cannot be brought online

2014-07-25T14:20:28.995Z cpu3:139464 opID=557b531c)Vol3: 714: Couldn't read volume header from 35b962d6-421e8f05-svmmirror: Device is permanently unavailable

2014-07-25T14:20:28.995Z cpu3:139464 opID=557b531c)Vol3: 714: Couldn't read volume header from 35b962d6-421e8f05-svmmirror: Device is permanently unavailable

2014-07-25T14:20:28.995Z cpu3:139464 opID=557b531c)FSS: 5092: No FS driver claimed device '35b962d6-421e8f05-svmmirror': Not supported

2014-07-25T14:20:28.996Z cpu3:139464 opID=557b531c)FSS: 5092: No FS driver claimed device '24bbde85-1c1c0c12-svmmirror': Not supported

2014-07-25T14:20:28.996Z cpu3:139464 opID=557b531c)Vol3: 714: Couldn't read volume header from control: Not supported

2014-07-25T14:20:28.996Z cpu3:139464 opID=557b531c)Vol3: 714: Couldn't read volume header from control: Not supported

2014-07-25T14:20:28.996Z cpu3:139464 opID=557b531c)FSS: 5092: No FS driver claimed device 'control': Not supported

2014-07-25T14:20:28.996Z cpu3:139464 opID=557b531c)Vol3: 2182: Could not open device '3a64561b-2ba78f83-svmmirror' for probing: Not found

2014-07-25T14:20:28.996Z cpu3:139464 opID=557b531c)Vol3: 2182: Could not open device '3a64561b-2ba78f83-svmmirror' for probing: Not found

2014-07-25T14:20:28.996Z cpu3:139464 opID=557b531c)Vol3: 768: Could not open device '3a64561b-2ba78f83-svmmirror' for volume open: Not found

2014-07-25T14:20:28.996Z cpu3:139464 opID=557b531c)Vol3: 768: Could not open device '3a64561b-2ba78f83-svmmirror' for volume open: Not found

2014-07-25T14:20:28.996Z cpu3:139464 opID=557b531c)FSS: 5092: No FS driver claimed device '3a64561b-2ba78f83-svmmirror': Not supported

2014-07-25T14:20:29.135Z cpu2:32815)ScsiDeviceIO: 2337: Cmd(0x412e87041fc0) 0x28, CmdSN 0x3cc0bd from world 139464 to dev "eui.00173800ef520028" failed H:0x1 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-07-25T14:20:29.135Z cpu2:139464 opID=557b531c)Vol3: 714: Couldn't read volume header from snap-31bcf621-52cebd31-c06b34fb-61a4-00215a9c05d0: Device is permanently unavailable

2014-07-25T14:20:29.135Z cpu2:139464 opID=557b531c)WARNING: Vol3: 1875: Failed to refresh FS 532e6f62-dfcf62a4-71fe-00215a9c0716 descriptor: Device is permanently unavailable

2014-07-25T14:20:29.135Z cpu2:32815)ScsiDeviceIO: 2337: Cmd(0x412e87041fc0) 0x28, CmdSN 0x3cc0be from world 139464 to dev "eui.00173800ef520028" failed H:0x1 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-07-25T14:20:29.135Z cpu2:139464 opID=557b531c)Vol3: 1925: Error refreshing PB resMeta: Device is permanently unavailable


vmkwarning.log

2014-07-25T01:29:26.518Z cpu26:23402477)WARNING: CBT: 2039: Unsupported ioctl 60

2014-07-25T01:29:26.518Z cpu26:23402477)WARNING: CBT: 2039: Unsupported ioctl 59

2014-07-25T01:29:26.523Z cpu18:11442392)WARNING: CBT: 2039: Unsupported ioctl 60

2014-07-25T01:29:26.523Z cpu18:11442392)WARNING: CBT: 2039: Unsupported ioctl 59

2014-07-25T01:29:26.536Z cpu19:23402471)WARNING: CBT: 2039: Unsupported ioctl 43

2014-07-25T01:29:26.537Z cpu19:23402471)WARNING: CBT: 2039: Unsupported ioctl 43

2014-07-25T13:54:03.081Z cpu3:139461 opID=f93369ef)WARNING: Vol3: 1875: Failed to refresh FS 532e6f62-dfcf62a4-71fe-00215a9c0716 descriptor: Device is permanently unavailable

2014-07-25T13:54:03.834Z cpu15:139461 opID=f93369ef)WARNING: Vol3: 1875: Failed to refresh FS 532e6f62-dfcf62a4-71fe-00215a9c0716 descriptor: Device is permanently unavailable

2014-07-25T13:58:07.062Z cpu10:23571420)WARNING: LinuxSignal: 180: Signal 20 has unimplemented cartel-level semantics (sending anyway)

2014-07-25T14:20:18.630Z cpu2:139896 opID=bfcdcbe0)WARNING: Vol3: 1875: Failed to refresh FS 532e6f62-dfcf62a4-71fe-00215a9c0716 descriptor: Device is permanently unavailable

2014-07-25T14:20:19.009Z cpu6:139896 opID=bfcdcbe0)WARNING: Vol3: 1875: Failed to refresh FS 532e6f62-dfcf62a4-71fe-00215a9c0716 descriptor: Device is permanently unavailable

2014-07-25T14:20:19.344Z cpu10:139896 opID=bfcdcbe0)WARNING: Vol3: 1875: Failed to refresh FS 532e6f62-dfcf62a4-71fe-00215a9c0716 descriptor: Device is permanently unavailable

2014-07-25T14:20:29.135Z cpu2:139464 opID=557b531c)WARNING: Vol3: 1875: Failed to refresh FS 532e6f62-dfcf62a4-71fe-00215a9c0716 descriptor: Device is permanently unavailable

2014-07-25T14:31:07.733Z cpu16:23576774)WARNING: LinuxSignal: 180: Signal 20 has unimplemented cartel-level semantics (sending anyway)

2014-07-25T14:34:51.099Z cpu1:23579627)WARNING: LinuxSignal: 180: Signal 20 has unimplemented cartel-level semantics (sending anyway)

2014-07-25T14:35:38.171Z cpu4:23579811)WARNING: LinuxSignal: 180: Signal 20 has unimplemented cartel-level semantics (sending anyway)



Please help me understanding above log snippet.


Reply
0 Kudos
2 Replies
RubyIvy
Enthusiast
Enthusiast

Hi,

Above message indicates that An unplanned PDL occured  when the storage device would have been unexpectedly unpresented from the storage array without the unmount and detach being executed on the ESXi host..This also happens if the LUN ran out space causing it to become inaccessible.

If you are unable to unmount the datastore , try the following steps:

  1. Confirm that there is no I/O queued to the LUN and that it is not a snapshot LUN.

For this all running virtual machines from the datastore must be powered off and unregistered from the vCenter Server.

2.Unpresent the LUN from the storage.

3.Rescan. This will unmount the LUN.

4,Represent from the storage.

5.Re-mount the LUN on the host(s).

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful.
Reply
0 Kudos
dhanarajramesh

can u check whether status of the lun is ON mode (ex esxcli storage core device list -d naa.. Staius= ON)  if not turn on it

Reply
0 Kudos