VMware Cloud Community
mcherry8978
Contributor
Contributor

Is my ESXi 6.0 datastore lost?

I'll start off by apologizing for the long post.  There's a lot of information that may be of no assistance to the issue at hand, but I figure it's best to paint as complete a picture as possible.

I have an ESXi 6.0 server with an attached RAID that houses my primary datastore.  Earlier this week the array had a failed disk, so I replaced it, rebuilt, and rebooted the server.  ESXi booted but none of my virtual machines started.  I logged into the console and found that the datastore appeared to be broken (the symbolic link in /vmfs/volumes/ was displayed in red and pointed to a seemingly non-existent uuid).

So, I hit the googles and found a suggestion that I check the partition table, and that's where I hit the first snag.  I ran the following command:

# partedUtil getptbl /vmfs/devices/disks/naa.6001e6775e8220001a267f61499bf7bd

and got the following output:

Error: The primary GPT table is corrupt, but the backup appears OK, so that will be used...

Following the instructions at How to use partedUtil to recover damaged VMFS5 (GPT) partition table - VMwaremine - Artur Krzywdzins... I was able to repair the partition table, but that did not fix the issue.  What it did do is cause the datastore symlink in /vmfs/volumes/ to disappear.  Oh crap!

After much more googling I found help at Recreating a missing VMFS datastore partition in VMware vSphere 5.0/5.1/5.5 (2046610) | VMware KB .  Following these steps I was able to get the symlink back but it still appears to be broken.

I can see the drive in the web console and it recognizes the VMFS partition.

esxi1.png

When I do a rescan I get the following output in /var/log/vmkernel.log:

2017-05-11T06:58:08.925Z cpu14:33255)NMP: nmp_ThrottleLogForDevice:3298: Cmd 0x1a (0x43a5c95de340, 0) to dev "naa.6001e6775e8220001a267f61499bf7bd" on path "vmhba1:C2:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE

2017-05-11T06:58:08.925Z cpu14:33255)ScsiDeviceIO: 2651: Cmd(0x43a5c95de340) 0x1a, CmdSN 0x1edd from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:09.134Z cpu14:32835)ScsiDeviceIO: 2651: Cmd(0x43a5c95de340) 0x1a, CmdSN 0x1eef from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:09.139Z cpu14:32821)ScsiDeviceIO: 2651: Cmd(0x43a5c95de340) 0x1a, CmdSN 0x1f13 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:09.143Z cpu14:32821)ScsiDeviceIO: 2651: Cmd(0x43a5c936a680) 0x1a, CmdSN 0x1f31 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:09.346Z cpu14:33020)ScsiDeviceIO: 2651: Cmd(0x43a5c936bb80) 0x1a, CmdSN 0x1f37 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:09.615Z cpu14:32805)ScsiDeviceIO: 2651: Cmd(0x43a5c9569280) 0x1a, CmdSN 0x1f3d from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:09.615Z cpu14:32805)ScsiDeviceIO: 2651: Cmd(0x43a5c9568f80) 0x1a, CmdSN 0x1f43 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:09.841Z cpu17:34779 opID=cfe8392e)World: 15516: VC opID d8eecbba maps to vmkernel opID cfe8392e

2017-05-11T06:58:09.841Z cpu17:34779 opID=cfe8392e)VC: 3551: Device rescan time 217 msec (total number of devices 6)

2017-05-11T06:58:09.841Z cpu17:34779 opID=cfe8392e)VC: 3554: Filesystem probe time 698 msec (devices probed 4 of 6)

2017-05-11T06:58:09.841Z cpu17:34779 opID=cfe8392e)VC: 3556: Refresh open volume time 0 msec

2017-05-11T06:58:09.858Z cpu19:33409)ScsiDeviceIO: 2651: Cmd(0x43a5c95c24c0) 0x1a, CmdSN 0x1fa3 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:09.858Z cpu19:33409)ScsiDeviceIO: 2651: Cmd(0x43a5c95c21c0) 0x1a, CmdSN 0x1fa9 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:09.876Z cpu19:33409)NMP: nmp_ThrottleLogForDevice:3231: last error status from device naa.6001e6775e8220001a267f61499bf7bd repeated 10 times

2017-05-11T06:58:09.876Z cpu19:33409)ScsiDeviceIO: 2651: Cmd(0x43a5c93e5380) 0x1a, CmdSN 0x1faf from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:09.876Z cpu19:33409)ScsiDeviceIO: 2651: Cmd(0x43a5c07ce340) 0x1a, CmdSN 0x1fb5 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:10.045Z cpu17:34779 opID=cfe8392e)WARNING: Fil3: 4226: Unexpected object type 1

2017-05-11T06:58:10.045Z cpu17:34779 opID=cfe8392e)Vol3: 3147: Failed to get object 28 type 1 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 0 gen 0 :Bad parameter

2017-05-11T06:58:10.045Z cpu17:34779 opID=cfe8392e)WARNING: Fil3: 2469: Failed to reserve volume f530 28 1 58b8e8fc 786af50a 1e00eb19 15ce4d67 0 0 0 0 0 0 0

2017-05-11T06:58:10.045Z cpu17:34779 opID=cfe8392e)Vol3: 3147: Failed to get object 28 type 2 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 4 gen 1 :Bad parameter

2017-05-11T06:58:10.046Z cpu19:33595)ScsiDeviceIO: 2651: Cmd(0x43a5c945b7c0) 0x1a, CmdSN 0x1fbb from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:10.047Z cpu19:33595)ScsiDeviceIO: 2651: Cmd(0x43a5c945b4c0) 0x1a, CmdSN 0x1fc1 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:10.064Z cpu19:32792)ScsiDeviceIO: 2651: Cmd(0x43a5c07c5c40) 0x1a, CmdSN 0x1fc7 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:10.065Z cpu19:32792)ScsiDeviceIO: 2651: Cmd(0x43a5c0800580) 0x1a, CmdSN 0x1fcd from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:10.082Z cpu17:34779 opID=cfe8392e)WARNING: Fil3: 4226: Unexpected object type 1

2017-05-11T06:58:10.082Z cpu17:34779 opID=cfe8392e)Vol3: 3147: Failed to get object 28 type 1 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 0 gen 0 :Bad parameter

2017-05-11T06:58:10.082Z cpu17:34779 opID=cfe8392e)WARNING: Fil3: 2469: Failed to reserve volume f530 28 1 58b8e8fc 786af50a 1e00eb19 15ce4d67 0 0 0 0 0 0 0

2017-05-11T06:58:10.082Z cpu17:34779 opID=cfe8392e)Vol3: 3147: Failed to get object 28 type 2 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 4 gen 1 :Bad parameter

2017-05-11T06:58:10.083Z cpu19:32792)ScsiDeviceIO: 2651: Cmd(0x43a5c07b9600) 0x1a, CmdSN 0x1fd3 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:10.083Z cpu19:32792)ScsiDeviceIO: 2651: Cmd(0x43a5c07b9300) 0x1a, CmdSN 0x1fd9 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:10.101Z cpu19:32792)ScsiDeviceIO: 2651: Cmd(0x43a5c07fca00) 0x1a, CmdSN 0x1fdf from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:10.101Z cpu19:32792)ScsiDeviceIO: 2651: Cmd(0x43a5c0844a00) 0x1a, CmdSN 0x1fe5 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:10.118Z cpu17:34779 opID=cfe8392e)WARNING: Fil3: 4226: Unexpected object type 1

2017-05-11T06:58:10.118Z cpu17:34779 opID=cfe8392e)Vol3: 3147: Failed to get object 28 type 1 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 0 gen 0 :Bad parameter

2017-05-11T06:58:10.118Z cpu17:34779 opID=cfe8392e)WARNING: Fil3: 2469: Failed to reserve volume f530 28 1 58b8e8fc 786af50a 1e00eb19 15ce4d67 0 0 0 0 0 0 0

2017-05-11T06:58:10.118Z cpu17:34779 opID=cfe8392e)Vol3: 3147: Failed to get object 28 type 2 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 4 gen 1 :Bad parameter

2017-05-11T06:58:10.120Z cpu19:32792)ScsiDeviceIO: 2651: Cmd(0x43a5c07a6b40) 0x1a, CmdSN 0x1feb from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:10.120Z cpu19:32792)ScsiDeviceIO: 2651: Cmd(0x43a5c07a6840) 0x1a, CmdSN 0x1ff1 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:10.137Z cpu19:32792)ScsiDeviceIO: 2651: Cmd(0x43a5c07bc780) 0x1a, CmdSN 0x1ff7 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:10.138Z cpu19:32792)ScsiDeviceIO: 2651: Cmd(0x43a5c07bca80) 0x1a, CmdSN 0x1ffd from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:10.155Z cpu17:34779 opID=cfe8392e)WARNING: Fil3: 4226: Unexpected object type 1

2017-05-11T06:58:10.155Z cpu17:34779 opID=cfe8392e)Vol3: 3147: Failed to get object 28 type 1 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 0 gen 0 :Bad parameter

2017-05-11T06:58:10.155Z cpu17:34779 opID=cfe8392e)WARNING: Fil3: 2469: Failed to reserve volume f530 28 1 58b8e8fc 786af50a 1e00eb19 15ce4d67 0 0 0 0 0 0 0

2017-05-11T06:58:10.155Z cpu17:34779 opID=cfe8392e)Vol3: 3147: Failed to get object 28 type 2 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 4 gen 1 :Bad parameter

2017-05-11T06:58:10.157Z cpu19:33409)ScsiDeviceIO: 2651: Cmd(0x43a5c07faf00) 0x1a, CmdSN 0x2003 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:10.157Z cpu19:33409)ScsiDeviceIO: 2651: Cmd(0x43a5c07fac00) 0x1a, CmdSN 0x2009 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:10.174Z cpu19:33409)ScsiDeviceIO: 2651: Cmd(0x43a5c07a6cc0) 0x1a, CmdSN 0x200f from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:10.175Z cpu19:33409)ScsiDeviceIO: 2651: Cmd(0x43a5c07a8040) 0x1a, CmdSN 0x2015 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:10.193Z cpu17:34779 opID=cfe8392e)WARNING: Fil3: 4226: Unexpected object type 1

2017-05-11T06:58:10.193Z cpu17:34779 opID=cfe8392e)Vol3: 3147: Failed to get object 28 type 1 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 0 gen 0 :Bad parameter

2017-05-11T06:58:10.193Z cpu17:34779 opID=cfe8392e)WARNING: Fil3: 2469: Failed to reserve volume f530 28 1 58b8e8fc 786af50a 1e00eb19 15ce4d67 0 0 0 0 0 0 0

2017-05-11T06:58:10.193Z cpu17:34779 opID=cfe8392e)Vol3: 3147: Failed to get object 28 type 2 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 4 gen 1 :Bad parameter

2017-05-11T06:58:24.860Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc91c3e40) 0x1a, CmdSN 0x201b from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:24.885Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc91c21c0) 0x1a, CmdSN 0x2021 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:24.938Z cpu7:33019)NMP: nmp_ThrottleLogForDevice:3231: last error status from device naa.6001e6775e8220001a267f61499bf7bd repeated 20 times

2017-05-11T06:58:24.938Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc921eb80) 0x1a, CmdSN 0x2027 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:24.939Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc921ed00) 0x1a, CmdSN 0x202d from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:24.963Z cpu0:34777)WARNING: Fil3: 4226: Unexpected object type 1

2017-05-11T06:58:24.963Z cpu0:34777)Vol3: 3147: Failed to get object 28 type 1 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 0 gen 0 :Bad parameter

2017-05-11T06:58:24.963Z cpu0:34777)WARNING: Fil3: 2469: Failed to reserve volume f530 28 1 58b8e8fc 786af50a 1e00eb19 15ce4d67 0 0 0 0 0 0 0

2017-05-11T06:58:24.963Z cpu0:34777)Vol3: 3147: Failed to get object 28 type 2 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 4 gen 1 :Bad parameter

2017-05-11T06:58:24.964Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc92330c0) 0x1a, CmdSN 0x2033 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:24.965Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc92333c0) 0x1a, CmdSN 0x2039 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:24.982Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc0812600) 0x1a, CmdSN 0x203f from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:24.983Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc0812300) 0x1a, CmdSN 0x2045 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:25.000Z cpu0:34777)WARNING: Fil3: 4226: Unexpected object type 1

2017-05-11T06:58:25.000Z cpu0:34777)Vol3: 3147: Failed to get object 28 type 1 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 0 gen 0 :Bad parameter

2017-05-11T06:58:25.000Z cpu0:34777)WARNING: Fil3: 2469: Failed to reserve volume f530 28 1 58b8e8fc 786af50a 1e00eb19 15ce4d67 0 0 0 0 0 0 0

2017-05-11T06:58:25.000Z cpu0:34777)Vol3: 3147: Failed to get object 28 type 2 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 4 gen 1 :Bad parameter

2017-05-11T06:58:25.000Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc919c140) 0x1a, CmdSN 0x204b from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:25.001Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc93dd780) 0x1a, CmdSN 0x2051 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:25.018Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc931abc0) 0x1a, CmdSN 0x2057 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:25.019Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc931b040) 0x1a, CmdSN 0x205d from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:25.036Z cpu0:34777)WARNING: Fil3: 4226: Unexpected object type 1

2017-05-11T06:58:25.036Z cpu0:34777)Vol3: 3147: Failed to get object 28 type 1 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 0 gen 0 :Bad parameter

2017-05-11T06:58:25.036Z cpu0:34777)WARNING: Fil3: 2469: Failed to reserve volume f530 28 1 58b8e8fc 786af50a 1e00eb19 15ce4d67 0 0 0 0 0 0 0

2017-05-11T06:58:25.036Z cpu0:34777)Vol3: 3147: Failed to get object 28 type 2 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 4 gen 1 :Bad parameter

2017-05-11T06:58:25.037Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc07e9180) 0x1a, CmdSN 0x2063 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:25.038Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc07cbb00) 0x1a, CmdSN 0x2069 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:25.055Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc9881a40) 0x1a, CmdSN 0x206f from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:25.055Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc988ba00) 0x1a, CmdSN 0x2075 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:25.072Z cpu0:34777)WARNING: Fil3: 4226: Unexpected object type 1

2017-05-11T06:58:25.072Z cpu0:34777)Vol3: 3147: Failed to get object 28 type 1 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 0 gen 0 :Bad parameter

2017-05-11T06:58:25.072Z cpu0:34777)WARNING: Fil3: 2469: Failed to reserve volume f530 28 1 58b8e8fc 786af50a 1e00eb19 15ce4d67 0 0 0 0 0 0 0

2017-05-11T06:58:25.072Z cpu0:34777)Vol3: 3147: Failed to get object 28 type 2 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 4 gen 1 :Bad parameter

2017-05-11T06:58:25.073Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc07be980) 0x1a, CmdSN 0x207b from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:25.074Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc07be680) 0x1a, CmdSN 0x2081 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:25.091Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc9336b80) 0x1a, CmdSN 0x2087 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:25.092Z cpu7:33019)ScsiDeviceIO: 2651: Cmd(0x439dc9336e80) 0x1a, CmdSN 0x208d from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T06:58:25.108Z cpu0:34777)WARNING: Fil3: 4226: Unexpected object type 1

2017-05-11T06:58:25.108Z cpu0:34777)Vol3: 3147: Failed to get object 28 type 1 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 0 gen 0 :Bad parameter

2017-05-11T06:58:25.108Z cpu0:34777)WARNING: Fil3: 2469: Failed to reserve volume f530 28 1 58b8e8fc 786af50a 1e00eb19 15ce4d67 0 0 0 0 0 0 0

2017-05-11T06:58:25.108Z cpu0:34777)Vol3: 3147: Failed to get object 28 type 2 uuid 58b8e8fc-786af50a-eb19-001e674dce15 FD 4 gen 1 :Bad parameter

2017-05-11T06:59:01.350Z cpu5:33293)NMP: nmp_ResetDeviceLogThrottling:3349: last error status from device naa.6001e6775e8220001a267f61499bf7bd repeated 17 times

2017-05-11T07:09:05.868Z cpu14:32792)NMP: nmp_ThrottleLogForDevice:3298: Cmd 0x1a (0x43a5c93e1480, 0) to dev "naa.6001e6775e8220001a267f61499bf7bd" on path "vmhba1:C2:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE

2017-05-11T07:09:06.082Z cpu14:32868)ScsiDeviceIO: 2651: Cmd(0x43a5c93e1480) 0x1a, CmdSN 0x20a6 from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T07:09:06.082Z cpu14:32868)NMP: nmp_ThrottleLogForDevice:3298: Cmd 0x85 (0x43a5c93e1480, 34509) to dev "naa.6001e6775e8220001a267f61499bf7bd" on path "vmhba1:C2:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2017-05-11T07:09:06.082Z cpu14:32868)ScsiDeviceIO: 2651: Cmd(0x43a5c93e1480) 0x4d, CmdSN 0xb from world 34509 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2017-05-11T07:09:06.082Z cpu14:32868)NMP: nmp_ThrottleLogForDevice:3298: Cmd 0x1a (0x43a5c93e1480, 34509) to dev "naa.6001e6775e8220001a267f61499bf7bd" on path "vmhba1:C2:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE

2017-05-11T07:09:06.082Z cpu14:32868)ScsiDeviceIO: 2651: Cmd(0x43a5c93e1480) 0x1a, CmdSN 0xc from world 34509 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T07:09:06.083Z cpu14:32868)ScsiDeviceIO: 2651: Cmd(0x43a5c93e1480) 0x1a, CmdSN 0x20ac from world 0 to dev "naa.6001e6775e8220001a267f61499bf7bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2017-05-11T07:09:06.083Z cpu14:32868)NMP: nmp_ThrottleLogForDevice:3298: Cmd 0x85 (0x43a5c93e1480, 34509) to dev "naa.6001e6775e8220001a267f61499bf7bd" on path "vmhba1:C2:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

Note the high frequency of "Warning" and "Failed".  Unfortunately the messages are a bit over my head and searching has yielded few results, none-of-which seem to be applicable to my case.

So, I'm slowly coming to terms with the fact that my array may be corrupt / lost / beyond repair, but I thought I'd hit the community boards before I gave up and started over.  This is a home lab so it's not the end of the world, but it would be nice to not have to start from scratch (yes, I have the pertinent data backed up).

Any help would be greatly appreciated!

Reply
0 Kudos
3 Replies
continuum
Immortal
Immortal

Read Collecting and applying raw metadata dumps on VMFS volumes using DD (Data Description) (1020645) | V...
If you provide a download for the dump I can tell you if there is still a chance to recover your data.
Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
mcherry8978
Contributor
Contributor

Thanks, Ulli.  I created an image and uploaded to naa.6001e6775e8220001a267f61499bf7bd.1500.bin - Google Drive .  Hopefully this holds some good news.  Please let me know if you need anything else to help diagnose.

Reply
0 Kudos
continuum
Immortal
Immortal

Hi Mike

I am not sure what happened exactly: but I doubt that  your attempts to fix the filesystem acchieved anything good.
Looks like the volume was either formatted with NTFS or the partitiontable recreation went completely wrong.
Here is a short table of the VMFS-content that I think may eventually still be present.
I highly recommend to stop any further attempts of recovering anything - and call me.

Ulli

machines/cyclopia/

cyclopia0-000001-delta.vmdk

cyclopia0-000001.vmdk

cyclopia0.vmdk

machines/infosphere/

infosphere0-000003-delta.vmdk

infosphere1-000002-sesparse.vmdk

machines/mars/

mars0-000001-delta.vmdk

mars0-000001.vmdk

machines/omega3/

omega30-000001-delta.vmdk

omega30-000001.vmdk

omega30-flat.vmdk

omega30.vmdk

machines/planex/

planex-000001-flat.vmdk

planex-000001.vmdk

planex-000002-flat.vmdk

planex-000002.vmdk

planex-000003-delta.vmdk

planex-000003.vmdk

planex-000004-delta.vmdk

planex-000004.vmdk

planex-000006-delta.vmdk

planex-000006.vmdk

planex-flat.vmdk

planex.vmdk

machines/robonia/

robonia-000003.vmdk~

robonia0-000001-delta.vmdk

robonia0-000001.vmdk

robonia0-000002-delta.vmdk

robonia0-000002.vmdk

robonia0-flat.vmdk

robonia0.vmdk

machines/wormulon/

wormulon-flat.vmdk

wormulon.vmdk

wormulon0-000001-delta.vmdk

wormulon0-000001.vmdk

wormulon0-000003.vmdk

wormulon0.vmdk

wormulon1-000001-delta.vmdk

wormulon1-000001.vmdk

wormulon1-000003-delta.vmdk

wormulon1-000003.vmdk

wormulon1-flat.vmdk

wormulon1.vmdk

machines/zsbase/

zsbase0-flat.vmdk

zsbase0.vmdk

zsbase-flat.vmdk


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos