Datastore disappeared, still see device with VMFS partition (and others) on it. Can I recover or recreate the datastore?

[justin2li]

All-

This is my first post, so hopefully this is in the right place. If I need to move to another forum, please let me know. Any assistance is greatly appreciated.

ESXi 6.5

Host: Dell R810

Local Disks: Three 2tb WD Reds, , PERC H700, RAID 5

When I logged in today, I quickly realized that all of my VM's were down:

When I looked at my Datastores, they were gone:

My devices appear there, though:

When I click into the PERC H700 Device, the VMFS partition (where all of my VM's reside) is there:

I tried following the guide here, using the partedUtil command, but it's not working. Perhaps I'm getting the start and end blocks wrong, but everything I try gives me an error like below:

2018-05-15T22:12:17.301Z cpu4:70361)WARNING: Partition: 1732: naa.6782bcb025e14800228de60d0f4bd976: in-use partition 2 modification is not supported

2018-05-15T22:12:17.301Z cpu4:70361)BC: 2466: Failed to write (uncached) object 'naa.6782bcb025e14800228de60d0f4bd976': Read only

Below is the output from a few commands that might be helpful.

[root@localhost:/var/log] partedUtil getptbl /vmfs/devices/disks/naa.6782bcb025e14800228de60d0f4bd976

gpt

486267 255 63 7811891200

1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 128

5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

7 1032224 1257471 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0

8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

9 1843200 7086079 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0

2 7086080 15472639 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

3 15472640 7811891166 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

[root@localhost:/var/log] partedUtil getUsableSectors /vmfs/devices/disks/naa.6782bcb025e14800228de60d0f4bd976

34 7811891166

[root@localhost:/var/log] fdisk -l

***

*** The fdisk command is deprecated: fdisk does not handle GPT partitions.  Please use partedUtil

***

fdisk: device has more than 2^32 sectors, can't use all of them

Found valid GPT with protective MBR; using GPT

Disk /dev/disks/naa.6782bcb025e14800228de60d0f4bd976: 4294967295 sectors, 4095M

Logical sector size: 512

Disk identifier (GUID): 1c9d6251-63fd-40b9-bd1b-db12ce2d0383

Partition table holds up to 128 entries

First usable sector is 34, last usable sector is 7811891166

Number  Start (sector)    End (sector)  Size       Code  Name

   1              64            8191       4064K   0700

   2         7086080        15472639       4095M   0700

   3        15472640      7811891166       3717G   0700

   5            8224          520191        249M   0700

   6          520224         1032191        249M   0700

   7         1032224         1257471        109M   0700

   8         1257504         1843199        285M   0700

   9         1843200         7086079       2560M   0700

I'm really at a loss as to what to do. Any advice is greatly appreciated. If there is any other information that I can provide, I'd be happy to get it.

Thanks,

Justin

[continuum]

Hello
try to resignature the VMFS-volume from commandline - see knowledgebase.
If that does not work it may still be possible to extract the VMs manually.
It would help to see the output of
dd if=/dev/disks/naa.6782bcb025e14800228de60d0f4bd976:3   bs=1M count=1536 | gzip -c >  /tmp/thuli.1536.gz

If you provide a download for the file thuli.1536.gz I can check it.

Ulli

[justin2li]

continuum​

Thanks for the quick reply!

Regarding your suggestion of trying to resignature the VMFS-volume from cli, which knowledgebase are you referring to? If you are able to link it, I'd greatly appreciate it.

Also, below should be a downloadable link to the requested output (you'll have to click Download once on the page, as it won't auto-download):

Dropbox - thuli.1536.gz

Edit: Here is another link. Both appear to be giving me problems when extracting with 7-zip, but maybe you'll have better luck. For some reason, this one is much smaller too:

Dropbox - justin.1536.gz

Thank you!
Justin

[continuum]

Hi Justin
are you running out of free space in /tmp ?
Both dump files are too short.
I would suggest that we continue this discussion via skype. If possible call me directly - my skype name is "sanbarrow"
Ulli

edit: after a first quick check I think we can recover:
Base Windows Server 2016
ThulDev
WinServ2016Base
download-pc
media-host
media-pc
thuli-dc
Your Vcenter VM may be lost - but I cant say that for sure with the truncated dump-file

[continuum]

Here are 2 links regarding the "resignature" process.
Be careful - I am not sure wether resignaturing the volume will help - but I am sure that I will be able to extract most of your VMs manually.
So rather do nothing than trying anything you are not absolutely sure about.

Resignature a VMFS Datastore Copy
VMFS Resignaturing

By the way - if we have to extract the VMs manually you will need a large empty SATA-disk - or anything else we can use as a temporary datastore.
Ulli

[continuum · Accepted solution]

UPDATE:
unfortunately resignaturing the LUN failed.
Now we started to extract VM by VM using dd.
The first VM we extract was thin provisioned so I had to create a 3000 lines dd-script ...
Ulli

[justin2li]

continuum​ you are an absolute rockstar! I have no idea how you did what you did (I assume magic was involved), but I could not be more thankful. So far, we were able to recover my domain controller, my DB server and a file server. I went from scared out of my mind to excited in a matter of hours, plus I learned a lot of best practice for my environment . I really appreciate your patience when working with me on this!

As far as a solution to the problem (aside from magic), I think continuum​ would be able to better explain it.

Thank you so much again!

Justin

[justin2li]

continuum​ took it upon himself to reach back out to me to continue attempting to restore the other VM's, which were successful!

As I said before, I don't fully understand what he did, but he somehow found the beginning and end sectors on the drive, then was able to rebuild the vmdk files (some have snapshots), vmx files, vmsd files, etc by running a script that took all fragments of the VM and rebuilt the file. I really don't know how it worked, but I was able to watch the file size increase until it was the correct size of the VM. Once done, we were able to start up the machines. In a couple cases, the size of the VM was slightly off (2-3mb), so he was able to use the same script to fix those issues. Once that was done, we were able to start them up correctly.

I'm also thankful that he provided some guidance on backing up the systems and how I should rebuild it (since it badly needs it). Overall, I'm just thrilled that I got all of my important VM's and data back

[SSoliman365]

Should i do so regardless of the case? would it make anything worse if i tried so and it wasn't the solution ?

[spaniola]

Hello Sir,

I am having the same issue for a non profit catholic church. Could you help? I am not seeing any of our datastores just like this person.
Thank you.

[gysbertgroenewa]

Hi Continuum

I have the exact same problem, would you be able to assist me. Thanx in advance.

Thank You

Gys