ESXi 6.7 - Datastore lost. Is it recoverable?

[WWVM]

Summary: My ESXi server which has worked well for a few years suddenly experienced an issue. It is a NUC which has a single 512GB SSD with a large partition for the datastore. I have made no changes to the Host configuration recently.

I can run ESXi, but no VMs started I had 6, with some on autostart. . I can get into the web UI and look at storage, no datastores. I only see the disk and the partitions, which seem intact. If I look at VMs I see the 6 entries, no machine information, just numbers where the names used to be and Status Invalid. (I attached a couple of files showing this).

I tried to run the commend "esxcli storage vmfs snapshot mount -l "datastore1"" and no datastores were found. 

I need to restore my datastore I think but am stuck. I can get into the ESXi shell and run commands.

Any suggestions would be really, really, welcome!

[continuum]

Can you run the command
hexdump -C /dev/disks/<device-id-of-partition3> | less
and show us just the first few lines ?

Replace <device-id-of-partition3> with t10*****ED7:3

Ulli

[WWVM]

Thank you Ulli...

As there is no way to cut and paste between the systems, I have attached a photo. Hopefully you can make out the details. If not I can manually copy these over.

Thanks also for showing me the "****" shortcut! I was typing the ID manually before now!

[WWVM]

I feel it is in there somewhere! I ran

esxcli storage vmfs extent list

and saw the "datastore1"  volume etc.

[continuum]

try to resignature it

[WWVM]

OK - sorry but I am not really an admin, just a user...

Can you explain how to do that?

[WWVM]

(Discovered how to SSH in from my local machine :))

[WWVM]

OK so tried a few things. I always seem to get stopped when attempting to commit to the disk though. Here's my partition detail.

partedUtil getptbl /vmfs/devices/disks/t10****ED7
gpt
63841 255 63 1025610768
1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 128
5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
7 1032224 1257471 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0
8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
9 1843200 7086079 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0
2 7086080 15472639 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
3 15472640 1025610734 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

Sector info:

partedUtil getUsableSectors /vmfs/devices/disks/t10***ED7
34 1025610734

I tried this:

partedUtil mklabel /vmfs/devices/disks/t10****ED7 gpt

And got this:

Error: Read-only file system during write on /dev/disks/t10.ATA_____Crucial_CT525MX300SSD4__________________________170815FF2ED7
WriteNewPtable: Unable to commit to device /vmfs/devices/disks/t10.ATA_____Crucial_CT525MX300SSD4__________________________170815FF2ED7

At least if that command would have fried me, it seems to have not :):

esxcli storage vmfs extent list
Volume Name VMFS UUID Extent Number Device Name Partition
----------- ----------------------------------- ------------- ------------------------------------------------------------------------ ---------
datastore1 5ada56d0-7c8ca3d4-f802-94c691186ee2 0 t10.ATA_____Crucial_CT525MX300SSD4__________________________170815FF2ED7 3

[WWVM]

A lot of suggestions out there are based on using vSphere client. This is not a vCenter server, just a simple ESXi system, if that clarifies (or muddies) anything.

[continuum]

Stop trying to fix your partitiontable - that looks fine to me.
Do you have a second datastore ?
Can you add a SATA-disk to the server so that we can add a second datastore ?

Read https://communities.vmware.com/t5/VMware-vSphere-Documents/Create-a-VMFS-Header-dump-using-an-ESXi-H...

Without a second datastore you would need to create split dumps like this ...

dd if=/dev/disks/Device:partNum bs=1M count=500 skip=0 | gzip -c >  /tmp/split-vmfs-header-dump.0.gz
download /tmp/split-vmfs-header-dump.0.gz and clean up /tmp
dd if=/dev/disks/Device:partNum bs=1M count=500 skip=500 | gzip -c >  /tmp/split-vmfs-header-dump.500.gz
download /tmp/split-vmfs-header-dump.500.gz and clean up /tmp
dd if=/dev/disks/Device:partNum bs=1M count=500 skip=1000 | gzip -c >  /tmp/split-vmfs-header-dump.1000.gz
download /tmp/split-vmfs-header-dump.1000.gz and clean up /tmp
dd if=/dev/disks/Device:partNum bs=1M count=500 skip=1500 | gzip -c >  /tmp/split-vmfs-header-dump.1500.gz
download /tmp/split-vmfs-header-dump.1500.gz and clean up /tmp
dd if=/dev/disks/Device:partNum bs=1M count=500 skip=2000 | gzip -c >  /tmp/split-vmfs-header-dump.2000.gz

Anyway - I would recommend that you organize a second disk / datastore and then call me via skype - dont try any obscure dangerous commands here !!!

Ulli

[WWVM]

Thanks Ulli!

I have (after some surgery) got another disk and created a datastore. I don't want to call you in the middle of the night, where are you located? I am in west coast USA, UK-8, EUR-9 timezone.

[continuum]

If you have a second datastore you can create a one-piece 3gb dump.
I am located in germany - if you call within next hour I will be available ...

Ulli

[WWVM]

New datastore:

esxcli storage vmfs extent list
Volume Name VMFS UUID Extent Number Device Name Partition
----------- ----------------------------------- ------------- ------------------------------------------------------------------------- ---------
datastore2 61fd7a64-57fa0d14-5e84-f44d306d6940 0 t10.ATA_____ST1000LM024_HN2DM101MBB__________________S2SMJ9GD422386______ 9

[WWVM]

Ulli - many thanks for all your help last week. Although ultimately it seems the data may be lost, you definitely went the extra mile with me, and your status on this board is well deserved. I wish you health and happiness.