VMware Cloud Community
labvm20
Contributor
Contributor

Datastore is missing after reboot

Hi all,

I built a new ESXi 6.5 with RAID6 local storage and configured host cache.  I migrated VMs from old hardware server to this new built. The ESXi was frozen and responded.  I rebooted the ESXi host.  Afterward, the datastore is missing.  The Host cache - datastore was full.  My vCenter is also running on this new ESXi host.

I also attached VMFS-Header-dump file.

Please help me to restore the VMs.  It would be sincerely appreciated.

Missing datastore01-raid6

pastedImage_1.png

pastedImage_0.png

vmkernel.log

_______________

2020-10-05T22:50:24.112Z cpu24:67260)Activating Jumpstart plugin rhttpproxy.

2020-10-05T22:50:24.349Z cpu0:67680)Res3: 2552: datastore01-raid6: Inconsistency detected for type 5 resource file metadata signature 0, version 0

2020-10-05T22:50:24.349Z cpu0:67680)Vol3: 3155: Failed to get object 28 type 1 uuid 5f7aec3f-bcd9772c-864b-f40270e0d568 FD 0 gen 0 :No filesystem on the device

2020-10-05T22:50:24.349Z cpu0:67680)WARNING: Fil3: 1389: Failed to reserve volume f532 28 1 5f7aec3f bcd9772c 2f4864b 68d5e070 0 0 0 0 0 0 0

2020-10-05T22:50:24.349Z cpu0:67680)Vol3: 3155: Failed to get object 28 type 2 uuid 5f7aec3f-bcd9772c-864b-f40270e0d568 FD 4 gen 1 :No filesystem on the device

2020-10-05T22:50:24.354Z cpu0:67680)Res3: 2552: datastore01-raid6: Inconsistency detected for type 5 resource file metadata signature 0, version 0

2020-10-05T22:50:24.354Z cpu0:67680)Vol3: 3155: Failed to get object 28 type 1 uuid 5f7aec3f-bcd9772c-864b-f40270e0d568 FD 0 gen 0 :No filesystem on the device

2020-10-05T22:50:24.354Z cpu0:67680)WARNING: Fil3: 1389: Failed to reserve volume f532 28 1 5f7aec3f bcd9772c 2f4864b 68d5e070 0 0 0 0 0 0 0

2020-10-05T22:50:24.354Z cpu0:67680)Vol3: 3155: Failed to get object 28 type 2 uuid 5f7aec3f-bcd9772c-864b-f40270e0d568 FD 4 gen 1 :No filesystem on the device

2020-10-05T22:50:24.359Z cpu0:67680)Res3: 2552: datastore01-raid6: Inconsistency detected for type 5 resource file metadata signature 0, version 0

2020-10-05T22:50:24.359Z cpu0:67680)Vol3: 3155: Failed to get object 28 type 1 uuid 5f7aec3f-bcd9772c-864b-f40270e0d568 FD 0 gen 0 :No filesystem on the device

:

[root@nyesxi65:~] esxcfg-scsidevs -m

VmFileSystem: SlowRefresh() failed: Cannot open volume: /vmfs/volumes/5f7aec3f-bcd9772c-864b-f40270e0d568

naa.62cea7f066bb9e00270d963cf8ff3fcf:3                           /vmfs/devices/disks/naa.62cea7f066bb9e00270d963cf8ff3fcf:3 5f7ae56d-fc1a455c-1fc1-f40270e0d568  0  datastore03-donot-used

naa.62cea7f066bb9e00270dacb6fee7b3a8:1                           /vmfs/devices/disks/naa.62cea7f066bb9e00270dacb6fee7b3a8:1 5f7af189-6c4748c6-fcff-f40270e0d568  0  datastore02-host-cache

naa.62cea7f066bb9e00270d965e2b5c4e51:1                           /vmfs/devices/disks/naa.62cea7f066bb9e00270d965e2b5c4e51:1 5f7aec3f-bcd9772c-864b-f40270e0d568  0  datastore01-raid6

[root@nyesxi65:~] voma -m vmfs -d  /vmfs/devices/disks/naa.62cea7f066bb9e00270d965e2b5c4e51:1

Checking if device is actively used by other hosts

Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs).

Running VMFS Checker version 2.1 in default mode

Initializing LVM metadata, Basic Checks will be done

Phase 1: Checking VMFS header and resource files

   Detected VMFS-6 file system (labeled:'datastore01-raid6') with UUID:5f7aec3f-bcd9772c-864b-f40270e0d568, Version 6:81

ON-DISK ERROR: Unexpected 'resourcesPerCluster' 0, expected 32

ON-DISK ERROR: Unexpected 'clustersPerGroup' 0, expected 8

ON-DISK ERROR: Unexpected 'clusterGroupOffset' 0, expected 65536

ON-DISK ERROR: Unexpected 'resourceSize' 0, expected 65536

ON-DISK ERROR: Unexpected 'clusterGroupSize' 0, expected 16842752

ON-DISK ERROR: Unexpected 'bitsPerResource' 0, expected 1

ON-DISK ERROR: Unexpected 'version' 0, expected 1

ON-DISK ERROR: Unexpected 'signature' 0, expected 1919315300

ON-DISK ERROR: Unexpected 'numAffinityInfoPerRC' 0, expected 32

ON-DISK ERROR: Unexpected 'numAffinityInfoPerRsrc' 0, expected 1

ON-DISK ERROR: Unexpected 'clustersPerGroup' 0, expected 8

ON-DISK ERROR: Unexpected 'clusterGroupOffset' 0, expected 65536

ON-DISK ERROR: Unexpected 'resourceSize' 0, expected 65536

ON-DISK ERROR: Unexpected 'clusterGroupSize' 0, expected 65536

ON-DISK ERROR: Unexpected 'bitsPerResource' 0, expected 1

ON-DISK ERROR: Unexpected 'version' 0, expected 1

ON-DISK ERROR: Unexpected 'signature' 0, expected 1919315300

ON-DISK ERROR: Unexpected 'numAffinityInfoPerRsrc' 0, expected 1

ON-DISK ERROR: <FD c0 r4>: invalid address PB2 cow 0 cnum 0 rnum 0

ON-DISK ERROR: Corruption too severe in resource file [SB]

         ERROR: Failed to check sbc.sf.

   VOMA failed to check device : Invalid address

Total Errors Found:           20

   Kindly Consult VMware Support for further assistance

Best regards to all

0 Kudos
4 Replies
ashilkrishnan
VMware Employee
VMware Employee

Hi,

VOMA is reporting on disk errors and it does not look good. Should consider raising a ticket with VMware so they can collect datastore dumps and attempt to fix the errors.

0 Kudos
continuum
Immortal
Immortal

Hi

this does not look good - not at all.

It actually looks like a raid rebuild gone wrong: vmdk-descriptorfiles appear in one piece but larger files like vmx-files for example are already broken into fragments - which should not happen with a healthy volume.

Results are quite strange - the heartbeat section (4 mbs) for example looks like it appears in one piece ....

Apparently your vmdk files are not thin provisioned - so you have a better than zero chance to eventually carve out flat.vmdks.

This vmdk-files still have named entries :

itcweb01-flat.vmdk

itcweb01_1-flat.vmdk

nyvcsa65-flat.vmdk

nyvcsa65_1-flat.vmdk

nyvcsa65_2-flat.vmdk

nyvcsa65_3-flat.vmdk

nyvcsa65_4-flat.vmdk

nyvcsa65_5-flat.vmdk

nyvcsa65_6-flat.vmdk

nyvcsa65_7-flat.vmdk

nyvcsa65_8-flat.vmdk

nyvcsa65_9-flat.vmdk

nyvcsa65_10-flat.vmdk

nyvcsa65_11-flat.vmdk

nyvcsa65_7-flat.vmdk

nyvcsa65_8-flat.vmdk

nyvcsa65_9-flat.vmdk

nyvcsa65_10-flat.vmdk

nyvcsa65_11-flat.vmdk

zzz_ArgosPBX-flat.vmdk

AGMail01-flat.vmdk

SqlServer-flat.vmdk

SqlServer_1-flat.vmdk

Usually I would consider those vmdks as recoverable - but I dont think that applies to this case - as even comparably small files like vmx-files are incomplete.

I assume that your best chance is an attempt to carve out flat.vmdks

Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

0 Kudos
labvm20
Contributor
Contributor

Hi Ulli,

Can you help to see if I can restore any VM on this?  I sent you chat via Skype.

Thank you!

0 Kudos
labvm20
Contributor
Contributor

Thank you!

0 Kudos