Hi all,
I built a new ESXi 6.5 with RAID6 local storage and configured host cache. I migrated VMs from old hardware server to this new built. The ESXi was frozen and responded. I rebooted the ESXi host. Afterward, the datastore is missing. The Host cache - datastore was full. My vCenter is also running on this new ESXi host.
I also attached VMFS-Header-dump file.
Please help me to restore the VMs. It would be sincerely appreciated.
Missing datastore01-raid6
vmkernel.log
_______________
2020-10-05T22:50:24.112Z cpu24:67260)Activating Jumpstart plugin rhttpproxy.
2020-10-05T22:50:24.349Z cpu0:67680)Res3: 2552: datastore01-raid6: Inconsistency detected for type 5 resource file metadata signature 0, version 0
2020-10-05T22:50:24.349Z cpu0:67680)Vol3: 3155: Failed to get object 28 type 1 uuid 5f7aec3f-bcd9772c-864b-f40270e0d568 FD 0 gen 0 :No filesystem on the device
2020-10-05T22:50:24.349Z cpu0:67680)WARNING: Fil3: 1389: Failed to reserve volume f532 28 1 5f7aec3f bcd9772c 2f4864b 68d5e070 0 0 0 0 0 0 0
2020-10-05T22:50:24.349Z cpu0:67680)Vol3: 3155: Failed to get object 28 type 2 uuid 5f7aec3f-bcd9772c-864b-f40270e0d568 FD 4 gen 1 :No filesystem on the device
2020-10-05T22:50:24.354Z cpu0:67680)Res3: 2552: datastore01-raid6: Inconsistency detected for type 5 resource file metadata signature 0, version 0
2020-10-05T22:50:24.354Z cpu0:67680)Vol3: 3155: Failed to get object 28 type 1 uuid 5f7aec3f-bcd9772c-864b-f40270e0d568 FD 0 gen 0 :No filesystem on the device
2020-10-05T22:50:24.354Z cpu0:67680)WARNING: Fil3: 1389: Failed to reserve volume f532 28 1 5f7aec3f bcd9772c 2f4864b 68d5e070 0 0 0 0 0 0 0
2020-10-05T22:50:24.354Z cpu0:67680)Vol3: 3155: Failed to get object 28 type 2 uuid 5f7aec3f-bcd9772c-864b-f40270e0d568 FD 4 gen 1 :No filesystem on the device
2020-10-05T22:50:24.359Z cpu0:67680)Res3: 2552: datastore01-raid6: Inconsistency detected for type 5 resource file metadata signature 0, version 0
2020-10-05T22:50:24.359Z cpu0:67680)Vol3: 3155: Failed to get object 28 type 1 uuid 5f7aec3f-bcd9772c-864b-f40270e0d568 FD 0 gen 0 :No filesystem on the device
:
[root@nyesxi65:~] esxcfg-scsidevs -m
VmFileSystem: SlowRefresh() failed: Cannot open volume: /vmfs/volumes/5f7aec3f-bcd9772c-864b-f40270e0d568
naa.62cea7f066bb9e00270d963cf8ff3fcf:3 /vmfs/devices/disks/naa.62cea7f066bb9e00270d963cf8ff3fcf:3 5f7ae56d-fc1a455c-1fc1-f40270e0d568 0 datastore03-donot-used
naa.62cea7f066bb9e00270dacb6fee7b3a8:1 /vmfs/devices/disks/naa.62cea7f066bb9e00270dacb6fee7b3a8:1 5f7af189-6c4748c6-fcff-f40270e0d568 0 datastore02-host-cache
naa.62cea7f066bb9e00270d965e2b5c4e51:1 /vmfs/devices/disks/naa.62cea7f066bb9e00270d965e2b5c4e51:1 5f7aec3f-bcd9772c-864b-f40270e0d568 0 datastore01-raid6
[root@nyesxi65:~] voma -m vmfs -d /vmfs/devices/disks/naa.62cea7f066bb9e00270d965e2b5c4e51:1
Checking if device is actively used by other hosts
Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs).
Running VMFS Checker version 2.1 in default mode
Initializing LVM metadata, Basic Checks will be done
Phase 1: Checking VMFS header and resource files
Detected VMFS-6 file system (labeled:'datastore01-raid6') with UUID:5f7aec3f-bcd9772c-864b-f40270e0d568, Version 6:81
ON-DISK ERROR: Unexpected 'resourcesPerCluster' 0, expected 32
ON-DISK ERROR: Unexpected 'clustersPerGroup' 0, expected 8
ON-DISK ERROR: Unexpected 'clusterGroupOffset' 0, expected 65536
ON-DISK ERROR: Unexpected 'resourceSize' 0, expected 65536
ON-DISK ERROR: Unexpected 'clusterGroupSize' 0, expected 16842752
ON-DISK ERROR: Unexpected 'bitsPerResource' 0, expected 1
ON-DISK ERROR: Unexpected 'version' 0, expected 1
ON-DISK ERROR: Unexpected 'signature' 0, expected 1919315300
ON-DISK ERROR: Unexpected 'numAffinityInfoPerRC' 0, expected 32
ON-DISK ERROR: Unexpected 'numAffinityInfoPerRsrc' 0, expected 1
ON-DISK ERROR: Unexpected 'clustersPerGroup' 0, expected 8
ON-DISK ERROR: Unexpected 'clusterGroupOffset' 0, expected 65536
ON-DISK ERROR: Unexpected 'resourceSize' 0, expected 65536
ON-DISK ERROR: Unexpected 'clusterGroupSize' 0, expected 65536
ON-DISK ERROR: Unexpected 'bitsPerResource' 0, expected 1
ON-DISK ERROR: Unexpected 'version' 0, expected 1
ON-DISK ERROR: Unexpected 'signature' 0, expected 1919315300
ON-DISK ERROR: Unexpected 'numAffinityInfoPerRsrc' 0, expected 1
ON-DISK ERROR: <FD c0 r4>: invalid address PB2 cow 0 cnum 0 rnum 0
ON-DISK ERROR: Corruption too severe in resource file [SB]
ERROR: Failed to check sbc.sf.
VOMA failed to check device : Invalid address
Total Errors Found: 20
Kindly Consult VMware Support for further assistance
Best regards to all
Hi,
VOMA is reporting on disk errors and it does not look good. Should consider raising a ticket with VMware so they can collect datastore dumps and attempt to fix the errors.
Hi
this does not look good - not at all.
It actually looks like a raid rebuild gone wrong: vmdk-descriptorfiles appear in one piece but larger files like vmx-files for example are already broken into fragments - which should not happen with a healthy volume.
Results are quite strange - the heartbeat section (4 mbs) for example looks like it appears in one piece ....
Apparently your vmdk files are not thin provisioned - so you have a better than zero chance to eventually carve out flat.vmdks.
This vmdk-files still have named entries :
itcweb01-flat.vmdk
itcweb01_1-flat.vmdk
nyvcsa65-flat.vmdk
nyvcsa65_1-flat.vmdk
nyvcsa65_2-flat.vmdk
nyvcsa65_3-flat.vmdk
nyvcsa65_4-flat.vmdk
nyvcsa65_5-flat.vmdk
nyvcsa65_6-flat.vmdk
nyvcsa65_7-flat.vmdk
nyvcsa65_8-flat.vmdk
nyvcsa65_9-flat.vmdk
nyvcsa65_10-flat.vmdk
nyvcsa65_11-flat.vmdk
nyvcsa65_7-flat.vmdk
nyvcsa65_8-flat.vmdk
nyvcsa65_9-flat.vmdk
nyvcsa65_10-flat.vmdk
nyvcsa65_11-flat.vmdk
zzz_ArgosPBX-flat.vmdk
AGMail01-flat.vmdk
SqlServer-flat.vmdk
SqlServer_1-flat.vmdk
Usually I would consider those vmdks as recoverable - but I dont think that applies to this case - as even comparably small files like vmx-files are incomplete.
I assume that your best chance is an attempt to carve out flat.vmdks
Ulli
Hi Ulli,
Can you help to see if I can restore any VM on this? I sent you chat via Skype.
Thank you!
Thank you!