ESXI 6.5 VM Recovery (missing vSAN vmdk)

[afikzach]

hello i'm having a similar issue i have VM which got corrupted while migrating in vsan from one host to another, vmdk number 10 chain got broken.

when i'm trying to recreate it using the vmkfstools, i'm facing an issue i'm not able to receive the correct size using "ls -l " it dos work....

ls -l  "vCenter 6.7_10.vmdk" returns: "-rw-r--r--    1 root     root           576 Aug 11 11:26 vCenter 6.7_10.vmdk"

vmkfstools -e returns: "Failed to open disk link /vCenter 6.7_10.vmdk :The system cannot find the file specified (25)Disk chain is not consistent : The system cannot find the file specified (25)" (i have cut the long directory name)

i was trying this articles but i can't find a way to figure out the disk size

https://kb.vmware.com/s/article/1002511?

https://kb.vmware.com/s/article/1026353?

https://kb.vmware.com/s/article/1007969?

i'll be happy to get some help

[a_p_]

Recreating a descriptor file for a vSAN disk, requires to also modify the vSAN object link within the newly created file, so that it points to the existing vSAN object. Did you do this?

Unless already overwritten, you may be able to retrieve the object ID from one of the VM's vmware.log files.

Please see whether Re: .vmdk File Missing​ helps.

If you are unsure, you may consider to contact VMware support.

André

[a_p_]

Moderator Note: Discussion moved from ESXi to VMware vSAN

[afikzach]

i have made a local copy of the vm and i would like to recover it locally on one of the hosts to make backup is there way to do this ?

since the vm is vcenter the vsan is doing some issue it is still reachable but i would like firstly to fix this vcenter vm locally

have attached the log file

[a_p_]

The provided log file unfortunately only shows an already erroneous VM. Do you have an older log file which from the time when the VM still worked?

How did you create the local copy without access to the missing virtual disk? Which files do you have on that local copy? Please provide a complete list, i.e. the text output of ls -elisa.

Btw. Do you have a VM based backup? In this case you may be able to retrieve the required information from the backed up descriptor .vmdk file.

André

[afikzach]

hello André

i have download all the log that i could find including file list please have a look i believe there is some older data in this logs

as i mentioned the vsan is still reachable i just copied it locally using winscp

sadly i was stupid enough and forgot to make backup 😞

i tried to revert it one snapshot back but it dons't work... if you have an idea it will be a great help...

my idea was to use vmkfstools to recreate the corrupted vCenter 6.7_10.vmdk file, but i cannot find the correct size since is not appear properly in the output of ls -l

[a_p_]

The downloaded files unfortunately won't help much, because the data .vmdk files are missing.

Anyway, please run

esxcli vsan debug object list > /tmp/esxcli_vsan_debug_object_list.txt

from the ESXi host's console to get an overview of the existing objects. Then download the resulting .txt file, compress/zip it along with all the already downloaded .vmdk descriptor files, and attach the .zip archive to a reply post.

Note: If you want, you may remove unrelated information from the .txt file. Just make sure that all of the vcenter .vmdk objects remain in the file.

André

[afikzach]

hello again i have made a complete download of all the files (except the *.vmsn) including the esxcli_vsan_debug_object_list.txt

thanks for the help André

[a_p_]

I'm afraid that I can't help you with recovering the data. The object list only shows the snapshots for "vCenter 6.7_10", built not the object for the base disk.

Please consider to open a support call with VMware regarding this issue.

One more question though. Is the vCSA the only VM on the vSAN datastore, or did you remove other objects?

André

[TheBobkin]

As André pointed out, the base disk (vCenter 6.7_10.vmdk - UUID:30334e5d-959b-08e3-2983-54a05086cd01) does not appear at all in the object list.

If the cluster is fully-formed (validate with 'esxcli vsan cluster get' and confirm the membership count is correct), can you please share the output of this run from any node in the cluster:

# cmmds-tool find -t DOM_OBJECT -u 30334e5d-959b-08e3-2983-54a05086cd01

# /usr/lib/vmware/osfs/bin/objtool getAttr -u 30334e5d-959b-08e3-2983-54a05086cd01

Please if you could elaborate on how you were migrating the VM and whether you still have a copy of the original base-disk (even an old one might suffice as this is the Auto Deploy disk).

Bob

[afikzach]

hello bob here is the output for this commands

about the vsan i had there three vm's : vcenter , vsanwhiteness and widows 7 test machine

about the migration process it was just a normal migration from vcenter sadly i was not so smart to make any backup

i have an which sound stupid but it might work.... since the VM was created from the vcsa ova i can make a new one and implant the corrupted vmdk? dose it worth a try or it's too much of creativity...

[TheBobkin]

Sorry to be the bearer of bad news but that Object is gone gone.

Are you 100% positive the original base-disk no longer resides on the original storage?

I don't think borrowing the same disk from another vCSA will work as I am fairly sure this has mechanism of knowing it is not its disk.

You can try (even temporarily) booting the VM without this disk (e.g. edit the vmx to add a # before all 5 lines referencing scsi0:11) and see if it is in any way functional.

Bob