11 Replies Latest reply on Aug 12, 2019 6:42 AM by TheBobkin Branched from an earlier discussion.

    ESXI 6.5 VM Recovery (missing vSAN vmdk)

    afikzach Novice

      hello i'm having a similar issue i have VM which got corrupted while migrating in vsan from one host to another, vmdk number 10 chain got broken.

      when i'm trying to recreate it using the vmkfstools, i'm facing an issue i'm not able to receive the correct size using "ls -l " it dos work....

       

      ls -l  "vCenter 6.7_10.vmdk" returns: "-rw-r--r--    1 root     root           576 Aug 11 11:26 vCenter 6.7_10.vmdk"

       

      vmkfstools -e returns: "Failed to open disk link /vCenter 6.7_10.vmdk :The system cannot find the file specified (25)Disk chain is not consistent : The system cannot find the file specified (25)" (i have cut the long directory name)

       

      i was trying this articles but i can't find a way to figure out the disk size

       

      https://kb.vmware.com/s/article/1002511?

      https://kb.vmware.com/s/article/1026353?

      https://kb.vmware.com/s/article/1007969?

       

       

      i'll be happy to get some help

        • 1. Re: ESXI 6.5 VM Recovery (missing vSAN vmdk)
          a.p. Guru
          User ModeratorsCommunity WarriorsvExpert

          Recreating a descriptor file for a vSAN disk, requires to also modify the vSAN object link within the newly created file, so that it points to the existing vSAN object. Did you do this?

          Unless already overwritten, you may be able to retrieve the object ID from one of the VM's vmware.log files.

          Please see whether Re: .vmdk File Missing helps.

           

          If you are unsure, you may consider to contact VMware support.


          André

          • 2. Re: ESXI 6.5 VM Recovery (missing vSAN vmdk)
            a.p. Guru
            vExpertCommunity WarriorsUser Moderators

            Moderator Note: Discussion moved from ESXi to VMware vSAN

            • 3. Re: ESXI 6.5 VM Recovery (missing vSAN vmdk)
              afikzach Novice

              i have made a local copy of the vm and i would like to recover it locally on one of the hosts to make backup is there way to do this ?

              since the vm is vcenter the vsan is doing some issue it is still reachable but i would like firstly to fix this vcenter vm locally

               

              have attached the log file

              • 4. Re: ESXI 6.5 VM Recovery (missing vSAN vmdk)
                a.p. Guru
                vExpertCommunity WarriorsUser Moderators

                The provided log file unfortunately only shows an already erroneous VM. Do you have an older log file which from the time when the VM still worked?

                How did you create the local copy without access to the missing virtual disk? Which files do you have on that local copy? Please provide a complete list, i.e. the text output of ls -elisa.

                 

                Btw. Do you have a VM based backup? In this case you may be able to retrieve the required information from the backed up descriptor .vmdk file.

                 

                André

                • 5. Re: ESXI 6.5 VM Recovery (missing vSAN vmdk)
                  afikzach Novice

                  hello André

                   

                  i have download all the log that i could find including file list please have a look i believe there is some older data in this logs

                  as i mentioned the vsan is still reachable i just copied it locally using winscp

                   

                  sadly i was stupid enough and forgot to make backup :-(

                  i tried to revert it one snapshot back but it dons't work... if you have an idea it will be a great help...

                   

                  my idea was to use vmkfstools to recreate the corrupted vCenter 6.7_10.vmdk file, but i cannot find the correct size since is not appear properly in the output of ls -l

                  • 6. Re: ESXI 6.5 VM Recovery (missing vSAN vmdk)
                    a.p. Guru
                    Community WarriorsUser ModeratorsvExpert

                    The downloaded files unfortunately won't help much, because the data .vmdk files are missing.

                    Anyway, please run

                    esxcli vsan debug object list > /tmp/esxcli_vsan_debug_object_list.txt

                    from the ESXi host's console to get an overview of the existing objects. Then download the resulting .txt file, compress/zip it along with all the already downloaded .vmdk descriptor files, and attach the .zip archive to a reply post.

                    Note: If you want, you may remove unrelated information from the .txt file. Just make sure that all of the vcenter .vmdk objects remain in the file.

                     

                    André

                    • 7. Re: ESXI 6.5 VM Recovery (missing vSAN vmdk)
                      afikzach Novice

                      hello again i have made a complete download of all the files (except the *.vmsn) including the esxcli_vsan_debug_object_list.txt

                       

                      thanks for the help André

                      • 8. Re: ESXI 6.5 VM Recovery (missing vSAN vmdk)
                        a.p. Guru
                        vExpertCommunity WarriorsUser Moderators

                        I'm afraid that I can't help you with recovering the data. The object list only shows the snapshots for "vCenter 6.7_10", built not the object for the base disk.

                        Please consider to open a support call with VMware regarding this issue.

                        One more question though. Is the vCSA the only VM on the vSAN datastore, or did you remove other objects?

                         

                        André

                        • 9. Re: ESXI 6.5 VM Recovery (missing vSAN vmdk)
                          TheBobkin Virtuoso
                          vExpertVMware Employees

                          As André pointed out, the base disk (vCenter 6.7_10.vmdk - UUID:30334e5d-959b-08e3-2983-54a05086cd01) does not appear at all in the object list.

                          If the cluster is fully-formed (validate with 'esxcli vsan cluster get' and confirm the membership count is correct), can you please share the output of this run from any node in the cluster:

                          # cmmds-tool find -t DOM_OBJECT -u 30334e5d-959b-08e3-2983-54a05086cd01

                          # /usr/lib/vmware/osfs/bin/objtool getAttr -u 30334e5d-959b-08e3-2983-54a05086cd01

                           

                          Please if you could elaborate on how you were migrating the VM and whether you still have a copy of the original base-disk (even an old one might suffice as this is the Auto Deploy disk).

                           

                          Bob

                          • 10. Re: ESXI 6.5 VM Recovery (missing vSAN vmdk)
                            afikzach Novice

                            hello bob here is the output for this commands

                            about the vsan i had there three vm's : vcenter , vsanwhiteness and widows 7 test machine

                            about the migration process it was just a normal migration from vcenter sadly i was not so smart to make any backup

                            i have an which sound stupid but it might work.... since the VM was created from the vcsa ova i can make a new one and implant the corrupted vmdk? dose it worth a try or it's too much of creativity...

                            • 11. Re: ESXI 6.5 VM Recovery (missing vSAN vmdk)
                              TheBobkin Virtuoso
                              VMware EmployeesvExpert

                              Sorry to be the bearer of bad news but that Object is gone gone.

                              Are you 100% positive the original base-disk no longer resides on the original storage?

                              I don't think borrowing the same disk from another vCSA will work as I am fairly sure this has mechanism of knowing it is not its disk.

                              You can try (even temporarily) booting the VM without this disk (e.g. edit the vmx to add a # before all 5 lines referencing scsi0:11) and see if it is in any way functional.

                               

                              Bob