1 2 Previous Next 21 Replies Latest reply on Aug 7, 2020 7:49 AM by a.p.

    ESX vmfs deleted

    mredwarren Novice

      Hello everyone,

      I am in a dire need of some assistance!  My tech and I was having an issue restoring an Exchange VM (something to do with VSS) and not enough space on the datastore. We decided to free up some space by deleting some old files from out of the NFS storage that is being shared with two ESXi hosts. Well here is where things went bad.

       

      Apparently some files inside one of those folders were somehow being used by our Domain File Server (DFS)/Printer Server. After they were deleted that server will not boot up and reference that it needs that '.vmdk' file. We lost access to all files located in our Netapp storage as well as access to the printer server for printing. We searched our backups Netbackup 7 (outdated I know) which backs up our Netapps storage volumes.

       

      We tried finding the /vmfs volume on the backup tapes but could not locate them - I think we need some sort of treasure map!!! We found the regular user files that are stored in NFS but no VMs and none of the ESX related folders.

       

      Question - Is there some way to recover what we have lost even though we deleted the files? I saw some tools such as VMFS Recovery?

       

      I am extremely desperate.....

       

      Thanks for any assistance.

       

      Ed

        • 1. Re: ESX vmfs deleted
          amirmohammadimir Enthusiast

          I am afraid you have to use 3rd party recovery software, I had experience with below which worked.

          https://www.diskinternals.com/vmfs-recovery/

          • 2. Re: ESX vmfs deleted
            a.p. Guru
            Community WarriorsvExpertUser Moderators

            Two quick questions:

            1. You mentioned Netapp. Does the NFS volume have storage based snapshots created, which may still contain the deleted files?
            2. Did you delete the files while the VMs were powered on? In that case, it may be possible that only the descriptor files got deleted (which can usually be recreated), because the data files were in use (locked)!? In case there are still files in the VMs' folders, please provide an overview (including names, extensions, time stamps, sizes).

             

            André

            • 3. Re: ESX vmfs deleted
              mredwarren Novice

              Andé,

              I will have to check whether there are NFS volume having storage based snapshots. We took over the network from a previous IT team and slowly understanding everything. Not the best of times for us but we move forward.

               

              Is there a quick way to check if they are?

               

              The VM was powered on at the time the files were deleted. The folder remains in the directory and there are two files still remaining. If I am not mistaken they are .vmdk Descriptor files but I will not be able to collect information until Monday. Below is what it said when trying to power on that vm server:

               

              "Failed to power on virtual machine <vm name>. File /vmfs/volumes/b37bd535-f5dc2f75/<vm name>_1.00000.vmdk was not found.

              Click here for more details."

               

              Thanks again,

               

              Ed

              • 4. Re: ESX vmfs deleted
                a.p. Guru
                Community WarriorsvExpertUser Moderators

                For the storage snapshots, I can't help you much without knowing configuration details, so that's something you have to find out first.

                Regarding the .vmdk files. As you may know, on ESXi each virtual disk consists of two files, a descriptor file (e.g. "<vmname>.vmdk"), and a data file (e.g. "<vmname>-flat.vmdk"). If the flat files are still in place, it should be possible to recreate the descriptor files, which only contain some metadata like the disk's provisioned size, geometry, ... Note that in case the VM has/had active snapshots (the file name that you posted makes me think that there are snapshots) there may also exist delta, or sesparse files.

                Anyway, if such files are still present, then please check whether there are still some older vmware*.log files in the VM's home folder, which contain details about the VM's virtual disks. These files will help to recreate missing descriptor files.

                 

                André

                • 5. Re: ESX vmfs deleted
                  mredwarren Novice

                  Good Morning André,

                  Below are the two files that remained in the folder directory. It looks as if the -flat.vmdk is quite large and I am hoping that the files can be restored from it.

                   

                  I checked for .log files but I did not see any of those.

                   

                  Without the .log files is it still possible to rebuild what was deleted?

                  • 6. Re: ESX vmfs deleted
                    a.p. Guru
                    User ModeratorsvExpertCommunity Warriors

                    Since the files are stored on NFS, they are thin provisioned, so the current files sizes do not represent the virtual disk's provisioned size, i.e. the size that showed up in the VM's settings. That's why it is important to find some log files, which - as mentioned before - are usually stored in the VM's home folder that also contains the VM's configuration (.vmx) file. Please double-check whether you can find log files there, or in recent backups. Other than this, please run ls -lisa in the VM's folder ("/vmfs/volumes/b37bd535-f5dc2f75/" according to an earlier reply) , and post the result.

                     

                    Regarding backups. You mentioned that you were searching for "/vmfs" in your backups. The "/vmfs/volumes" folder only exists on ESXi where the different datastores are mounted, so a better approach is to search for .vmdk files.

                    From the presented files' time stamps it seems that these .vmdk files were not active for quite some time, so if there's a way to restore the missing descriptor files, expect some data loss.

                     

                    André

                    • 7. Re: ESX vmfs deleted
                      mredwarren Novice

                      [root@vsvh-glc-uea02:/vmfs/volumes/b37bd535-f5dc2f75] ls -lisa

                      total 336976

                           64      4 drwxr-xr-x   17 root     root          4096 Jul 29 16:19 .

                      73686543891504      0 drwxr-xr-x    1 root     root           512 Aug  4 12:40 ..

                         3465      4 drwxr-xr-x    2 root     root          4096 Aug  4 12:37 .iorm.sf

                         3466      8 -rwxr-xr-x    1 root     root         16511 Aug  4 12:37 .iormstats.sf

                        13945      4 -rwxrwxr-x    1 root     root            84 Mar 29  2017 .lck-7836000000000000

                         3468      4 -rwxrwxr-x    1 root     root            84 Aug  4 12:37 .lck-8a0d000000000000

                           67      4 drwxrwxrwx    2 root     root          4096 Jan 28  2015 .snapshot

                          103      4 drwx------    5 root     root          4096 May 26 13:10 .vSphere-HA

                          102      4 drwxr-xr-x    2 root     root          4096 Jul 31 12:19 GLC-HOST-02.MDGLC.COM

                        30063      4 drwxr-xr-x    3 root     root          4096 Jun 19  2019 ISOs

                        13944 336888 -rw-------    1 root     root     343615488 Oct  5  2016 VMware-VMvisor-Installer-5.5.0.update01-1623387.x86_64.iso

                          962      4 drwxr-xr-x    2 root     root          4096 Aug  3 23:10 asrm-glc-uea02

                          945      4 drwxr-xr-x    2 root     root          4096 Aug  3 23:13 dsdh-glc-uea01

                        28219      4 drwxr-xr-x    2 root     root          4096 Aug  3 23:28 dsdh-glc-uea02

                         2858      8 drwxr-xr-x    2 root     root          8192 Aug  4 00:30 emdr-glc-uea01

                         2857      8 drwxr-xr-x    2 root     root          8192 Jul  8 08:04 emvm-glc-uea01

                         2871      4 drwxr-xr-x    2 root     root          4096 Aug  4 00:08 emvm-glc-uea03

                        28203      4 drwxr-xr-x    2 root     root          4096 Aug  4 02:18 msmr-glc-uea01

                         2856      4 drwxr-xr-x    2 root     root          4096 Aug  4 00:46 mums-glc-uea01

                          964      4 drwxr-xr-x    2 root     root          4096 Aug  4 00:52 mums-glc-uea02

                           96      4 drwxr-xr-x    2 root     root          4096 May  8  2015 wsl4.2

                      • 8. Re: ESX vmfs deleted
                        a.p. Guru
                        User ModeratorsCommunity WarriorsvExpert

                        Sorry, my bad, it's "/vmfs/volumes/b37bd535-f5dc2f75/GLC-HOST-02.MDGLC.COM" which contains the files.

                         

                        André

                        • 9. Re: ESX vmfs deleted
                          mredwarren Novice

                          [root@vsvh-glc-uea02:/vmfs/volumes/b37bd535-f5dc2f75/GLC-HOST-02.MDGLC.COM] ls -lisa

                          total 756236788

                              102      4 drwxr-xr-x    2 root     root          4096 Jul 31 12:19 .

                               64      4 drwxr-xr-x   18 root     root          4096 Aug  4 13:29 ..

                            23709   2468 -rw-------    1 root     root      19070976 Jul 17 20:42 GLC-HOST-02.MDGLC.COM_1-000001-delta.vmdk

                           

                           

                          I also was recommended to use the DiskInternals VMFS Recovery software. Have you had experience with that?

                          • 10. Re: ESX vmfs deleted
                            a.p. Guru
                            Community WarriorsvExpertUser Moderators

                            Where's the flat.vmdk file that showed up ion the previously presented screenshot?

                            I also was recommended to use the DiskInternals VMFS Recovery software. Have you had experience with that?

                            When it comes to data recovery from disks/LUNs, then continuum may need to take over.


                            André

                            • 11. Re: ESX vmfs deleted
                              mredwarren Novice

                              I did the ls -lisa in that folder.

                               

                              [root@vsvh-glc-uea02:/vmfs/volumes/b37bd535-f5dc2f75/GLC-HOST-02.MDGLC.COM] ls -lisa

                              total 756236788

                                  102      4 drwxr-xr-x    2 root     root          4096 Jul 31 12:19 .

                                   64      4 drwxr-xr-x   18 root     root          4096 Aug  4 13:29 ..

                                23709   2468 -rw-------    1 root     root      19070976 Jul 17 20:42 GLC-HOST-02.MDGLC.COM_1-000001-delta.vmdk

                                31471 756234312 -rw-------    1 root     root     1199865593856 Jun 25 14:28 GLC-HOST-02.MDGLC.COM_1-flat.vmdk

                              • 12. Re: ESX vmfs deleted
                                a.p. Guru
                                Community WarriorsvExpertUser Moderators

                                Ok, then let's see whether we can find out the virtual disk's details from the metadata in the delta file.

                                For this, please extract the first 2048 Bytes from the delta file, and attach the extracted metadata.bin file to your next reply post. You may use e.g. the following command:

                                dd if=GLC-HOST-02.MDGLC.COM_1-000001-delta.vmdk of=metadata.bin bs=1024 count=2

                                André

                                • 13. Re: ESX vmfs deleted
                                  mredwarren Novice

                                  Good Morning André,

                                  I was able to extract the .bin file as you recommended. I did not know what to open it up in to view it.

                                   

                                  I attached it here.

                                   

                                  I ran a command od -cx metadata.bin

                                   

                                  [root@vsvh-glc-uea02:/vmfs/volumes/b37bd535-f5dc2f75/GLC-HOST-02.MDGLC.COM] od -cx metadata.bin

                                  0000000   C   O   W   D 001  \0  \0  \0 003  \0  \0  \0  \0 0I0 0E6 0A3

                                          4f43 4457 0001 0000 0003 0000 c800 8bae

                                  0000020 001  \0  \0  \0 004  \0  \0  \0  \0 0G3  \b  \0   8 022  \0  \0

                                          0001 0000 0004 0000 bb00 0008 1238 0000

                                  0000040  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0

                                          0000 0000 0000 0000 0000 0000 0000 0000

                                  *

                                  0003160 001  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0

                                          0001 0000 0000 0000 0000 0000 0000 0000

                                  0003200  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0

                                          0000 0000 0000 0000 0000 0000 0000 0000

                                  *

                                  0004000

                                  • 14. Re: ESX vmfs deleted
                                    mredwarren Novice

                                    I downloaded the trial version and was able to mount and look into the datastore. I was not able to find the deleted files but that is probably due to me not knowing exactly where to look for deleted files.

                                     

                                    Is there a specific place they could be housed?

                                    1 2 Previous Next