1 2 Previous Next 19 Replies Latest reply on Dec 6, 2016 10:12 PM by continuum

    Recover Deleted VM From VMFS5?

    aaronwsmith Hot Shot

      Hello Everyone,


      Due to a miscommunication, a VM was powered down and deleted from disk (VMFS5.)  Any solution/options to recover the VM file, assuming the free space has not been consumed by something else?



        • 1. Re: Recover Deleted VM From VMFS5?
          continuum Guru
          Community WarriorsUser ModeratorsvExpert

          was that a thick provisioned vmdk ?


          anyway - there is a better than zero chance that the file still exists if you read the volume with "vmfs-fuse"


          if "vmfs-fuse" does not help and the the vmdk was thick provisioned I may be able to help - call me via skype "sanbarrow"



          1 person found this helpful
          • 2. Re: Recover Deleted VM From VMFS5?
            aaronwsmith Hot Shot

            Thanks for the response!


            Yes VMs are all thick provisioned eager zeroed.  Not familiar with vmfs-tools.


            To use it, do I need to do the following?


            1. Setup a Linux VM (either LiveCD or traditional install)

            2. Download/build vmfs-tools

            3. Mount the VMFS as a RDM to the Linux VM

            4. Use vmfs-fuse to mount the the VMFS within the VM.

            • 3. Re: Recover Deleted VM From VMFS5?
              continuum Guru
              User ModeratorsvExpertCommunity Warriors

              with latest Ubuntu LiveCDs you can install "vmfs-tools" via apt - the latest LiveCDs have the version that can read VMFS 5.
              Then find out the partition device name such as /dev/sdb1 for example.


              As root do


              mkdir /vmfs
              vmfs-fuse /dev/sdb1 /vmfs


              Report the message you get

              • 4. Re: Recover Deleted VM From VMFS5?
                aaronwsmith Hot Shot

                Is the process then to boot the ESXi host itself (which has all the SAN volumes exposed to it via the FC cards) with Ubuntu LiveCD, then install vmfs-tools and run the commands above to mount the VMFS5 volumes?

                • 5. Re: Recover Deleted VM From VMFS5?
                  aaronwsmith Hot Shot

                  Reason I ask about booting the ESXi host from the Ubuntu Live CD (downloading now...) is because the VMFS volumes have other live VMs running on it.  Since we don't even know which datastore had the deleted VM originally, I would need to check each volume, but do so in a way that doesn't interrupt what's running.  And since we don't know where the VM originally resided, moving VMs around to temporarily convert them into RDMs for a VM would be very time consuming and much more likely to overwrite the deleted VM.

                  • 6. Re: Recover Deleted VM From VMFS5?
                    continuum Guru
                    User ModeratorsCommunity WarriorsvExpert

                    If possible on your host - use a Linux VM and map the corrupt LUN as a RDM.


                    To find out where the original VM was located dump the first 1500 mb of each vmfs-volume.
                    Copy the dumps to a Windows or Linux machine or VM and run strings(.exe) against the dump.
                    Then you can grep the results for the displayname of the VM you look for

                    • 7. Re: Recover Deleted VM From VMFS5?
                      aaronwsmith Hot Shot

                      Thanks, but as I mentioned, the VMFS is not corrupt, and has live VMs running on it.  So I cannot switch them to RDMs to mount within a VM without powering down the VMs (not really an option) or moving the VMs to another disk (risks that I'd clobber whichever VMFS had the original VM.)


                      I have the ESXi host booted from Ubuntu, but I may have a new limitation ... not sure yet if this distro has any drivers to support the FC cards installed on the server.  And we have to open access on our firewall to enable the host (would be the same limitation for a VM) to download the software.


                      I'll keep you posted, but this is looking less possible to achieve in our case.

                      • 8. Re: Recover Deleted VM From VMFS5?
                        aaronwsmith Hot Shot

                        FYI - At this point we cut our loss on the VM, but I'm still looking into this as a side project for building a Linux OS with the necessary drivers to try booting an ESXi host with vmfs-tools to read all the attached volumes and search for deleted VMDK files.  If/when I reach a point where I am able to test this, I'll post again with my findings.

                        • 9. Re: Recover Deleted VM From VMFS5?
                          continuum Guru
                          User ModeratorsCommunity WarriorsvExpert

                          I have LiveCDs for this purpose - right now I am finalizing one that boots from EFI and BIOS hosts


                          send me a PM and I give you a link when the new version is ready

                          • 10. Re: Recover Deleted VM From VMFS5?
                            OscarDavey Hot Shot



                            Try to use file recovery programs(not install them, try Live CD versions at first).

                            For example Actice Boot Disk(Active Undelete), GetDataBack...


                            Hope will help.

                            Your Oscarhttps://imagicon.info/cat/5-59/1.gif

                            • 11. Re: Recover Deleted VM From VMFS5?
                              aaronwsmith Hot Shot

                              Hey continuum, will do soon.  Thanks!  Lower priority at this point for me, but still interested in exploring the idea.

                              • 12. Re: Recover Deleted VM From VMFS5?
                                aaronwsmith Hot Shot

                                OscarDavey, wouldn't that require the OS to be able to mount/read the VMFS itself?  Or are you assuming that's been done.  Either way, for FC attached storage in this case, the lpfc820 driver is required to recognize the HBAs on the server.  Otherwise without that driver, it sees nothing.

                                • 13. Re: Recover Deleted VM From VMFS5?
                                  continuum Guru
                                  Community WarriorsvExpertUser Moderators

                                  OscarDaveys suggestion is what one would usually try as the very last option - when everything else has already failed.


                                  Plan A: fix the VMFS-volume with tools like partedUtil, vmkfstools ,
                                  Plan B: use a LiveCD with ESXi - often this helps to retrieve files that are invisible to the ESXi that runs on the host
                                  Plan C: use a Linux LiveCD eith vmfs-fuse - there is a good chance to find vmdks that have been recently deleted manually
                                  Plan D: use a Windows LiveCD either with UFS-explorer or DiskInternals VMFS Recovery Tool
                                  Plan E: carve out complete vmdk files manually
                                  if all that failed ... use standard Recovery Tools like the once mentioned by OscarDavey


                                  If you search a raw volume with tools like GetDataBack best you can get are small unfragmented files.
                                  In such a scenario the NTFS-MasterFileTable can not be used - so the chance to recover large fragmented files almost drops down to zero

                                  • 14. Re: Recover Deleted VM From VMFS5?
                                    WhoIsJohnGalt Lurker

                                    I have a similar problem, but have MUCH more information about my particular mess and have not touched the drives since deleting my file.


                                    Using esxi 4.1, using GUI only.

                                    I was preparing to migrate an old Server 2003 VM to a new Server 2008 R2 VM so I could delete the old server for all my Windows Server needs. My friend, with a desire to be future-ready and wanting to have RAID redundancy had me install a network iSCSI device with Raid 1 drives (1TB each). I did so by copying the server files from his host datastore to the iSCSI LUN and created a new VM using the existing .vmdx file for the 'existing disk'. My friend wanted to have access to this massive new storage space, so I relented and allocated 850GB (of 900GB LUN) to his Server 2003. (server 2008 came later and was on different datastore). This process created a new .vmdx file in my newly created Server 2003 v2 folder on the datastore and I deleted the Server 2003 VM but left the Server 2003 folder for backup.


                                    I was asked this week to migrate Active Directory and his file server to his 2008 server which has been running for 6 months now and delete the old 2003 server. My plan was to shrink the partition on the 2003 server, move the 2008 server to the iSCSI, prep the 2003 for AD migration, dcpromo the 2008 server, create new shares, transfer files, change login scripts for mapping, and call it a night. Plan was tomorrow to verify full replication and take the 2003 offline.


                                    I started by booting the 2003 server to a live CD and tried to use GParted to resize the partition. After about 10min of the shrink operation running, I got an error message from esxi that my datastore was out of space, retry or cancel. Upon checking the host's storage, I saw that it was reporting only 2.5GB (or MB?). I started moving my 'backup' folder from that datastore to the esx server's internal datastore hoping to free up some space, but it promised to take 2hrs due to a large VM disk file. So I went in and tried to delete the Server 2003 folder (old server, I thought). It wouldn't let me. After an hour of waiting, I finally hit 'cancel' on the esx popup error message and the 'server 2003 v2' VM shut down. I tried again to delete the Server 2003 folder and it deleted, freeing up 800GB of space it appeared. My 'Server 2003 v2' VM showed only 17GB of used space of it's provisioned 850GB. Thinking I had room to spare, I tried to restart the VM and discovered that I had just done something wrong. An hour later, I realized that I deleted the Parent .vmdx of my VM, and the VM was not going to start without it.

                                    WHAT I KNOW:

                                    I know the datastore the files were in

                                    I know the names of the files

                                    The LUN is accessible

                                    Nothing has been written to the drives since deleting the folder (I turned off both VMs -- weekends rock)

                                    I do not have a recent backup - important data is the files that store my friend's database of customers to visit in the future and past visits (plumber)

                                    I made LOTS of thoughtless mistakes and am worthy of serious chiding and fun-poking

                                    WHAT I DON'T KNOW:

                                    How to undelete these files:

                                    the .vmdx file -and-

                                    the .vmx file

                                    Who knows the information I should know to solve this problem - and -

                                    What will compel them to share this information with me before my friend's business suffers from my mistake 36 hours from now.


                                    PLEASE HELP. THANK YOU FOR WHATEVER YOU CAN OFFER!

                                    1 2 Previous Next