VMware Cloud Community
aaronwsmith
Enthusiast
Enthusiast

Recover Deleted VM From VMFS5?

Hello Everyone,

Due to a miscommunication, a VM was powered down and deleted from disk (VMFS5.)  Any solution/options to recover the VM file, assuming the free space has not been consumed by something else?

Thanks!

Reply
0 Kudos
19 Replies
continuum
Immortal
Immortal

was that a thick provisioned vmdk ?

anyway - there is a better than zero chance that the file still exists if you read the volume with "vmfs-fuse"

if "vmfs-fuse" does not help and the the vmdk was thick provisioned I may be able to help - call me via skype "sanbarrow"

Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

aaronwsmith
Enthusiast
Enthusiast

Thanks for the response!

Yes VMs are all thick provisioned eager zeroed.  Not familiar with vmfs-tools.

To use it, do I need to do the following?

1. Setup a Linux VM (either LiveCD or traditional install)

2. Download/build vmfs-tools

3. Mount the VMFS as a RDM to the Linux VM

4. Use vmfs-fuse to mount the the VMFS within the VM.

Reply
0 Kudos
continuum
Immortal
Immortal

with latest Ubuntu LiveCDs you can install "vmfs-tools" via apt - the latest LiveCDs have the version that can read VMFS 5.
Then find out the partition device name such as /dev/sdb1 for example.

As root do

mkdir /vmfs
vmfs-fuse /dev/sdb1 /vmfs

Report the message you get


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
aaronwsmith
Enthusiast
Enthusiast

Is the process then to boot the ESXi host itself (which has all the SAN volumes exposed to it via the FC cards) with Ubuntu LiveCD, then install vmfs-tools and run the commands above to mount the VMFS5 volumes?

Reply
0 Kudos
aaronwsmith
Enthusiast
Enthusiast

Reason I ask about booting the ESXi host from the Ubuntu Live CD (downloading now...) is because the VMFS volumes have other live VMs running on it.  Since we don't even know which datastore had the deleted VM originally, I would need to check each volume, but do so in a way that doesn't interrupt what's running.  And since we don't know where the VM originally resided, moving VMs around to temporarily convert them into RDMs for a VM would be very time consuming and much more likely to overwrite the deleted VM.

Reply
0 Kudos
continuum
Immortal
Immortal

If possible on your host - use a Linux VM and map the corrupt LUN as a RDM.

To find out where the original VM was located dump the first 1500 mb of each vmfs-volume.
Copy the dumps to a Windows or Linux machine or VM and run strings(.exe) against the dump.
Then you can grep the results for the displayname of the VM you look for


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
aaronwsmith
Enthusiast
Enthusiast

Thanks, but as I mentioned, the VMFS is not corrupt, and has live VMs running on it.  So I cannot switch them to RDMs to mount within a VM without powering down the VMs (not really an option) or moving the VMs to another disk (risks that I'd clobber whichever VMFS had the original VM.)

I have the ESXi host booted from Ubuntu, but I may have a new limitation ... not sure yet if this distro has any drivers to support the FC cards installed on the server.  And we have to open access on our firewall to enable the host (would be the same limitation for a VM) to download the software.

I'll keep you posted, but this is looking less possible to achieve in our case.

Reply
0 Kudos
aaronwsmith
Enthusiast
Enthusiast

FYI - At this point we cut our loss on the VM, but I'm still looking into this as a side project for building a Linux OS with the necessary drivers to try booting an ESXi host with vmfs-tools to read all the attached volumes and search for deleted VMDK files.  If/when I reach a point where I am able to test this, I'll post again with my findings.

Reply
0 Kudos
continuum
Immortal
Immortal

I have LiveCDs for this purpose - right now I am finalizing one that boots from EFI and BIOS hosts

send me a PM and I give you a link when the new version is ready


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
OscarDavey
Hot Shot
Hot Shot

Hello,

Try to use file recovery programs(not install them, try Live CD versions at first).

For example Actice Boot Disk(Active Undelete), GetDataBack...

Hope will help.

Your Oscar

Reply
0 Kudos
aaronwsmith
Enthusiast
Enthusiast

Hey continuum, will do soon.  Thanks!  Lower priority at this point for me, but still interested in exploring the idea.

Reply
0 Kudos
aaronwsmith
Enthusiast
Enthusiast

OscarDavey, wouldn't that require the OS to be able to mount/read the VMFS itself?  Or are you assuming that's been done.  Either way, for FC attached storage in this case, the lpfc820 driver is required to recognize the HBAs on the server.  Otherwise without that driver, it sees nothing.

Reply
0 Kudos
continuum
Immortal
Immortal

OscarDaveys suggestion is what one would usually try as the very last option - when everything else has already failed.

Plan A: fix the VMFS-volume with tools like partedUtil, vmkfstools ,
Plan B: use a LiveCD with ESXi - often this helps to retrieve files that are invisible to the ESXi that runs on the host
Plan C: use a Linux LiveCD eith vmfs-fuse - there is a good chance to find vmdks that have been recently deleted manually
Plan 😧 use a Windows LiveCD either with UFS-explorer or DiskInternals VMFS Recovery Tool
Plan E: carve out complete vmdk files manually
....
if all that failed ... use standard Recovery Tools like the once mentioned by OscarDavey

If you search a raw volume with tools like GetDataBack best you can get are small unfragmented files.
In such a scenario the NTFS-MasterFileTable can not be used - so the chance to recover large fragmented files almost drops down to zero


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
WhoIsJohnGalt
Contributor
Contributor

I have a similar problem, but have MUCH more information about my particular mess and have not touched the drives since deleting my file.

BACKGROUND:

Using esxi 4.1, using GUI only.

I was preparing to migrate an old Server 2003 VM to a new Server 2008 R2 VM so I could delete the old server for all my Windows Server needs. My friend, with a desire to be future-ready and wanting to have RAID redundancy had me install a network iSCSI device with Raid 1 drives (1TB each). I did so by copying the server files from his host datastore to the iSCSI LUN and created a new VM using the existing .vmdx file for the 'existing disk'. My friend wanted to have access to this massive new storage space, so I relented and allocated 850GB (of 900GB LUN) to his Server 2003. (server 2008 came later and was on different datastore). This process created a new .vmdx file in my newly created Server 2003 v2 folder on the datastore and I deleted the Server 2003 VM but left the Server 2003 folder for backup.

I was asked this week to migrate Active Directory and his file server to his 2008 server which has been running for 6 months now and delete the old 2003 server. My plan was to shrink the partition on the 2003 server, move the 2008 server to the iSCSI, prep the 2003 for AD migration, dcpromo the 2008 server, create new shares, transfer files, change login scripts for mapping, and call it a night. Plan was tomorrow to verify full replication and take the 2003 offline.

I started by booting the 2003 server to a live CD and tried to use GParted to resize the partition. After about 10min of the shrink operation running, I got an error message from esxi that my datastore was out of space, retry or cancel. Upon checking the host's storage, I saw that it was reporting only 2.5GB (or MB?). I started moving my 'backup' folder from that datastore to the esx server's internal datastore hoping to free up some space, but it promised to take 2hrs due to a large VM disk file. So I went in and tried to delete the Server 2003 folder (old server, I thought). It wouldn't let me. After an hour of waiting, I finally hit 'cancel' on the esx popup error message and the 'server 2003 v2' VM shut down. I tried again to delete the Server 2003 folder and it deleted, freeing up 800GB of space it appeared. My 'Server 2003 v2' VM showed only 17GB of used space of it's provisioned 850GB. Thinking I had room to spare, I tried to restart the VM and discovered that I had just done something wrong. An hour later, I realized that I deleted the Parent .vmdx of my VM, and the VM was not going to start without it.


WHAT I KNOW:

I know the datastore the files were in

I know the names of the files

The LUN is accessible

Nothing has been written to the drives since deleting the folder (I turned off both VMs -- weekends rock)

I do not have a recent backup - important data is the files that store my friend's database of customers to visit in the future and past visits (plumber)

I made LOTS of thoughtless mistakes and am worthy of serious chiding and fun-poking


WHAT I DON'T KNOW:

How to undelete these files:

the .vmdx file -and-

the .vmx file

Who knows the information I should know to solve this problem - and -

What will compel them to share this information with me before my friend's business suffers from my mistake 36 hours from now.

PLEASE HELP. THANK YOU FOR WHATEVER YOU CAN OFFER!

Reply
0 Kudos
Stainless2000
Contributor
Contributor

This is my exact problem. Did you solve it back then?

Reply
0 Kudos
WhoIsJohnGalt
Contributor
Contributor

I was not able to solve the problem. I spent three days without sleep trying to find a solution. I even had an expert from somewhere in Europe volunteer about 12 hours to help me try to recover the data from the drives. We did a raw copy of the drive's sectors and all kinds of stuff I still do not understand, but no solution was found. We sent the drives off to a data-recovery center and they were unable to recover the data as well. We restored from the super-old backups and accepted the tragic loss.

Having thought about it for years now I have come up with a variety of new ways to try to recover the data I have thought of a variety of things I could try. I actually bought the old server from my friend's boss when they upgraded so I could test my ideas, but have not actually done so. My most promising idea was to install a PCI-X controller card so I could install a new version of ESXi on a different drive (like SATA or IDE), and then install a windows VM on that new virtual host. Then using that new windows VM, I would mount the old datastore as a drive (this is the part I haven't worked out yet, how to mount it without changing anything), and then run data recovery software on the drive from there. Something like GetDataBack NTFS should do the trick. It should be able to find NTFS file headers and I should be able to see the files even if the FAT structure was not totally in tact. I haven't tried this yet, because the price of a SCSI controller that would accept the drives from that server is expensive (considering it's too late to be useful), and the PCI-X controller I bought to use in the original server didn't allow me to install ESXi onto the new SATA drive. I moved on with my life, and someday will try again when I have the time and more cash to spend on antiquated test hardware.


Depending on the hardware you have available in your machine, you may be able to try something like that without the limitations my situation had. I know how you feel right now. I offer my sincere condolences for your troubles and my prayers that you figure out a way to solve them. The long-term lesson that I have learned, that has saved me dozens of times since, is to always under-utilize your resources so you have the flexibility to change things, and to copy and backup your important data.

Reply
0 Kudos
continuum
Immortal
Immortal

Here is a quick test to evaluate chances for a recovery of accidentaly deleted flat.vmdks.
Dump the headers of  the VMFS-volume into a file. Read that file with a Linux or Windows host using strings ( https://technet.microsoft.com/en-us/sysinternals/strings.aspx )
strings VMFS-header-dump | grep "flat.vmdk"  | grep -v "\"" > recovery-probable.txt
When you find the name of your deleted flat.vmdk in recovery-probable.txt and it appears on its own line without quotes or any other "obscure" characters then the vmdk should be recoverable with comparably small amount of work.
If it does not appear in that list it does not mean all hope is gone - it sure means that its more work.Upto ESX 5.0 the chances to recover a deleted.vmdk were generally quite good.
Not so on any later versions - nowadays it is more like a 50:50 chance.
If you need a more detailed report for your special case - send a downloadlink to a VMFS-header dump and call me via skype.
Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
Stainless2000
Contributor
Contributor

this is actually esxi 4.0. You'll have to walk me through this since this is not my specialty. Do you want me to use a boot disk at the host to look for those files? Unfortunately, Skype is blocked for my network, but i am sure I could find other ways around. My first thought, was to use a liveCD to look at the host file system (if possible with vmware) and see if I could simply undelete the vmdk files missing, but there are several vm on that host that are mission critical so i would have to know exactly what i was doing beforehand to minimize downtime.

Reply
0 Kudos
continuum
Immortal
Immortal

Easiest way for you: call me using skype - see my signature.
As soon as we can talk I can give you safe advice.
Dont worry about language issues - if you can post here we should have no problems.
If you are worried about still running crtical VMs - yes - thats not ideal but as it is often requested I have a minimal invasive - though slower - procedure that can handle all such problems.
Only important tip: call as soon as possible - creating the VMFS header-dump should be done as early as possilble !!!

Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos