4 Replies Latest reply on Jan 17, 2014 6:28 PM by openDelphi

    Help recovering thick-provisioned flat.vmdk files needed!

    openDelphi Lurker

      Hello all,


      First, some background about my current setup (and I'll try to keep it short and sweet)...

           I have 3 1TB hard drives in an ESXi 5.x host that I use in a RAID5 array.  The RAID is actually set up and managed by a guest Windows Server 2008R2 VM on the host.  When I created the datastores to build the array, I formatted each of the hard drives and added them inside the vSphere Client as VMFS datastores.  When I added the drives to the guest VM's config, I set them to be thick-provisioned lazy-zeroed, and all 3 are the exact same size (930.2899996 GB).  Each disk has one and only one VMDK, and that VMDK takes up most of the usable VMFS partition's usable sectors.  Inside the guest, the virtual disks were formatted using Windows Disk Management as GPT dynamic non-boot disks and then set up into a RAID5 array.


      What happened...

           A week or so ago during a VM backup/cleanup, I accidentally deleted a 2nd guest VM that also accessed those datastores (I was testing Windows Server 2012R2, unimpressed) without unlinking the datastores from the 2nd VM's config first, and ended up deleting the VMDK files from the disk.  I am trying to recover the flat.vmdk files on at least 2 of these disks.  Since this happened, I have spent many hours searching online about recovering VMDKs and running disk scans attempting to find the missing VMDK (as yet unsuccessfully). Nothing has been written to any of these disks, and the VMFS partition is still in-tact.


      What I know...

           Disk geometry, file names, logical sector size, and number of sectors of the missing VMDK files (retrieved from the log file of the VM's last successful boot)

           Using the above info, I have a template created for a replacement VMDK descriptor file ready once I can extract the flat.vmdk images


      What I have tried so far...

           ESXi Shell & vmfs-fuse -> Browsing datastores shows the volumes are empty.  vmfs-fuse mounts the volume, but no VMDK files can be found.

           UFS Explorer -> Same basic idea.  VMFS partitions are found easily and show as healthy, but I can't seem to find the files there, either.

           Using UFS Explorer's hex view, I have spot-checked several areas that I know are inside the data area and verified the RAID5 parity checks out (confirmed), so it appears the data is still there and in-tact.

           DiskInternals VMFS Recovery -> Attempted to scan the disk, gave up after 36 hrs of scanning on a single disk still hadn't reached 10%.


      What I would like to do...

           Find the starting location for the flat.vmdk file for each disk and do a binary image dump from that position to the end of the VMDK file (end sector calculated by start location + number of sectors for the      VMDK from the log).  Since all 3 disks are identical in the VMWare sense, this starting location should be the same for all 3 disks.  I have a 3TB external HD on standby for the dump once I know where to      start.


           * The trouble I am having is actually locating the starting position in the hex view (I don't know what hex values I need to be searching for).  From Sanbarrow's (continuum) site (VMDK-basics), he mentioned that he searches for the mbr as the file's header, but I haven't found that in my disks.


      Final note 1:  This is NOT a corporate or business system.  It is my personal home server and I have legitimate licenses for all OS's running on the host.


      Final note 2:  Yes, I *DO* have a backup, but it is about 2 months old and missing about 300GB of files that were added since the last backup.  I can recover a fair portion of the missing data (about 60GB) by going back to the source and re-copying/downloading the files, but there is still approx. 240GB that I can't re-copy (source no longer exists).  My fault for not having more current backups, but worst-case is still not a TOTAL loss (approx 85% recovered from backup & original sources).  All guest system disks VMDKs are stored on a separate SSD and have known good backups of the primary system disks on a separate drive).


      Any assistance with this would be very much appreciated!




        • 1. Re: Help recovering thick-provisioned flat.vmdk files needed!
          openDelphi Lurker

          More info about the host system:


          The system has 5 HDDs connected via SATA ports on the motherboard, and ESXi boots from a 4GB USB stick.  The primary datastore for OS disks/partitions is a 240GB SSD (for super-quick guest booting/restarts) and there is a 2nd 240GB disk that I use to do periodic copies of the guest VMs (whole folder while VM is powered off) from the SSD in case recovery of the system partition is needed.  The other 3 HDDs are the 1TB drives mentioned above, and were added to the guest VM as independent & persistent disks.  Due to my method of system backups, there is no need to worry about incorporating snapshots or delta VMDKs on the disks I'm trying to recover.

          • 2. Re: Help recovering thick-provisioned flat.vmdk files needed!
            openDelphi Lurker



            So, after several weeks and countless hours of my best effort, I finally decided to give up and call it quits.  I finally reformatted the HDDs and rebuilt the RAID from scratch - it is in the process of the initial sync now and I will fully restore my last backup to the array as soon as that phase completes.  The most important things I have known and confirmed backups of already, though I'll never recover the archival images I had made of previous laptops, but at least it's not a TOTAL loss.  I haven't gone through them in a long time so I doubt there's anything of  super importance, but it still bugs the stew out of me. 


            What bugs me more, though, is the lack of any response at all here.  I read up on previous posts on the subject, described the situation in detail as well as what I needed assistance with, and after 3 weeks and 60 views... nothing.  I thought VMware would have a much more active, responsive, and helpful community than this, and must say I'm very disappointed.  I hope anyone else who may find themselves in this situation is able to get the help they need.


            I hope everyone has a great 2014!

            • 3. Re: Help recovering thick-provisioned flat.vmdk files needed!
              weinstein5 Guru

              I apologize - the community is usually more reponsive and especially the moderators - I think the issue is the location that this was posted as the VMDK doctors is not a widely viewed forum - I always recommend starting at a more generic level and I think if you had asked your question in the Hypervisor level (e.g. ESXi5 or ESXi4)


              I hope this will not stop you from asking a question in the VMTN community again - I am sure your experience will be much better

              1 person found this helpful
              • 4. Re: Help recovering thick-provisioned flat.vmdk files needed!
                openDelphi Lurker



                Thanks for the response.  I've been reading (and searching) forums for many years now (more than I care to count), and I know one of the biggest gripes across them all are posts in the wrong section, so I was trying to avoid that.  I knew going in that it wasn't a high-traffic sub-forum, but I was figuring someone would at least browse by with a hint, idea, or suggestion (like your recommendation of re-posting in a different forum) within a couple weeks.  I would definitely have done so after your post, but unfortunately the drives have been wiped and over-written as part of my RAID re-build, so there's no going back now.


                I've been using the free ESXi as my home network host for well over a year now and have loved it - I can't really complain.  At this point, it's simply a lesson learned to do a better job at backing things up, I can't blame VMWare or anyone here for that.  Sadly, I am now considering other hypervisor options due to severe limitations (like requiring the web client to edit/manage HW version 10 VMs, not available in the free edition) in the new 5.5 version and the huge cost of the paid versions of ESXi.  I really wish they'd make a home edition and sell it for $100-200 or so.  Limit to 1 host with 64gb RAM, 12-16 vCPU/guest, no vMotion, HA, failover clustering, etc., and restrict it to non-commercial use only.  I'd jump on that in a heartbeat, and I'm sure a TON of other IT (and technically-inclined) people would, too.


                Thanks again for the words of support, I really do appreciate that you responded!