ESX vmfs deleted

[mredwarren]

Hello everyone,

I am in a dire need of some assistance!  My tech and I was having an issue restoring an Exchange VM (something to do with VSS) and not enough space on the datastore. We decided to free up some space by deleting some old files from out of the NFS storage that is being shared with two ESXi hosts. Well here is where things went bad.

Apparently some files inside one of those folders were somehow being used by our Domain File Server (DFS)/Printer Server. After they were deleted that server will not boot up and reference that it needs that '.vmdk' file. We lost access to all files located in our Netapp storage as well as access to the printer server for printing. We searched our backups Netbackup 7 (outdated I know) which backs up our Netapps storage volumes.

We tried finding the /vmfs volume on the backup tapes but could not locate them - I think we need some sort of treasure map!!! We found the regular user files that are stored in NFS but no VMs and none of the ESX related folders.

Question - Is there some way to recover what we have lost even though we deleted the files? I saw some tools such as VMFS Recovery?

I am extremely desperate.....

Thanks for any assistance.

Ed

[amohammadimir]

I am afraid you have to use 3rd party recovery software, I had experience with below which worked.

https://www.diskinternals.com/vmfs-recovery/

[a_p_]

Two quick questions:

1. You mentioned Netapp. Does the NFS volume have storage based snapshots created, which may still contain the deleted files?
2. Did you delete the files while the VMs were powered on? In that case, it may be possible that only the descriptor files got deleted (which can usually be recreated), because the data files were in use (locked)!? In case there are still files in the VMs' folders, please provide an overview (including names, extensions, time stamps, sizes).

André

[mredwarren]

Andé,

I will have to check whether there are NFS volume having storage based snapshots. We took over the network from a previous IT team and slowly understanding everything. Not the best of times for us but we move forward.

Is there a quick way to check if they are?

The VM was powered on at the time the files were deleted. The folder remains in the directory and there are two files still remaining. If I am not mistaken they are .vmdk Descriptor files but I will not be able to collect information until Monday. Below is what it said when trying to power on that vm server:

"Failed to power on virtual machine <vm name>. File /vmfs/volumes/b37bd535-f5dc2f75/<vm name>_1.00000.vmdk was not found.

Click here for more details."

Thanks again,

Ed

[a_p_]

For the storage snapshots, I can't help you much without knowing configuration details, so that's something you have to find out first.

Regarding the .vmdk files. As you may know, on ESXi each virtual disk consists of two files, a descriptor file (e.g. "<vmname>.vmdk"), and a data file (e.g. "<vmname>-flat.vmdk"). If the flat files are still in place, it should be possible to recreate the descriptor files, which only contain some metadata like the disk's provisioned size, geometry, ... Note that in case the VM has/had active snapshots (the file name that you posted makes me think that there are snapshots) there may also exist delta, or sesparse files.

Anyway, if such files are still present, then please check whether there are still some older vmware*.log files in the VM's home folder, which contain details about the VM's virtual disks. These files will help to recreate missing descriptor files.

André

[mredwarren]

Good Morning André,

Below are the two files that remained in the folder directory. It looks as if the -flat.vmdk is quite large and I am hoping that the files can be restored from it.

I checked for .log files but I did not see any of those.

Without the .log files is it still possible to rebuild what was deleted?

[a_p_]

Since the files are stored on NFS, they are thin provisioned, so the current files sizes do not represent the virtual disk's provisioned size, i.e. the size that showed up in the VM's settings. That's why it is important to find some log files, which - as mentioned before - are usually stored in the VM's home folder that also contains the VM's configuration (.vmx) file. Please double-check whether you can find log files there, or in recent backups. Other than this, please run ls -lisa in the VM's folder ("/vmfs/volumes/b37bd535-f5dc2f75/" according to an earlier reply) , and post the result.

Regarding backups. You mentioned that you were searching for "/vmfs" in your backups. The "/vmfs/volumes" folder only exists on ESXi where the different datastores are mounted, so a better approach is to search for .vmdk files.

From the presented files' time stamps it seems that these .vmdk files were not active for quite some time, so if there's a way to restore the missing descriptor files, expect some data loss.

André

[mredwarren]

[root@vsvh-glc-uea02:/vmfs/volumes/b37bd535-f5dc2f75] ls -lisa

total 336976

     64      4 drwxr-xr-x   17 root     root          4096 Jul 29 16:19 .

73686543891504      0 drwxr-xr-x    1 root     root           512 Aug  4 12:40 ..

   3465      4 drwxr-xr-x    2 root     root          4096 Aug  4 12:37 .iorm.sf

   3466      8 -rwxr-xr-x    1 root     root         16511 Aug  4 12:37 .iormstats.sf

  13945      4 -rwxrwxr-x    1 root     root            84 Mar 29  2017 .lck-7836000000000000

   3468      4 -rwxrwxr-x    1 root     root            84 Aug  4 12:37 .lck-8a0d000000000000

     67      4 drwxrwxrwx    2 root     root          4096 Jan 28  2015 .snapshot

    103      4 drwx------    5 root     root          4096 May 26 13:10 .vSphere-HA

    102      4 drwxr-xr-x    2 root     root          4096 Jul 31 12:19 GLC-HOST-02.MDGLC.COM

  30063      4 drwxr-xr-x    3 root     root          4096 Jun 19  2019 ISOs

  13944 336888 -rw-------    1 root     root     343615488 Oct  5  2016 VMware-VMvisor-Installer-5.5.0.update01-1623387.x86_64.iso

    962      4 drwxr-xr-x    2 root     root          4096 Aug  3 23:10 asrm-glc-uea02

    945      4 drwxr-xr-x    2 root     root          4096 Aug  3 23:13 dsdh-glc-uea01

  28219      4 drwxr-xr-x    2 root     root          4096 Aug  3 23:28 dsdh-glc-uea02

   2858      8 drwxr-xr-x    2 root     root          8192 Aug  4 00:30 emdr-glc-uea01

   2857      8 drwxr-xr-x    2 root     root          8192 Jul  8 08:04 emvm-glc-uea01

   2871      4 drwxr-xr-x    2 root     root          4096 Aug  4 00:08 emvm-glc-uea03

  28203      4 drwxr-xr-x    2 root     root          4096 Aug  4 02:18 msmr-glc-uea01

   2856      4 drwxr-xr-x    2 root     root          4096 Aug  4 00:46 mums-glc-uea01

    964      4 drwxr-xr-x    2 root     root          4096 Aug  4 00:52 mums-glc-uea02

     96      4 drwxr-xr-x    2 root     root          4096 May  8  2015 wsl4.2

[a_p_]

Sorry, my bad, it's "/vmfs/volumes/b37bd535-f5dc2f75/GLC-HOST-02.MDGLC.COM" which contains the files.

André

[mredwarren]

[root@vsvh-glc-uea02:/vmfs/volumes/b37bd535-f5dc2f75/GLC-HOST-02.MDGLC.COM] ls -lisa

total 756236788

    102      4 drwxr-xr-x    2 root     root          4096 Jul 31 12:19 .

     64      4 drwxr-xr-x   18 root     root          4096 Aug  4 13:29 ..

  23709   2468 -rw-------    1 root     root      19070976 Jul 17 20:42 GLC-HOST-02.MDGLC.COM_1-000001-delta.vmdk

I also was recommended to use the DiskInternals VMFS Recovery software. Have you had experience with that?

[a_p_]

Where's the flat.vmdk file that showed up ion the previously presented screenshot?

I also was recommended to use the DiskInternals VMFS Recovery software. Have you had experience with that?

When it comes to data recovery from disks/LUNs, then continuum​ may need to take over.

André

[mredwarren]

I did the ls -lisa in that folder.

[root@vsvh-glc-uea02:/vmfs/volumes/b37bd535-f5dc2f75/GLC-HOST-02.MDGLC.COM] ls -lisa

total 756236788

    102      4 drwxr-xr-x    2 root     root          4096 Jul 31 12:19 .

     64      4 drwxr-xr-x   18 root     root          4096 Aug  4 13:29 ..

  23709   2468 -rw-------    1 root     root      19070976 Jul 17 20:42 GLC-HOST-02.MDGLC.COM_1-000001-delta.vmdk

  31471 756234312 -rw-------    1 root     root     1199865593856 Jun 25 14:28 GLC-HOST-02.MDGLC.COM_1-flat.vmdk

[a_p_]

Ok, then let's see whether we can find out the virtual disk's details from the metadata in the delta file.

For this, please extract the first 2048 Bytes from the delta file, and attach the extracted metadata.bin file to your next reply post. You may use e.g. the following command:

dd if=GLC-HOST-02.MDGLC.COM_1-000001-delta.vmdk of=metadata.bin bs=1024 count=2

André

[mredwarren]

Good Morning André,

I was able to extract the .bin file as you recommended. I did not know what to open it up in to view it.

I attached it here.

I ran a command od -cx metadata.bin

[root@vsvh-glc-uea02:/vmfs/volumes/b37bd535-f5dc2f75/GLC-HOST-02.MDGLC.COM] od -cx metadata.bin

0000000   C   O   W   D 001  \0  \0  \0 003  \0  \0  \0  \0 0I0 0E6 0A3

        4f43 4457 0001 0000 0003 0000 c800 8bae

0000020 001  \0  \0  \0 004  \0  \0  \0  \0 0G3  \b  \0   8 022  \0  \0

        0001 0000 0004 0000 bb00 0008 1238 0000

0000040  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0

        0000 0000 0000 0000 0000 0000 0000 0000

*

0003160 001  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0

        0001 0000 0000 0000 0000 0000 0000 0000

0003200  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0

        0000 0000 0000 0000 0000 0000 0000 0000

*

0004000

[mredwarren]

I downloaded the trial version and was able to mount and look into the datastore. I was not able to find the deleted files but that is probably due to me not knowing exactly where to look for deleted files.

Is there a specific place they could be housed?

[a_p_]

Just for verification that I re-create the descriptor files correctly, can you please paste the contents of the .vmdk descriptor file for another VM that has its files stored on the NAS share?

André

[mredwarren]

So be be sure that I collect what you are asking for I will run the same command on another VMs -delta.vmdk file?

dd if=<new VM>-delta.vmdk of=metadata.bin bs=1024 count=2

[a_p_]

What I need is a .vmdk descriptor file only, i.e. the small (a few hundred bytes) text file without delta, of flat in its name.

It's usually hidden on the vSphere Client GUI, so you need to access it from e.g. the command line. Simply run type <vmname>.vmdk and provide the command's output.

Btw. which ESXi version do you use? I'm asking because of the virtual hardware version.

André

[mredwarren]

Unfortunately we are still using ESXi 6.0 taken over from a previous IT management group who prolonged updating it (politics).

[a_p_]

I've recreated the two missing descriptor .vmdk files (see attached file).

Please upload the two .vmdk descriptor files to the datatstore, on which you have the flat, and delta files. Then verify that the VM's configuration (Edit Settings) points to GLC-HOST-02.MDGLC.COM_1-000001.vmdk, i.e. the current snapshot. Prior to powering on the VM, I'd suggest that you take another snasphot, so the the current .vmdk files won't get modified, and you can return to that state in case something doesn't work as expected.

André