Hello everyone,
I am in a dire need of some assistance! My tech and I was having an issue restoring an Exchange VM (something to do with VSS) and not enough space on the datastore. We decided to free up some space by deleting some old files from out of the NFS storage that is being shared with two ESXi hosts. Well here is where things went bad.
Apparently some files inside one of those folders were somehow being used by our Domain File Server (DFS)/Printer Server. After they were deleted that server will not boot up and reference that it needs that '.vmdk' file. We lost access to all files located in our Netapp storage as well as access to the printer server for printing. We searched our backups Netbackup 7 (outdated I know) which backs up our Netapps storage volumes.
We tried finding the /vmfs volume on the backup tapes but could not locate them - I think we need some sort of treasure map!!! We found the regular user files that are stored in NFS but no VMs and none of the ESX related folders.
Question - Is there some way to recover what we have lost even though we deleted the files? I saw some tools such as VMFS Recovery?
I am extremely desperate.....
Thanks for any assistance.
Ed
I am afraid you have to use 3rd party recovery software, I had experience with below which worked.
https://www.diskinternals.com/vmfs-recovery/
Two quick questions:
André
Andé,
I will have to check whether there are NFS volume having storage based snapshots. We took over the network from a previous IT team and slowly understanding everything. Not the best of times for us but we move forward.
Is there a quick way to check if they are?
The VM was powered on at the time the files were deleted. The folder remains in the directory and there are two files still remaining. If I am not mistaken they are .vmdk Descriptor files but I will not be able to collect information until Monday. Below is what it said when trying to power on that vm server:
"Failed to power on virtual machine <vm name>. File /vmfs/volumes/b37bd535-f5dc2f75/<vm name>_1.00000.vmdk was not found.
Click here for more details."
Thanks again,
Ed
For the storage snapshots, I can't help you much without knowing configuration details, so that's something you have to find out first.
Regarding the .vmdk files. As you may know, on ESXi each virtual disk consists of two files, a descriptor file (e.g. "<vmname>.vmdk"), and a data file (e.g. "<vmname>-flat.vmdk"). If the flat files are still in place, it should be possible to recreate the descriptor files, which only contain some metadata like the disk's provisioned size, geometry, ... Note that in case the VM has/had active snapshots (the file name that you posted makes me think that there are snapshots) there may also exist delta, or sesparse files.
Anyway, if such files are still present, then please check whether there are still some older vmware*.log files in the VM's home folder, which contain details about the VM's virtual disks. These files will help to recreate missing descriptor files.
André
Good Morning André,
Below are the two files that remained in the folder directory. It looks as if the -flat.vmdk is quite large and I am hoping that the files can be restored from it.
I checked for .log files but I did not see any of those.
Without the .log files is it still possible to rebuild what was deleted?
Since the files are stored on NFS, they are thin provisioned, so the current files sizes do not represent the virtual disk's provisioned size, i.e. the size that showed up in the VM's settings. That's why it is important to find some log files, which - as mentioned before - are usually stored in the VM's home folder that also contains the VM's configuration (.vmx) file. Please double-check whether you can find log files there, or in recent backups. Other than this, please run ls -lisa in the VM's folder ("/vmfs/volumes/b37bd535-f5dc2f75/" according to an earlier reply) , and post the result.
Regarding backups. You mentioned that you were searching for "/vmfs" in your backups. The "/vmfs/volumes" folder only exists on ESXi where the different datastores are mounted, so a better approach is to search for .vmdk files.
From the presented files' time stamps it seems that these .vmdk files were not active for quite some time, so if there's a way to restore the missing descriptor files, expect some data loss.
André
[root@vsvh-glc-uea02:/vmfs/volumes/b37bd535-f5dc2f75] ls -lisa
total 336976
64 4 drwxr-xr-x 17 root root 4096 Jul 29 16:19 .
73686543891504 0 drwxr-xr-x 1 root root 512 Aug 4 12:40 ..
3465 4 drwxr-xr-x 2 root root 4096 Aug 4 12:37 .iorm.sf
3466 8 -rwxr-xr-x 1 root root 16511 Aug 4 12:37 .iormstats.sf
13945 4 -rwxrwxr-x 1 root root 84 Mar 29 2017 .lck-7836000000000000
3468 4 -rwxrwxr-x 1 root root 84 Aug 4 12:37 .lck-8a0d000000000000
67 4 drwxrwxrwx 2 root root 4096 Jan 28 2015 .snapshot
103 4 drwx------ 5 root root 4096 May 26 13:10 .vSphere-HA
102 4 drwxr-xr-x 2 root root 4096 Jul 31 12:19 GLC-HOST-02.MDGLC.COM
30063 4 drwxr-xr-x 3 root root 4096 Jun 19 2019 ISOs
13944 336888 -rw------- 1 root root 343615488 Oct 5 2016 VMware-VMvisor-Installer-5.5.0.update01-1623387.x86_64.iso
962 4 drwxr-xr-x 2 root root 4096 Aug 3 23:10 asrm-glc-uea02
945 4 drwxr-xr-x 2 root root 4096 Aug 3 23:13 dsdh-glc-uea01
28219 4 drwxr-xr-x 2 root root 4096 Aug 3 23:28 dsdh-glc-uea02
2858 8 drwxr-xr-x 2 root root 8192 Aug 4 00:30 emdr-glc-uea01
2857 8 drwxr-xr-x 2 root root 8192 Jul 8 08:04 emvm-glc-uea01
2871 4 drwxr-xr-x 2 root root 4096 Aug 4 00:08 emvm-glc-uea03
28203 4 drwxr-xr-x 2 root root 4096 Aug 4 02:18 msmr-glc-uea01
2856 4 drwxr-xr-x 2 root root 4096 Aug 4 00:46 mums-glc-uea01
964 4 drwxr-xr-x 2 root root 4096 Aug 4 00:52 mums-glc-uea02
96 4 drwxr-xr-x 2 root root 4096 May 8 2015 wsl4.2
Sorry, my bad, it's "/vmfs/volumes/b37bd535-f5dc2f75/GLC-HOST-02.MDGLC.COM" which contains the files.
André
[root@vsvh-glc-uea02:/vmfs/volumes/b37bd535-f5dc2f75/GLC-HOST-02.MDGLC.COM] ls -lisa
total 756236788
102 4 drwxr-xr-x 2 root root 4096 Jul 31 12:19 .
64 4 drwxr-xr-x 18 root root 4096 Aug 4 13:29 ..
23709 2468 -rw------- 1 root root 19070976 Jul 17 20:42 GLC-HOST-02.MDGLC.COM_1-000001-delta.vmdk
I also was recommended to use the DiskInternals VMFS Recovery software. Have you had experience with that?
Where's the flat.vmdk file that showed up ion the previously presented screenshot?
I also was recommended to use the DiskInternals VMFS Recovery software. Have you had experience with that?
When it comes to data recovery from disks/LUNs, then continuum may need to take over.
André
I did the ls -lisa in that folder.
[root@vsvh-glc-uea02:/vmfs/volumes/b37bd535-f5dc2f75/GLC-HOST-02.MDGLC.COM] ls -lisa
total 756236788
102 4 drwxr-xr-x 2 root root 4096 Jul 31 12:19 .
64 4 drwxr-xr-x 18 root root 4096 Aug 4 13:29 ..
23709 2468 -rw------- 1 root root 19070976 Jul 17 20:42 GLC-HOST-02.MDGLC.COM_1-000001-delta.vmdk
31471 756234312 -rw------- 1 root root 1199865593856 Jun 25 14:28 GLC-HOST-02.MDGLC.COM_1-flat.vmdk
Ok, then let's see whether we can find out the virtual disk's details from the metadata in the delta file.
For this, please extract the first 2048 Bytes from the delta file, and attach the extracted metadata.bin file to your next reply post. You may use e.g. the following command:
dd if=GLC-HOST-02.MDGLC.COM_1-000001-delta.vmdk of=metadata.bin bs=1024 count=2
André
Good Morning André,
I was able to extract the .bin file as you recommended. I did not know what to open it up in to view it.
I attached it here.
I ran a command od -cx metadata.bin
[root@vsvh-glc-uea02:/vmfs/volumes/b37bd535-f5dc2f75/GLC-HOST-02.MDGLC.COM] od -cx metadata.bin
0000000 C O W D 001 \0 \0 \0 003 \0 \0 \0 \0 0I0 0E6 0A3
4f43 4457 0001 0000 0003 0000 c800 8bae
0000020 001 \0 \0 \0 004 \0 \0 \0 \0 0G3 \b \0 8 022 \0 \0
0001 0000 0004 0000 bb00 0008 1238 0000
0000040 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000 0000 0000 0000 0000 0000 0000 0000
*
0003160 001 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0001 0000 0000 0000 0000 0000 0000 0000
0003200 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000 0000 0000 0000 0000 0000 0000 0000
*
0004000
I downloaded the trial version and was able to mount and look into the datastore. I was not able to find the deleted files but that is probably due to me not knowing exactly where to look for deleted files.
Is there a specific place they could be housed?
Just for verification that I re-create the descriptor files correctly, can you please paste the contents of the .vmdk descriptor file for another VM that has its files stored on the NAS share?
André
So be be sure that I collect what you are asking for I will run the same command on another VMs -delta.vmdk file?
dd if=<new VM>-delta.vmdk of=metadata.bin bs=1024 count=2
What I need is a .vmdk descriptor file only, i.e. the small (a few hundred bytes) text file without delta, of flat in its name.
It's usually hidden on the vSphere Client GUI, so you need to access it from e.g. the command line. Simply run type <vmname>.vmdk and provide the command's output.
Btw. which ESXi version do you use? I'm asking because of the virtual hardware version.
André
I've recreated the two missing descriptor .vmdk files (see attached file).
Please upload the two .vmdk descriptor files to the datatstore, on which you have the flat, and delta files. Then verify that the VM's configuration (Edit Settings) points to GLC-HOST-02.MDGLC.COM_1-000001.vmdk, i.e. the current snapshot. Prior to powering on the VM, I'd suggest that you take another snasphot, so the the current .vmdk files won't get modified, and you can return to that state in case something doesn't work as expected.
André