TroySchulz
Enthusiast
Enthusiast

Can you recover an accidentally deleted VMDK file?

Jump to solution

I was donated a Dell T420 server which I turned into a web server. I've installed ESXi 6 into which I created a Linux Mint LAMP server for hosting a few non profits and a few small business owners I know for free. This is a hobby level thing - not for money. I am very new to all of this.

There is a pair of 500GB SAS drives in this. One is Datastore 1 and the other Datastore 2. The web server is installed in Datastore 1.

At one point the web server crashed and it appeared I have run out of disk space so it wouldn't reboot. After reading a lot online it was suggested to see if removing one or more snapshots might not free up the space. I only had two snapshots in place. I deleted the oldest one. However, this did not appear to free up enough space. I did not want to delete my only remaining snapshot.

It was then suggested that during the process the associated VMDK files in the datastore may not have been cleaned out properly during the snapshot deletion. I browsed the datastore in which are several different files LOG files, XML files, VMSN, NVRAM, VMSD, VMX, VMXF, and a series of VMDK files, which I've been told would the be the "main" files for the virtual machine.

The files I had listed were:

Webserver0.vmdk

Webserver0-000001.vmdk

Webserver0-000002.vmdk

Webserver0-000003.vmdk

Webserver0-000004.vmdk

Webserver0-000005.vmdk

Webserver0-000006.vmdk <---This one being the newest date.

Not knowing what to do with these I right clicked on the Webserver0-000001.vmdk to see what was available and saw a dropdown menu of options. However, I may have accidentally double clicked and hit the "delete" option from the menu. Before I knew it that file was gone. I then could not boot the VM after. All I get is an error message:

Failed to power on virtual machine WebServer. File /vmfs/volumes/58866b0f-0a0319b0-38e0-e0db55027e66/WebServer/WebServer0-000001.vmdk was not found

I looked back in the datastore and now in place of that file is a Webserver0-000001-delta.vmdk file instead.

I assume all these files are required for the VM to operate? Have I just royally screwed myself and all these people I've been hosting? Is there a way to recover or rebuild that file and restore this machine? I do not have a backup since Veeam said their products won't work with free ESXi. I also have no money and cannot afford to pay a support ticket or buy recovery software that cost $699 at minimum!

I need help desperately!

1 Solution

Accepted Solutions
a_p_
Leadership
Leadership

How much free disk space do you have on the other datastore (Datastore2)? What we need is ~200GB free disk space.

If you have that amount of free disk space on Datastore2, then you may follow the below steps to clone the Webserver VM to Datastore2.

  1. create a new folder "Webserver" on the target Datastore2
  2. remove the VM from the ESXi host's inventory (DON'T use delete from diks)
  3. clone the Webserver VM's virtual disk from the source Datstore1 to the target Datsatore2 (this will consolidate all snapshots)
    vmkfstools -i "/vmfs/volumes/Datastore1/Webserver/WebServer0-000006.vmdk" "/vmfs/volumes/Datastore2/Webserver/WebServer.vmdk"
  4. copy the VM's .vmxf, and .nvram files from the old folder to the new one
  5. upload the attached configuration (.vmx) file to the new folder - I've modified this file to point to the new .vmdk file name
  6. right click the .vmx file in the new folder, and select "Add to Inventory"

You should now be able to power on the VM. Note that you may be asked whether you copied, or moved the VM at first power on. In this case select "I moved it".

Once everything works as expected you may procedd with deleting the "old" VM's folder on Datstore1.

Remember that each single snapshot can grow up to the virtual disk's provisioned size (200GB in this case). Best practice is to avoid snapshots on production VMs, i.e. only use them if needed, e.g. before making critical updates to have a way back in case it's necessary.

André

PS: I assume you've already fixed the missing .vmdk descriptor file issue.

View solution in original post

0 Kudos
28 Replies
ThompsG
Virtuoso
Virtuoso

Hi Troy and welcome to the community!

From what you have described it appears that you might have been fortunate and have removed the VMDK descriptor file rather than the actual data component. Now while I could have a go at rebuilding, your best bet is this gentleman: continuum

If you check his signature, you can contact him on Skype: sanbarrow

In the meantime could you run the following from the console in the directory containing the VM:

ls -lsh

Post the output here please. Also attach the vmx file if that is okay to do.

Also whatever you do, do not be tempted to attach the base disk directly to another VM or remove references to the snapshots. If the parent disk changes then you are possibly a much worse position.

Kind regards.

TroySchulz
Enthusiast
Enthusiast
In the meantime could you run the following from the console in the directory containing the VM:

I'm sorry, but I am very new to this. Where do I go to do this?

I'm signed into the server remotely through the web client, but don't see where I can enter a console from here? Or, do I have to plug a monitor, mouse, and keyboard into the server and directly interface with it that way?

I have attached the vmx file as requested. Thank you so much for your quick response and assistance! I'm still freaking out about this.

0 Kudos
SureshKumarMuth
Commander
Commander

1. Launch ssh session(using putty) to the ESXi host

2. type the following

cd /vmfs/volumes/<datastore name>/<vm directory>

ls -l | grep -i vmdk

please post the output, if ssh service is stopped , you may not be able to launch putty session. To start the ssh service , launch ESXi host client from browser

https://<ESXi ip>/ui

Then go to configure tab --> security profile --> Services --> there you can start the ssh service

Regards, Suresh https://vconnectit.wordpress.com/
SureshKumarMuth
Commander
Commander

Also, please do not attempt to delete any other files, as per the error message it looks like you have deleted only the descriptor file i.e 0006.vmdk , we can recreate this file using the available delta file.

Basically , each virtual disk will have two files

1. Descriptor file

2. Data file

for a disk without snapshot , you will have the file in this naming convention (taken your server name as example)

Webserver0.vmdk -- is the descriptor file of base disk

Webserver0-flat.vmdk -- actual data file ( if we delete this then we cannot recover data if we do not have any proper back up or using proper recovery tool)

In case if you have snapshot , you will find many disks in the following naming convention depends on the number of snapshots you have . in your case if appears that you have 6 snapshots

Webserver0-000001.vmdk -- descriptor file, can be recovered using the delta disk.

Webserver0-000001-delta.vmdk (data disk)

In your case, fortunately you deleted only the descriptor file which can be recovered using the steps given in the following video and kb article

How to recreate a missing Virtual Machine Disk Descriptor File (VMDK) - YouTube

VMware Knowledge Base

Waiting for the output from ESXi ssh session for further steps

Regards, Suresh https://vconnectit.wordpress.com/
0 Kudos
continuum
Immortal
Immortal

Please do not recommend that video again:
How to recreate a missing Virtual Machine Disk Descriptor File (VMDK) - YouTube

Following that bad advice can make matters worse - I have seen that way too often.

@ Troy
please connect to your ESXi via WinSCP and download all descriptor-vmdks of that VM.
That are the small vmdks with -flat, -ctk, -delta or -sesparse on the name.
Infortunately the Datastorebrowser will not display them correctly so you need to use WinSCP.
Zip them together and attach them to your next reply.
I will create the missing file for you.
Ulli

Do you need support with a recovery problem ? - send a message via skype "sanbarrow"
a_p_
Leadership
Leadership

There a re couple of things that need to be clarified to recommend the proper steps to resolve the issue.

Some things have already been said, but let me put everything together.

  1. enable SSH on the host, and use e.g. putty to connect to it
  2. go to the VM's folder (cd /vmfs/volumes/<datastore-name>/<folder-name>/) and run
    ls -elisa > filelist.txt
  3. use e.g. WinSCP to connect to the host, and download the following files from the VM's folder
    filelist.txt, *.log, *.vmsd, *.vmdk (only the small descriptor .vmdk files without flat, delta, or sesparse in their names)
  4. compress/zip the downloaded files, and attach the .zip archive to a reply post

How much free disk space do you currently have on the datastore?

André

ThompsG
Virtuoso
Virtuoso

Hi Troy,

No problems and sorry I should have provided the steps to get this file listing via a PuTTY session. Thanks SureshKumarMuthukrishnan for doing this.

I see continuum has responded so I will revert to just watching this thread and let him work with you. You are in good hands with his advice and I recommend just following what he says. a.p. has replied as well so definitely in good company there.

NOTE: not saying others cannot or will not offer good advice, just that it’s probably better to follow only one when dealing with potential data loss. Don’t want potential cross contamination of tasks Smiley Wink

Kind regards.

0 Kudos
gregsn
Enthusiast
Enthusiast

I do not have a backup since Veeam said their products won't work with free ESXi.

Try Nakivo.  It works with the free version of ESXi and the free version of Nakivo will backup up to two virtual machines.

0 Kudos
TroySchulz
Enthusiast
Enthusiast

-rw-------    1 root     root     5788553216 Feb  6  2017 WebServer0-000001-delta.vmdk

-rw-------    1 root     root     25032019968 Feb 22  2017 WebServer0-000002-delta.vmdk

-rw-------    1 root     root           331 Feb  6  2017 WebServer0-000002.vmdk

-rw-------    1 root     root     80111620096 Jul 14  2017 WebServer0-000003-delta.vmdk

-rw-------    1 root     root           358 Jul 14  2017 WebServer0-000003.vmdk

-rw-------    1 root     root     105864646656 Oct 22  2017 WebServer0-000004-delta.vmdk

-rw-------    1 root     root           331 Jul 14  2017 WebServer0-000004.vmdk

-rw-------    1 root     root     109454970880 Jul  5  2018 WebServer0-000005-delta.vmdk

-rw-------    1 root     root           331 Jun 18  2018 WebServer0-000005.vmdk

-rw-------    1 root     root     121953996800 Feb 25 23:58 WebServer0-000006-delta.vmdk

-rw-------    1 root     root           358 Feb 25 23:26 WebServer0-000006.vmdk

-rw-------    1 root     root     214748364800 Jan 30  2017 WebServer0-flat.vmdk

-rw-------    1 root     root           527 Jan 30  2017 WebServer0.vmdk

0 Kudos
TroySchulz
Enthusiast
Enthusiast

It looks like WinSCP is just an FTP program? I don't have Windows. The only other computer I have available is Linux Mint. I will attempt to use Filezilla or another FTP program to download those and will post as soon as I can. Thanks!!

0 Kudos
TroySchulz
Enthusiast
Enthusiast

I have attached a ZIP file with all the smallish files I could find. I hope this is all you need. I soooo appreciate your assistance! What a wonderful community!

0 Kudos
TroySchulz
Enthusiast
Enthusiast

There's a physical 500GB hard drive. The virtual machine is set for 200GB. However, there is only 11MB free in the datastore.

0 Kudos
a_p_
Leadership
Leadership

I've attached a recreated descriptor file.

However, this won't help much due to the low disk space.

According to the descriptor files, the VM's disk has been thin provisioned, i.e. will most likely grow when you try to delete snapshots, so don't try this at the moment.

Please post the output for ls -elisa, as this will show not only the files' provisioned disk space, but also the used space. Base on this we need to find out how to resolve this issue.

How much free disk space do you have on the other datastore? I'm asking to find out whether it is an option to clone the virtual disk from the command line.

André

continuum
Immortal
Immortal

@ Glen
I received a PM from you with the following question:

Trust you don't mind but I've been playing along at home and this stuff interests me Smiley Happy

Would the descriptor file you are going to create look like this (named "WebServer0-000001.vmdk":

# Disk DescriptorFile

version=1

encoding="UTF-8"

CID=e056a0a1

parentCID=1065ec08

isNativeSnapshot="no"

createType="vmfsSparse"

parentFileNameHint="WebServer0.vmdk"

# Extent description

RW 419430400 VMFSSPARSE "WebServer0-000001-delta.vmdk"

# The Disk Data Base #DDB

ddb.longContentID = "914a7521b682d3f525c773e16c3a7f6b"

Thanks in advance!

This is no secret knowhow - and the more folks can handle issues like this correctly the better it is for the community.
So your question is welcome.

Compare your suggestion with the one that Andre uploaded.
It has the exact same values - so your suggestion was spot on.
You may notice that Andre did not include the parameter:
ddb.longContentID = "914a7521b682d3f525c773e16c3a7f6b"
which you suggested to include.
I would have skipped this line as well.
It is optional and skipping it will do no harm so we both simply skip it.

Basically you have 5 parameters in a snapshot-descriptorfile that you have to check:

CID=e056a0a1

parentCID=1065ec08

parentFileNameHint="WebServer0.vmdk"

RW 419430400 VMFSSPARSE "WebServer0-000001-delta.vmdk"

Lets see how to get the values for each of them:
CID=e056a0a1
This value can be looked up in the descriptor of the next child vmdk. It must match the parentCID value of the next child vmdk.
parentCID=1065ec08
This value can be looked up in the descriptor of the parent vmdk. It must match the CID value of the parent vmdk.
parentFileNameHint="WebServer0.vmdk"

This value references the  path to the parent vmdk.. If both vmdks reside in the same directory the filename is enough. If the parent-vmdk resides in a different directory you must use the full path.
RW 419430400
This references the "nominal" size of the VMDK. It gives the size in sectors.This value should be the same for EVERY vmdk in the snapshot-chain.
If not you will receive an error message: SIZE does not match. Parent has been expanded.

VMFSSPARSE "WebServer0-000001-delta.vmdk"

This parameter gives the path to the associated binary file that has the actual data. In 99.99 % of the cases both the descriptor vmdk as well as the binary part reside in the same directory so using the filename will work in almost all cases.

@ Glen
Feel free to contact me via skype if you are interested in this topic.
Ulli

Do you need support with a recovery problem ? - send a message via skype "sanbarrow"
ThompsG
Virtuoso
Virtuoso

continuum​ - you are too kind and thanks for the explanation. Very clear and very helpful.

I will take you up on the offer at some stage because this is something that definitely interests me. Often try to follow the bread crumbs as you guys do the work Smiley Wink

This is one thing I love about the VMware community - the free sharing of knowledge. It’s great to be able to try and give back in some small way especially as you learn along the way!

Thanks again and normal service resumes Smiley Happy

TroySchulz
Enthusiast
Enthusiast

Here's the output you were looking for.

total 791560

      4   1024 drwxr-xr-t    1 root     root          1400 Tue Jan 24 21:50:00 2017 .

73675999881280      0 drwxr-xr-x    1 root     root           512 Thu Feb 28 13:18:17 2019 ..

4194308   3072 -r--------    1 root     root       2490368 Mon Jan 23 20:43:59 2017 .fbb.sf

8388612 261120 -r--------    1 root     root     267026432 Mon Jan 23 20:43:59 2017 .fdc.sf

25165828   2048 -r--------    1 root     root       1179648 Mon Jan 23 20:43:59 2017 .pb2.sf

12582916 262144 -r--------    1 root     root     268435456 Mon Jan 23 20:43:59 2017 .pbc.sf

16777220 257024 -r--------    1 root     root     262733824 Mon Jan 23 20:43:59 2017 .sbc.sf

29360132   1024 drwx------    1 root     root           280 Mon Jan 23 20:44:00 2017 .sdd.sf

20971524   4096 -r--------    1 root     root       4194304 Mon Jan 23 20:43:59 2017 .vh.sf

83890564      8 drwxr-xr-x    1 root     root          4060 Tue Feb 26 22:39:17 2019 WebServer

I have another 500GB drive in this server which is Datastore2. I had copied a bunch of the files from Datastore1 to it, but that's it. I can delete the contents of that datastore so we can use all the space if necessary.

0 Kudos
a_p_
Leadership
Leadership

Here's the output you were looking for.

Not exactly. Sorry if I wasn't clear enough. What I need is this output for the files in the VM's folder.


André

0 Kudos
TroySchulz
Enthusiast
Enthusiast

I'm sorry. I should have gone one more folder deeper. Try this.

total 462701592

83890564      8 drwxr-xr-x    1 root     root          4060 Tue Feb 26 22:39:17 2019 .

      4   1024 drwxr-xr-t    1 root     root          1400 Tue Jan 24 21:50:00 2017 ..

465572228 8388608 -rw-------    1 root     root     8589934592 Thu Jul  5 01:40:24 2018 WebServer-Snapshot8.vmem

461377924   7168 -rw-------    1 root     root       6386305 Thu Jul  5 01:40:24 2018 WebServer-Snapshot8.vmsn

301994372      0 -rw-r--r--    1 root     root            13 Mon Jan 30 22:44:34 2017 WebServer-aux.xml

125833604   1024 -rw-------    1 root     root          8684 Mon Feb 25 23:58:39 2019 WebServer.nvram

100667780      0 -rw-r--r--    1 root     root           438 Mon Feb 25 22:57:43 2019 WebServer.vmsd

88084868      8 -rwxr-xr-x    1 root     root          3256 Mon Feb 25 23:58:39 2019 WebServer.vmx

268439940      8 -rw-------    1 root     root          3283 Mon Feb 25 23:26:01 2019 WebServer.vmxf

314577284 5653504 -rw-------    1 root     root     5788553216 Mon Feb  6 03:25:30 2017 WebServer0-000001-delta.vmdk

331354500 24445952 -rw-------    1 root     root     25032019968 Wed Feb 22 22:47:34 2017 WebServer0-000002-delta.vmdk

335548804      0 -rw-------    1 root     root           331 Mon Feb  6 03:25:35 2017 WebServer0-000002.vmdk

348131716 78234624 -rw-------    1 root     root     80111620096 Fri Jul 14 02:47:18 2017 WebServer0-000003-delta.vmdk

352326020      0 -rw-------    1 root     root           358 Fri Jul 14 02:43:52 2017 WebServer0-000003.vmdk

390074756 103384064 -rw-------    1 root     root     105864646656 Sun Oct 22 22:43:28 2017 WebServer0-000004-delta.vmdk

394269060      0 -rw-------    1 root     root           331 Fri Jul 14 02:47:22 2017 WebServer0-000004.vmdk

406851972 106890240 -rw-------    1 root     root     109454970880 Thu Jul  5 01:38:30 2018 WebServer0-000005-delta.vmdk

411046276      0 -rw-------    1 root     root           331 Mon Jun 18 09:01:35 2018 WebServer0-000005.vmdk

469766532 119096320 -rw-------    1 root     root     121953996800 Mon Feb 25 23:58:39 2019 WebServer0-000006-delta.vmdk

473960836      0 -rw-------    1 root     root           358 Mon Feb 25 23:26:00 2019 WebServer0-000006.vmdk

92279172 16591872 -rw-------    1 root     root     214748364800 Mon Jan 30 22:45:05 2017 WebServer0-flat.vmdk

96473476      0 -rw-------    1 root     root           527 Mon Jan 30 22:44:36 2017 WebServer0.vmdk

578818436   1024 -rw-r--r--    1 root     root        186726 Mon Feb 25 22:55:54 2019 vmware-24.log

595595652   1024 -rw-r--r--    1 root     root        219630 Mon Feb 25 23:58:39 2019 vmware-25.log

612372868   1024 -rw-r--r--    1 root     root         84517 Tue Feb 26 02:54:15 2019 vmware-26.log

629150084   1024 -rw-r--r--    1 root     root         85491 Tue Feb 26 03:07:33 2019 vmware-27.log

645927300   1024 -rw-r--r--    1 root     root         85494 Tue Feb 26 03:07:39 2019 vmware-28.log

662704516   1024 -rw-r--r--    1 root     root         85623 Tue Feb 26 21:17:06 2019 vmware-29.log

679481732   1024 -rw-r--r--    1 root     root         84319 Tue Feb 26 22:39:17 2019 vmware.log

0 Kudos
a_p_
Leadership
Leadership

How much free disk space do you have on the other datastore (Datastore2)? What we need is ~200GB free disk space.

If you have that amount of free disk space on Datastore2, then you may follow the below steps to clone the Webserver VM to Datastore2.

  1. create a new folder "Webserver" on the target Datastore2
  2. remove the VM from the ESXi host's inventory (DON'T use delete from diks)
  3. clone the Webserver VM's virtual disk from the source Datstore1 to the target Datsatore2 (this will consolidate all snapshots)
    vmkfstools -i "/vmfs/volumes/Datastore1/Webserver/WebServer0-000006.vmdk" "/vmfs/volumes/Datastore2/Webserver/WebServer.vmdk"
  4. copy the VM's .vmxf, and .nvram files from the old folder to the new one
  5. upload the attached configuration (.vmx) file to the new folder - I've modified this file to point to the new .vmdk file name
  6. right click the .vmx file in the new folder, and select "Add to Inventory"

You should now be able to power on the VM. Note that you may be asked whether you copied, or moved the VM at first power on. In this case select "I moved it".

Once everything works as expected you may procedd with deleting the "old" VM's folder on Datstore1.

Remember that each single snapshot can grow up to the virtual disk's provisioned size (200GB in this case). Best practice is to avoid snapshots on production VMs, i.e. only use them if needed, e.g. before making critical updates to have a way back in case it's necessary.

André

PS: I assume you've already fixed the missing .vmdk descriptor file issue.

View solution in original post

0 Kudos