I was donated a Dell T420 server which I turned into a web server. I've installed ESXi 6 into which I created a Linux Mint LAMP server for hosting a few non profits and a few small business owners I know for free. This is a hobby level thing - not for money. I am very new to all of this.
There is a pair of 500GB SAS drives in this. One is Datastore 1 and the other Datastore 2. The web server is installed in Datastore 1.
At one point the web server crashed and it appeared I have run out of disk space so it wouldn't reboot. After reading a lot online it was suggested to see if removing one or more snapshots might not free up the space. I only had two snapshots in place. I deleted the oldest one. However, this did not appear to free up enough space. I did not want to delete my only remaining snapshot.
It was then suggested that during the process the associated VMDK files in the datastore may not have been cleaned out properly during the snapshot deletion. I browsed the datastore in which are several different files LOG files, XML files, VMSN, NVRAM, VMSD, VMX, VMXF, and a series of VMDK files, which I've been told would the be the "main" files for the virtual machine.
The files I had listed were:
Webserver0.vmdk
Webserver0-000001.vmdk
Webserver0-000002.vmdk
Webserver0-000003.vmdk
Webserver0-000004.vmdk
Webserver0-000005.vmdk
Webserver0-000006.vmdk <---This one being the newest date.
Not knowing what to do with these I right clicked on the Webserver0-000001.vmdk to see what was available and saw a dropdown menu of options. However, I may have accidentally double clicked and hit the "delete" option from the menu. Before I knew it that file was gone. I then could not boot the VM after. All I get is an error message:
Failed to power on virtual machine WebServer. File /vmfs/volumes/58866b0f-0a0319b0-38e0-e0db55027e66/WebServer/WebServer0-000001.vmdk was not found
I looked back in the datastore and now in place of that file is a Webserver0-000001-delta.vmdk file instead.
I assume all these files are required for the VM to operate? Have I just royally screwed myself and all these people I've been hosting? Is there a way to recover or rebuild that file and restore this machine? I do not have a backup since Veeam said their products won't work with free ESXi. I also have no money and cannot afford to pay a support ticket or buy recovery software that cost $699 at minimum!
I need help desperately!
How much free disk space do you have on the other datastore (Datastore2)? What we need is ~200GB free disk space.
If you have that amount of free disk space on Datastore2, then you may follow the below steps to clone the Webserver VM to Datastore2.
You should now be able to power on the VM. Note that you may be asked whether you copied, or moved the VM at first power on. In this case select "I moved it".
Once everything works as expected you may procedd with deleting the "old" VM's folder on Datstore1.
Remember that each single snapshot can grow up to the virtual disk's provisioned size (200GB in this case). Best practice is to avoid snapshots on production VMs, i.e. only use them if needed, e.g. before making critical updates to have a way back in case it's necessary.
André
PS: I assume you've already fixed the missing .vmdk descriptor file issue.
Hi Troy and welcome to the community!
From what you have described it appears that you might have been fortunate and have removed the VMDK descriptor file rather than the actual data component. Now while I could have a go at rebuilding, your best bet is this gentleman: continuum
If you check his signature, you can contact him on Skype: sanbarrow
In the meantime could you run the following from the console in the directory containing the VM:
ls -lsh
Post the output here please. Also attach the vmx file if that is okay to do.
Also whatever you do, do not be tempted to attach the base disk directly to another VM or remove references to the snapshots. If the parent disk changes then you are possibly a much worse position.
Kind regards.
In the meantime could you run the following from the console in the directory containing the VM:
I'm sorry, but I am very new to this. Where do I go to do this?
I'm signed into the server remotely through the web client, but don't see where I can enter a console from here? Or, do I have to plug a monitor, mouse, and keyboard into the server and directly interface with it that way?
I have attached the vmx file as requested. Thank you so much for your quick response and assistance! I'm still freaking out about this.
1. Launch ssh session(using putty) to the ESXi host
2. type the following
cd /vmfs/volumes/<datastore name>/<vm directory>
ls -l | grep -i vmdk
please post the output, if ssh service is stopped , you may not be able to launch putty session. To start the ssh service , launch ESXi host client from browser
https://<ESXi ip>/ui
Then go to configure tab --> security profile --> Services --> there you can start the ssh service
Also, please do not attempt to delete any other files, as per the error message it looks like you have deleted only the descriptor file i.e 0006.vmdk , we can recreate this file using the available delta file.
Basically , each virtual disk will have two files
1. Descriptor file
2. Data file
for a disk without snapshot , you will have the file in this naming convention (taken your server name as example)
Webserver0.vmdk -- is the descriptor file of base disk
Webserver0-flat.vmdk -- actual data file ( if we delete this then we cannot recover data if we do not have any proper back up or using proper recovery tool)
In case if you have snapshot , you will find many disks in the following naming convention depends on the number of snapshots you have . in your case if appears that you have 6 snapshots
Webserver0-000001.vmdk -- descriptor file, can be recovered using the delta disk.
Webserver0-000001-delta.vmdk (data disk)
In your case, fortunately you deleted only the descriptor file which can be recovered using the steps given in the following video and kb article
How to recreate a missing Virtual Machine Disk Descriptor File (VMDK) - YouTube
Waiting for the output from ESXi ssh session for further steps
Please do not recommend that video again:
How to recreate a missing Virtual Machine Disk Descriptor File (VMDK) - YouTube
Following that bad advice can make matters worse - I have seen that way too often.
@ Troy
please connect to your ESXi via WinSCP and download all descriptor-vmdks of that VM.
That are the small vmdks with -flat, -ctk, -delta or -sesparse on the name.
Infortunately the Datastorebrowser will not display them correctly so you need to use WinSCP.
Zip them together and attach them to your next reply.
I will create the missing file for you.
Ulli
There a re couple of things that need to be clarified to recommend the proper steps to resolve the issue.
Some things have already been said, but let me put everything together.
How much free disk space do you currently have on the datastore?
André
Hi Troy,
No problems and sorry I should have provided the steps to get this file listing via a PuTTY session. Thanks SureshKumarMuthukrishnan for doing this.
I see continuum has responded so I will revert to just watching this thread and let him work with you. You are in good hands with his advice and I recommend just following what he says. a.p. has replied as well so definitely in good company there.
NOTE: not saying others cannot or will not offer good advice, just that it’s probably better to follow only one when dealing with potential data loss. Don’t want potential cross contamination of tasks
Kind regards.
I do not have a backup since Veeam said their products won't work with free ESXi.
Try Nakivo. It works with the free version of ESXi and the free version of Nakivo will backup up to two virtual machines.
-rw------- 1 root root 5788553216 Feb 6 2017 WebServer0-000001-delta.vmdk
-rw------- 1 root root 25032019968 Feb 22 2017 WebServer0-000002-delta.vmdk
-rw------- 1 root root 331 Feb 6 2017 WebServer0-000002.vmdk
-rw------- 1 root root 80111620096 Jul 14 2017 WebServer0-000003-delta.vmdk
-rw------- 1 root root 358 Jul 14 2017 WebServer0-000003.vmdk
-rw------- 1 root root 105864646656 Oct 22 2017 WebServer0-000004-delta.vmdk
-rw------- 1 root root 331 Jul 14 2017 WebServer0-000004.vmdk
-rw------- 1 root root 109454970880 Jul 5 2018 WebServer0-000005-delta.vmdk
-rw------- 1 root root 331 Jun 18 2018 WebServer0-000005.vmdk
-rw------- 1 root root 121953996800 Feb 25 23:58 WebServer0-000006-delta.vmdk
-rw------- 1 root root 358 Feb 25 23:26 WebServer0-000006.vmdk
-rw------- 1 root root 214748364800 Jan 30 2017 WebServer0-flat.vmdk
-rw------- 1 root root 527 Jan 30 2017 WebServer0.vmdk
It looks like WinSCP is just an FTP program? I don't have Windows. The only other computer I have available is Linux Mint. I will attempt to use Filezilla or another FTP program to download those and will post as soon as I can. Thanks!!
There's a physical 500GB hard drive. The virtual machine is set for 200GB. However, there is only 11MB free in the datastore.
I've attached a recreated descriptor file.
However, this won't help much due to the low disk space.
According to the descriptor files, the VM's disk has been thin provisioned, i.e. will most likely grow when you try to delete snapshots, so don't try this at the moment.
Please post the output for ls -elisa, as this will show not only the files' provisioned disk space, but also the used space. Base on this we need to find out how to resolve this issue.
How much free disk space do you have on the other datastore? I'm asking to find out whether it is an option to clone the virtual disk from the command line.
André
@ Glen
I received a PM from you with the following question:
Trust you don't mind but I've been playing along at home and this stuff interests me
Would the descriptor file you are going to create look like this (named "WebServer0-000001.vmdk":
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=e056a0a1
parentCID=1065ec08
isNativeSnapshot="no"
createType="vmfsSparse"
parentFileNameHint="WebServer0.vmdk"
# Extent description
RW 419430400 VMFSSPARSE "WebServer0-000001-delta.vmdk"
# The Disk Data Base #DDB
ddb.longContentID = "914a7521b682d3f525c773e16c3a7f6b"
Thanks in advance!
This is no secret knowhow - and the more folks can handle issues like this correctly the better it is for the community.
So your question is welcome.
Compare your suggestion with the one that Andre uploaded.
It has the exact same values - so your suggestion was spot on.
You may notice that Andre did not include the parameter:
ddb.longContentID = "914a7521b682d3f525c773e16c3a7f6b"
which you suggested to include.
I would have skipped this line as well.
It is optional and skipping it will do no harm so we both simply skip it.
Basically you have 5 parameters in a snapshot-descriptorfile that you have to check:
CID=e056a0a1
parentCID=1065ec08
parentFileNameHint="WebServer0.vmdk"
RW 419430400 VMFSSPARSE "WebServer0-000001-delta.vmdk"
Lets see how to get the values for each of them:
CID=e056a0a1
This value can be looked up in the descriptor of the next child vmdk. It must match the parentCID value of the next child vmdk.
parentCID=1065ec08
This value can be looked up in the descriptor of the parent vmdk. It must match the CID value of the parent vmdk.
parentFileNameHint="WebServer0.vmdk"
This value references the path to the parent vmdk.. If both vmdks reside in the same directory the filename is enough. If the parent-vmdk resides in a different directory you must use the full path.
RW 419430400
This references the "nominal" size of the VMDK. It gives the size in sectors.This value should be the same for EVERY vmdk in the snapshot-chain.
If not you will receive an error message: SIZE does not match. Parent has been expanded.
VMFSSPARSE "WebServer0-000001-delta.vmdk"
This parameter gives the path to the associated binary file that has the actual data. In 99.99 % of the cases both the descriptor vmdk as well as the binary part reside in the same directory so using the filename will work in almost all cases.
@ Glen
Feel free to contact me via skype if you are interested in this topic.
Ulli
continuum - you are too kind and thanks for the explanation. Very clear and very helpful.
I will take you up on the offer at some stage because this is something that definitely interests me. Often try to follow the bread crumbs as you guys do the work
This is one thing I love about the VMware community - the free sharing of knowledge. It’s great to be able to try and give back in some small way especially as you learn along the way!
Thanks again and normal service resumes
Here's the output you were looking for.
total 791560
4 1024 drwxr-xr-t 1 root root 1400 Tue Jan 24 21:50:00 2017 .
73675999881280 0 drwxr-xr-x 1 root root 512 Thu Feb 28 13:18:17 2019 ..
4194308 3072 -r-------- 1 root root 2490368 Mon Jan 23 20:43:59 2017 .fbb.sf
8388612 261120 -r-------- 1 root root 267026432 Mon Jan 23 20:43:59 2017 .fdc.sf
25165828 2048 -r-------- 1 root root 1179648 Mon Jan 23 20:43:59 2017 .pb2.sf
12582916 262144 -r-------- 1 root root 268435456 Mon Jan 23 20:43:59 2017 .pbc.sf
16777220 257024 -r-------- 1 root root 262733824 Mon Jan 23 20:43:59 2017 .sbc.sf
29360132 1024 drwx------ 1 root root 280 Mon Jan 23 20:44:00 2017 .sdd.sf
20971524 4096 -r-------- 1 root root 4194304 Mon Jan 23 20:43:59 2017 .vh.sf
83890564 8 drwxr-xr-x 1 root root 4060 Tue Feb 26 22:39:17 2019 WebServer
I have another 500GB drive in this server which is Datastore2. I had copied a bunch of the files from Datastore1 to it, but that's it. I can delete the contents of that datastore so we can use all the space if necessary.
Here's the output you were looking for.
Not exactly. Sorry if I wasn't clear enough. What I need is this output for the files in the VM's folder.
André
I'm sorry. I should have gone one more folder deeper. Try this.
total 462701592
83890564 8 drwxr-xr-x 1 root root 4060 Tue Feb 26 22:39:17 2019 .
4 1024 drwxr-xr-t 1 root root 1400 Tue Jan 24 21:50:00 2017 ..
465572228 8388608 -rw------- 1 root root 8589934592 Thu Jul 5 01:40:24 2018 WebServer-Snapshot8.vmem
461377924 7168 -rw------- 1 root root 6386305 Thu Jul 5 01:40:24 2018 WebServer-Snapshot8.vmsn
301994372 0 -rw-r--r-- 1 root root 13 Mon Jan 30 22:44:34 2017 WebServer-aux.xml
125833604 1024 -rw------- 1 root root 8684 Mon Feb 25 23:58:39 2019 WebServer.nvram
100667780 0 -rw-r--r-- 1 root root 438 Mon Feb 25 22:57:43 2019 WebServer.vmsd
88084868 8 -rwxr-xr-x 1 root root 3256 Mon Feb 25 23:58:39 2019 WebServer.vmx
268439940 8 -rw------- 1 root root 3283 Mon Feb 25 23:26:01 2019 WebServer.vmxf
314577284 5653504 -rw------- 1 root root 5788553216 Mon Feb 6 03:25:30 2017 WebServer0-000001-delta.vmdk
331354500 24445952 -rw------- 1 root root 25032019968 Wed Feb 22 22:47:34 2017 WebServer0-000002-delta.vmdk
335548804 0 -rw------- 1 root root 331 Mon Feb 6 03:25:35 2017 WebServer0-000002.vmdk
348131716 78234624 -rw------- 1 root root 80111620096 Fri Jul 14 02:47:18 2017 WebServer0-000003-delta.vmdk
352326020 0 -rw------- 1 root root 358 Fri Jul 14 02:43:52 2017 WebServer0-000003.vmdk
390074756 103384064 -rw------- 1 root root 105864646656 Sun Oct 22 22:43:28 2017 WebServer0-000004-delta.vmdk
394269060 0 -rw------- 1 root root 331 Fri Jul 14 02:47:22 2017 WebServer0-000004.vmdk
406851972 106890240 -rw------- 1 root root 109454970880 Thu Jul 5 01:38:30 2018 WebServer0-000005-delta.vmdk
411046276 0 -rw------- 1 root root 331 Mon Jun 18 09:01:35 2018 WebServer0-000005.vmdk
469766532 119096320 -rw------- 1 root root 121953996800 Mon Feb 25 23:58:39 2019 WebServer0-000006-delta.vmdk
473960836 0 -rw------- 1 root root 358 Mon Feb 25 23:26:00 2019 WebServer0-000006.vmdk
92279172 16591872 -rw------- 1 root root 214748364800 Mon Jan 30 22:45:05 2017 WebServer0-flat.vmdk
96473476 0 -rw------- 1 root root 527 Mon Jan 30 22:44:36 2017 WebServer0.vmdk
578818436 1024 -rw-r--r-- 1 root root 186726 Mon Feb 25 22:55:54 2019 vmware-24.log
595595652 1024 -rw-r--r-- 1 root root 219630 Mon Feb 25 23:58:39 2019 vmware-25.log
612372868 1024 -rw-r--r-- 1 root root 84517 Tue Feb 26 02:54:15 2019 vmware-26.log
629150084 1024 -rw-r--r-- 1 root root 85491 Tue Feb 26 03:07:33 2019 vmware-27.log
645927300 1024 -rw-r--r-- 1 root root 85494 Tue Feb 26 03:07:39 2019 vmware-28.log
662704516 1024 -rw-r--r-- 1 root root 85623 Tue Feb 26 21:17:06 2019 vmware-29.log
679481732 1024 -rw-r--r-- 1 root root 84319 Tue Feb 26 22:39:17 2019 vmware.log
How much free disk space do you have on the other datastore (Datastore2)? What we need is ~200GB free disk space.
If you have that amount of free disk space on Datastore2, then you may follow the below steps to clone the Webserver VM to Datastore2.
You should now be able to power on the VM. Note that you may be asked whether you copied, or moved the VM at first power on. In this case select "I moved it".
Once everything works as expected you may procedd with deleting the "old" VM's folder on Datstore1.
Remember that each single snapshot can grow up to the virtual disk's provisioned size (200GB in this case). Best practice is to avoid snapshots on production VMs, i.e. only use them if needed, e.g. before making critical updates to have a way back in case it's necessary.
André
PS: I assume you've already fixed the missing .vmdk descriptor file issue.