For a couple of years we used bareos with vmware plugin as our backup system. A few days ago due to an accident we lost our office DC completely. For this case the plugin is capable of restoring virtual machines into local vmdk files. The problem as it seems, it restores them in a kind of monolithic format which neither application nor utility recognizes as a valid VMDK, even though the plugin is using VDDK to create the images. My guess about the monolithic nature of the format is based on HEX dump of the file's header which looks like this:
I tried several methods of fixing the file including vmkfstools cloning, recreating the descriptor file, etc... None worked. Most of the time all tools and applications simply doesn't recognize the file as a virtual disk. For example, vmkfstools reports it as:
The file specified is not a virtual disk (15).
I wonder if anybody could point me at any means of recovering the file.
A funny part of it all is that 7zip did manage to find gpart partitions in the file and currently extracts zfs0.img for me. But I'd prefer to be able to boot normally from the disk.
Can you show how it looks with hexdump -C useing the esxi native command ?
> I tried several methods of fixing the file including vmkfstools cloning, recreating the descriptor file, etc..
Hmmm - that sounds scary - you know that this vmdk has an embedded descriptor that looks like it was either modified by someone who does not know enough about this ?
It will help if you show the hexdump with standard tools.
This really looks like someone tried several times to inject an embedded descriptor into a vmdk and always got the offset wrong - no offense intended - just telling how it looks like.
Dont use the hexeditor tool you have for changing embedded descriptors !!!
Use dd instead.
Here is the dump of the header follows. There were no manual manipulations to the file itself, it's exactly in the form it was restored. I'm too reluctant to any modification because 200Gb isn't the most lightweight size to be quickly restored or transferred.
What is confusing me in the descriptor part of the header is that it is using 0x00 as separator. Whatever I saw online says that the descriptor is embedded as-is, as a plain text.
00000000 12 20 12 12 01 00 00 00 00 00 00 00 32 00 00 00 |. ..........2...|
00000010 00 00 00 00 00 00 00 00 fc 65 00 00 ff 00 00 00 |.........e......|
00000020 3f 00 00 00 fc 65 00 00 ff 00 00 00 3f 00 00 00 |?....e......?...|
00000030 00 00 00 19 00 00 00 00 03 00 00 00 00 00 00 00 |................|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000070 00 00 00 00 00 00 00 00 00 00 00 00 12 20 12 12 |............. ..|
00000080 12 20 12 12 0a 00 00 00 06 00 00 00 12 20 12 12 |. ........... ..|
00000090 64 65 6c 65 74 61 62 6c 65 00 74 72 75 65 00 00 |deletable.true..|
000000a0 12 20 12 12 0e 00 00 00 22 00 00 00 12 20 12 12 |. ......".... ..|
000000b0 6c 6f 6e 67 43 6f 6e 74 65 6e 74 49 44 00 30 38 |longContentID.08|
000000c0 32 38 62 63 38 64 34 66 64 39 30 39 30 61 31 61 |28bc8d4fd9090a1a|
000000d0 33 61 31 34 34 38 36 63 61 33 39 36 33 63 00 00 |3a14486ca3963c..|
000000e0 12 20 12 12 0d 00 00 00 0c 00 00 00 12 20 12 12 |. ........... ..|
000000f0 74 6f 6f 6c 73 56 65 72 73 69 6f 6e 00 32 31 34 |toolsVersion.214|
00000100 37 34 38 33 36 34 37 00 6e 12 20 12 12 0a 00 00 |7483647.n. .....|
00000110 00 06 00 00 00 12 20 12 12 64 65 6c 65 74 61 62 |...... ..deletab|
00000120 6c 65 00 74 72 75 65 00 00 12 20 12 12 0e 00 00 |le.true... .....|
00000130 00 22 00 00 00 12 20 12 12 6c 6f 6e 67 43 6f 6e |.".... ..longCon|
00000140 74 65 6e 74 49 44 00 30 38 32 38 62 63 38 64 34 |tentID.0828bc8d4|
00000150 66 64 39 30 39 30 61 31 61 33 61 31 34 34 38 36 |fd9090a1a3a14486|
00000160 63 61 33 39 36 33 63 00 00 12 20 12 12 0d 00 00 |ca3963c... .....|
00000170 00 0c 00 00 00 12 20 12 12 74 6f 6f 6c 73 56 65 |...... ..toolsVe|
00000180 72 73 69 6f 6e 00 32 31 34 37 34 38 33 36 34 37 |rsion.2147483647|
00000190 00 6e 12 20 12 12 0c 00 00 00 0a 00 00 00 12 20 |.n. ........... |
000001a0 12 12 61 64 61 70 74 65 72 54 79 70 65 00 6c 73 |..adapterType.ls|
000001b0 69 6c 6f 67 69 63 00 70 12 20 12 12 0a 00 00 00 |ilogic.p. ......|
000001c0 06 00 00 00 12 20 12 12 64 65 6c 65 74 61 62 6c |..... ..deletabl|
000001d0 65 00 74 72 75 65 00 00 12 20 12 12 13 00 00 00 |e.true... ......|
000001e0 07 00 00 00 12 20 12 12 67 65 6f 6d 65 74 72 79 |..... ..geometry|
000001f0 2e 63 79 6c 69 6e 64 65 72 73 00 32 36 31 30 38 |.cylinders.26108|
00000200 00 00 12 20 12 12 0f 00 00 00 05 00 00 00 12 20 |... ........... |
00000210 12 12 67 65 6f 6d 65 74 72 79 2e 68 65 61 64 73 |..geometry.heads|
00000220 00 32 35 35 00 00 12 20 12 12 11 00 00 00 04 00 |.255... ........|
00000230 00 00 12 20 12 12 67 65 6f 6d 65 74 72 79 2e 73 |... ..geometry.s|
00000240 65 63 74 6f 72 73 00 36 33 00 01 12 20 12 12 0e |ectors.63... ...|
00000250 00 00 00 22 00 00 00 12 20 12 12 6c 6f 6e 67 43 |...".... ..longC|
00000260 6f 6e 74 65 6e 74 49 44 00 30 38 32 38 62 63 38 |ontentID.0828bc8|
00000270 64 34 66 64 39 30 39 30 61 31 61 33 61 31 34 34 |d4fd9090a1a3a144|
00000280 38 36 63 61 33 39 36 33 63 00 00 12 20 12 12 10 |86ca3963c... ...|
00000290 00 00 00 03 00 00 00 12 20 12 12 74 68 69 6e 50 |........ ..thinP|
000002a0 72 6f 76 69 73 69 6f 6e 65 64 00 31 00 40 12 20 |rovisioned.1.@. |
000002b0 12 12 11 00 00 00 03 00 00 00 12 20 12 12 74 6f |........... ..to|
000002c0 6f 6c 73 49 6e 73 74 61 6c 6c 54 79 70 65 00 30 |olsInstallType.0|
000002d0 00 40 12 20 12 12 0d 00 00 00 0c 00 00 00 12 20 |.@. ........... |
000002e0 12 12 74 6f 6f 6c 73 56 65 72 73 69 6f 6e 00 32 |..toolsVersion.2|
000002f0 31 34 37 34 38 33 36 34 37 00 6e 12 20 12 12 05 |147483647.n. ...|
00000300 00 00 00 31 00 00 00 12 20 12 12 75 75 69 64 00 |...1.... ..uuid.|
00000310 36 30 20 30 30 20 43 32 20 39 32 20 35 37 20 34 |60 00 C2 92 57 4|
00000320 65 20 63 39 20 66 30 2d 39 33 20 37 35 20 34 39 |e c9 f0-93 75 49|
00000330 20 38 36 20 63 30 20 38 64 20 31 34 20 31 61 00 | 86 c0 8d 14 1a.|
00000340 6f 12 20 12 12 11 00 00 00 04 00 00 00 12 20 12 |o. ........... .|
00000350 12 76 69 72 74 75 61 6c 48 57 56 65 72 73 69 6f |.virtualHWVersio|
00000360 6e 00 31 31 00 01 12 20 12 12 00 00 00 00 00 00 |n.11... ........|
00000370 00 00 12 20 12 12 12 20 12 12 00 00 00 00 00 00 |... ... ........|
00000380 00 00 00 00 00 00 00 00 20 00 03 00 00 00 12 20 |........ ...... |
00000390 12 12 00 00 00 00 fc 31 c0 8e c0 8e d8 8e d0 bc |.......1........|
000003a0 00 0e be 1a 7c bf 1a 06 b9 e6 01 f3 a4 e9 00 8a |....|...........|
000003b0 80 fa 80 72 0b 8a 36 75 04 80 c6 80 38 f2 72 02 |...r..6u....8.r.|
000003c0 b2 80 e8 c8 00 b6 01 bb 00 0e be 88 07 e8 a3 00 |................|
000003d0 66 81 3e 00 0e 45 46 49 20 75 0d 66 81 3e 04 0e |f.>..EFI u.f.>..|
000003e0 50 41 52 54 75 02 eb 19 80 fe 01 0f 85 b2 00 b6 |PARTu...........|
An embedded descriptor file inside a vmdk should look like
This is where you normally find the string EFI PART in a GPT diskimage or in a flat.vmdk
00000200 45 46 49 20 50 41 52 54 00 00 01 00 5c 00 00 00 |EFI PART....\...|
00000210 5c 1e 1d d6 00 00 00 00 01 00 00 00 00 00 00 00 |\...............|
00000220 af 6d 70 74 00 00 00 00 22 00 00 00 00 00 00 00 |.mpt....".......|
Your file once had a GPT partition table - see the EFI PART that you have at
000003d0 66 81 3e 00 0e 45 46 49 20 75 0d 66 81 3e 04 0e |f.>..EFI u.f.>..|
000003e0 50 41 52 54 75 02 eb 19 80 fe 01 0f 85 b2 00 b6 |PARTu...........|
Honestly this looks to me like a hexedit gone wrong - like somebody used "inject" where he should have used "replace"
dump the first mb of your vmdk to a file , zip it and attach it here please.
dd if=your.vmdk bs=1M count=1 of=badheader.bin
Please send the first 2 MB
I guess this once was a regular flat.vmdk - I might be able to fix it.
But I cant promise that I find the time today or tomorrow
Do you know with which filesystem the first partition of that vmdk was formatted ?
It is gpart with FreeBSD swap and zfs partiions.
BTW, so far I have found the signature of the flat VMDK format and cut off all of the header preceding it. VMWare Fusion rejects the file. Waiting for ESXi to finish the same operation and going to try either vmkfstools descriptor recovery process or creating a dummy VM and replacing the name of flat VMDK in its descriptor.
With the extra header cut off I have managed to re-create the descriptor using vmkfstools and used that description to run the disk through vmware-vdiskmamanger on my MacBook. The good thing is that the fixed disk can be attached to a VirtualBox VM. The bad things is that it's geometry is totally wrong and GPT is broken. Though I tried -t 0 with vmware-vdiskmanager whereas it more likely to be 6 (thin, ESX 3.x and above).
You dont need to try anything obscure - if this
Number Start (sector) End (sector) Size Code Name
1 40 1063 512.0 KiB A501 gptboot0
2 2048 25167871 12.0 GiB A502 swap0
3 25167872 419428351 188.0 GiB A504 zfs0
Is the correct partitiontable - then this is the start of the flat.vmdk.
and this vmdk descriptor should work:
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=12345678
parentCID=12345678
isNativeSnapshot="no"
createType="vmfs"
# Extent description
RW 419430400 VMFS "rename-as-needed-flat.vmdk"
# The Disk Data Base
#DDB
ddb.adapterType = "lsilogic"
ddb.geometry.cylinders = "26108"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
Filesize of the flat.vmdk is supposed to be 419430400 sectors or 200gb.
I hope you cut at the same offset ???
The header you sent is very strange - if this was created by a program you should report a bug.
Ulli
Yes, my cut file starts with FC 31 C0 8E too:
00000000 fc 31 c0 8e c0 8e d8 8e d0 bc 00 0e be 1a 7c bf |.1............|.|
00000010 1a 06 b9 e6 01 f3 a4 e9 00 8a 80 fa 80 72 0b 8a |.............r..|
00000020 36 75 04 80 c6 80 38 f2 72 02 b2 80 e8 c8 00 b6 |6u....8.r.......|
00000030 01 bb 00 0e be 88 07 e8 a3 00 66 81 3e 00 0e 45 |..........f.>..E|
00000040 46 49 20 75 0d 66 81 3e 04 0e 50 41 52 54 75 02 |FI u.f.>..PARTu.|
I tried the same approach with the descriptor. My version was somewhat more verbose (as vmkfstools generated it), but otherwise similar. I.e. I tried both the default geometry suggested by vmkfstools and the values you use in your descriptor (in other words: those found in the restored file header). VMWare Fusion never complains and uses the disk. Direct boot is not possible and I decided to use a workaround by installing a fresh FreeBSD 12.1 and using the problematic image as a secondary drive. When the system complained about the disk geometry, I have adjusted the size after RW directive in the descriptor. It seemingly fixed the geometry problem but zfs pool remained faulty and unrecoverable. Changing of cyl/head/sectors in the descriptor didn't help.
Then I tried the same on a ESXi host where it also done cutting off the header by the time. Same problem with booting from the drive. But when I tried to change the size in RW to the expected value 419430400 (which I also got from gdisk utility, BTW) ESXi rejected the vmdk with the familiar message of the file not being a virtual disk.
In fact, the actual disk size is really smaller that required: 213678817344 bytes or 417341440.125 sectors. Considering that the original VMDK was thin-provisioned and corresponding configuration variable can be found in the restored file header, it shouldn't be a problem.
BTW, thanks a lot for your assistance! Even though I haven't succeed yet, but your information was certainly very useful.
Useful trick I learned from my recovery work ...
Often when I recover a flat.vmdk manually my restored flat.vmdk is slightly smaller than the size given in the descriptor.
ESXi then complains : file is not a virtual disk - and the message is appropriate.
Then the ugly workaround is to slightly reduce the size in the descriptor until it matches the flat.vmdk
The clean fix is to add blank space at the end of the file with dd.
By the way - nowadays the geometry values in the descriptor are no longer required.
You can skip them completely.
I use the standard values that ESXi would use too:
That means
This 2 parameters use fixed values - they are always 255 and 63
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
The value for
ddb.geometry.cylinders
can be calculated by
<extent-size in sectors> / 16065 rounded down = ddb.geometry.cylinders
In your case 419430400 / 16065 rouded down = 26108
I never came across a vmdk that complained about a bad geometry - but that is probably because most of the times I deal woth recovered Windows-VMs.
If I run into a geometry problem I would use a Linux-liveCD with Testdisk and let it fix the geometry.
But first of all I would make sure the complete flat.vmdk is not smaller then declared by the value for <size-in-sectors> given in the original descriptor.
... forgot to mentiom ...
> and GPT is broken.
That suggest that your flat.vmdk is indeed to short.
Add blank space at the end of the file - boot into Linux with gdisk, gparted or testdisk and let testdisk or gdisk
fix the GPT by copying the master copy of the GPT to the backup GPT-copy which should sit at the end of disk.
> In fact, the actual disk size is really smaller that required: 213678817344 bytes or 417341440.125 sectors.
> Considering that the original VMDK was thin-provisioned and corresponding configuration variable can be found in the restored file header, it shouldn't be a problem.
Wrong:
- the parameter for thin provisioning in the descriptor is just for your information.
In doubt or after a recovery - just remove it.
And keep in mind that during your initial creation of this VMDK some tool decided where to store the backup GPT.
This tools follow strict rules on where to store the backup GPT - and when you allowed that tool to use the full disk it will make the partitions as large as allowed.
So do NOT assume that thin provisioned vmdks dont care if they are too small.
The guestOS does not know if it is using a thin provisioned vmdk !!!
The more I fail the more get convinced that the problem is a bug in bareos which resulted in backing up only the delta of a previously taken snapshot. After another round of experimenting with the descriptor I had to re-cut off the headers and tried the following descriptor with the -flat file after padding it with zeroes:
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=12345678
parentCID=12345678
isNativeSnapshot="no"
createType="vmfs"
# Extent description
RW 419430400 VMFS "real-flat.vmdk"
# The Disk Data Base
#DDB
ddb.adapterType = "lsilogic"
ddb.geometry.cylinders = "26108"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.longContentID = "0828bc8d4fd9090a1a3a14486ca3963c"
#ddb.thinProvisioned = "1"
ddb.virtualHWVersion = "11"
Now Fusion rejects the disk with the following message: "The called function cannot be performed on partial chains." Not sure if this is something in the descriptor or in the flat file itself which resulted in the message. But if it's something in the flat file then I'm unlikely to get the problem fixed in any "legitimate" way.
I love interesting problems - if it does not go away I would like to visit you and have a look myself - but not tonight ...
Shall I contact you by skype tomorrow?
Further updater on the situation.
The following descriptor works as expected:
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=fffffffe
parentCID=ffffffff
isNativeSnapshot="no"
createType="vmfs"
# Extent description
RW 419430400 VMFS "LFLat-real-flat.vmdk"
# The Disk Data Base
#DDB
ddb.adapterType = "lsilogic"
ddb.geometry.cylinders = "26108"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.longContentID = "d3ccce696e2f3ce86fb9bb70fffffffe"
ddb.uuid = "60 00 C2 97 5e fd 86 15-52 80 f7 bf 05 00 ee 12"
ddb.virtualHWVersion = "11"
ddb.thinProvisioned = "1"
ddb.uuid.image="042d4405-aa6f-49e0-bbb7-5b6298cb30c6"
ddb.uuid.modification="f9cb167b-4715-45da-a1ec-3df7e7f695a5"
ddb.uuid.parent="00000000-0000-0000-0000-000000000000"
ddb.uuid.parentmodification="00000000-0000-0000-0000-000000000000"
A temporary installation of FreeBSD see the disk as a secondary SCSI, partitions are properly recognized. zdb tool can now see the first 2 of 4 zfs labels. Inside the flat file I managed to find the other two labels near the end of the disk.
Another thing I realized is that after all it's not a difference image because I found files inside of it which were not touched for years. Perhaps they don't even exists any more and what I saw was leftovers of ZFS file writing strategy. In either case, those blocks are most certainly were not updated for long enough not be included into a difference file.
What it currently looks like to me is that a likely bug in bareos vmware plugin caused it to skip some number of blocks while restoring the disk. This would explain why it was shorter than expected. But it also means that when I padded it with zeroes at the end it didn't help placing the other two zfs labels into the correct locations. At this point I'm currently stuck with no further ideas as to where to go.