How to fix a seeming monolithic vmdk?

[vrurg]

For a couple of years we used bareos with vmware plugin as our backup system. A few days ago due to an accident we lost our office DC completely. For this case the plugin is capable of restoring virtual machines into local vmdk files. The problem as it seems, it restores them in a kind of monolithic format which neither application nor utility recognizes as a valid VMDK, even though the plugin is using VDDK to create the images. My guess about the monolithic nature of the format is based on HEX dump of the file's header which looks like this:

I tried several methods of fixing the file including vmkfstools cloning, recreating the descriptor file, etc... None worked. Most of the time all tools and applications simply doesn't recognize the file as a virtual disk. For example, vmkfstools reports it as:

The file specified is not a virtual disk (15).

I wonder if anybody could point me at any means of recovering the file.

A funny part of it all is that 7zip did manage to find gpart partitions in the file and currently extracts zfs0.img for me. But I'd prefer to be able to boot normally from the disk.

[continuum]

Can you show how it looks with hexdump -C useing the esxi native command ?

> I tried several methods of fixing the file including vmkfstools cloning, recreating the descriptor file, etc..

Hmmm - that sounds scary - you know that this vmdk has an embedded descriptor that looks like it was either modified by someone who does not know enough about this ?

It will help if you show the hexdump with standard tools.

This really looks like someone tried several times to inject an embedded descriptor into a vmdk and always got the offset wrong - no offense intended - just telling how it looks like.

Dont use the hexeditor tool you have for changing embedded descriptors !!!

Use dd instead.

[vrurg]

Here is the dump of the header follows. There were no manual manipulations to the file itself, it's exactly in the form it was restored. I'm too reluctant to any modification because 200Gb isn't the most lightweight size to be quickly restored or transferred.

What is confusing me in the descriptor part of the header is that it is using 0x00 as separator. Whatever I saw online says that the descriptor is embedded as-is, as a plain text.

00000000  12 20 12 12 01 00 00 00  00 00 00 00 32 00 00 00  |. ..........2...|

00000010  00 00 00 00 00 00 00 00  fc 65 00 00 ff 00 00 00  |.........e......|

00000020  3f 00 00 00 fc 65 00 00  ff 00 00 00 3f 00 00 00  |?....e......?...|

00000030  00 00 00 19 00 00 00 00  03 00 00 00 00 00 00 00  |................|

00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

*

00000070  00 00 00 00 00 00 00 00  00 00 00 00 12 20 12 12  |............. ..|

00000080  12 20 12 12 0a 00 00 00  06 00 00 00 12 20 12 12  |. ........... ..|

00000090  64 65 6c 65 74 61 62 6c  65 00 74 72 75 65 00 00  |deletable.true..|

000000a0  12 20 12 12 0e 00 00 00  22 00 00 00 12 20 12 12  |. ......".... ..|

000000b0  6c 6f 6e 67 43 6f 6e 74  65 6e 74 49 44 00 30 38  |longContentID.08|

000000c0  32 38 62 63 38 64 34 66  64 39 30 39 30 61 31 61  |28bc8d4fd9090a1a|

000000d0  33 61 31 34 34 38 36 63  61 33 39 36 33 63 00 00  |3a14486ca3963c..|

000000e0  12 20 12 12 0d 00 00 00  0c 00 00 00 12 20 12 12  |. ........... ..|

000000f0  74 6f 6f 6c 73 56 65 72  73 69 6f 6e 00 32 31 34  |toolsVersion.214|

00000100  37 34 38 33 36 34 37 00  6e 12 20 12 12 0a 00 00  |7483647.n. .....|

00000110  00 06 00 00 00 12 20 12  12 64 65 6c 65 74 61 62  |...... ..deletab|

00000120  6c 65 00 74 72 75 65 00  00 12 20 12 12 0e 00 00  |le.true... .....|

00000130  00 22 00 00 00 12 20 12  12 6c 6f 6e 67 43 6f 6e  |.".... ..longCon|

00000140  74 65 6e 74 49 44 00 30  38 32 38 62 63 38 64 34  |tentID.0828bc8d4|

00000150  66 64 39 30 39 30 61 31  61 33 61 31 34 34 38 36  |fd9090a1a3a14486|

00000160  63 61 33 39 36 33 63 00  00 12 20 12 12 0d 00 00  |ca3963c... .....|

00000170  00 0c 00 00 00 12 20 12  12 74 6f 6f 6c 73 56 65  |...... ..toolsVe|

00000180  72 73 69 6f 6e 00 32 31  34 37 34 38 33 36 34 37  |rsion.2147483647|

00000190  00 6e 12 20 12 12 0c 00  00 00 0a 00 00 00 12 20  |.n. ........... |

000001a0  12 12 61 64 61 70 74 65  72 54 79 70 65 00 6c 73  |..adapterType.ls|

000001b0  69 6c 6f 67 69 63 00 70  12 20 12 12 0a 00 00 00  |ilogic.p. ......|

000001c0  06 00 00 00 12 20 12 12  64 65 6c 65 74 61 62 6c  |..... ..deletabl|

000001d0  65 00 74 72 75 65 00 00  12 20 12 12 13 00 00 00  |e.true... ......|

000001e0  07 00 00 00 12 20 12 12  67 65 6f 6d 65 74 72 79  |..... ..geometry|

000001f0  2e 63 79 6c 69 6e 64 65  72 73 00 32 36 31 30 38  |.cylinders.26108|

00000200  00 00 12 20 12 12 0f 00  00 00 05 00 00 00 12 20  |... ........... |

00000210  12 12 67 65 6f 6d 65 74  72 79 2e 68 65 61 64 73  |..geometry.heads|

00000220  00 32 35 35 00 00 12 20  12 12 11 00 00 00 04 00  |.255... ........|

00000230  00 00 12 20 12 12 67 65  6f 6d 65 74 72 79 2e 73  |... ..geometry.s|

00000240  65 63 74 6f 72 73 00 36  33 00 01 12 20 12 12 0e  |ectors.63... ...|

00000250  00 00 00 22 00 00 00 12  20 12 12 6c 6f 6e 67 43  |...".... ..longC|

00000260  6f 6e 74 65 6e 74 49 44  00 30 38 32 38 62 63 38  |ontentID.0828bc8|

00000270  64 34 66 64 39 30 39 30  61 31 61 33 61 31 34 34  |d4fd9090a1a3a144|

00000280  38 36 63 61 33 39 36 33  63 00 00 12 20 12 12 10  |86ca3963c... ...|

00000290  00 00 00 03 00 00 00 12  20 12 12 74 68 69 6e 50  |........ ..thinP|

000002a0  72 6f 76 69 73 69 6f 6e  65 64 00 31 00 40 12 20  |rovisioned.1.@. |

000002b0  12 12 11 00 00 00 03 00  00 00 12 20 12 12 74 6f  |........... ..to|

000002c0  6f 6c 73 49 6e 73 74 61  6c 6c 54 79 70 65 00 30  |olsInstallType.0|

000002d0  00 40 12 20 12 12 0d 00  00 00 0c 00 00 00 12 20  |.@. ........... |

000002e0  12 12 74 6f 6f 6c 73 56  65 72 73 69 6f 6e 00 32  |..toolsVersion.2|

000002f0  31 34 37 34 38 33 36 34  37 00 6e 12 20 12 12 05  |147483647.n. ...|

00000300  00 00 00 31 00 00 00 12  20 12 12 75 75 69 64 00  |...1.... ..uuid.|

00000310  36 30 20 30 30 20 43 32  20 39 32 20 35 37 20 34  |60 00 C2 92 57 4|

00000320  65 20 63 39 20 66 30 2d  39 33 20 37 35 20 34 39  |e c9 f0-93 75 49|

00000330  20 38 36 20 63 30 20 38  64 20 31 34 20 31 61 00  | 86 c0 8d 14 1a.|

00000340  6f 12 20 12 12 11 00 00  00 04 00 00 00 12 20 12  |o. ........... .|

00000350  12 76 69 72 74 75 61 6c  48 57 56 65 72 73 69 6f  |.virtualHWVersio|

00000360  6e 00 31 31 00 01 12 20  12 12 00 00 00 00 00 00  |n.11... ........|

00000370  00 00 12 20 12 12 12 20  12 12 00 00 00 00 00 00  |... ... ........|

00000380  00 00 00 00 00 00 00 00  20 00 03 00 00 00 12 20  |........ ...... |

00000390  12 12 00 00 00 00 fc 31  c0 8e c0 8e d8 8e d0 bc  |.......1........|

000003a0  00 0e be 1a 7c bf 1a 06  b9 e6 01 f3 a4 e9 00 8a  |....|...........|

000003b0  80 fa 80 72 0b 8a 36 75  04 80 c6 80 38 f2 72 02  |...r..6u....8.r.|

000003c0  b2 80 e8 c8 00 b6 01 bb  00 0e be 88 07 e8 a3 00  |................|

000003d0  66 81 3e 00 0e 45 46 49  20 75 0d 66 81 3e 04 0e  |f.>..EFI u.f.>..|

000003e0  50 41 52 54 75 02 eb 19  80 fe 01 0f 85 b2 00 b6  |PARTu...........|

[continuum]

An embedded descriptor file inside a vmdk should look like

This is where you normally find the string EFI PART in a GPT diskimage or in a flat.vmdk

00000200  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|

00000210  5c 1e 1d d6 00 00 00 00  01 00 00 00 00 00 00 00  |\...............|

00000220  af 6d 70 74 00 00 00 00  22 00 00 00 00 00 00 00  |.mpt....".......|

Your file once had a GPT partition table - see the EFI PART  that you have at

000003d0  66 81 3e 00 0e 45 46 49  20 75 0d 66 81 3e 04 0e  |f.>..EFI u.f.>..|

000003e0  50 41 52 54 75 02 eb 19  80 fe 01 0f 85 b2 00 b6  |PARTu...........|

Honestly this looks to me like a hexedit gone wrong - like somebody used  "inject" where he should have used "replace"

dump  the first mb of your vmdk to a file , zip it and attach it here please.

dd if=your.vmdk bs=1M count=1 of=badheader.bin

[vrurg]

Here is the zip attached.

Meanwhile I got a suspicion that the problem could have been caused by snapshots I took for that VM a while ago. Don't have any clear idea how it may affect the vmware plugin, but something like it has received a wrong file from the host comes to mind.

[continuum]

Please send the first 2 MB

I guess this once was a regular flat.vmdk - I might be able to fix it.

But  I cant promise that I find the time today or tomorrow

[vrurg]

Here is the first 2MB.

Thank you!

[continuum]

Do you know with which filesystem the first partition of that vmdk was formatted ?

[vrurg]

It is gpart with FreeBSD swap and zfs partiions.

BTW, so far I have found the signature of the flat VMDK format and cut off all of the header preceding it. VMWare Fusion rejects the file. Waiting for ESXi to finish the same operation and going to try either vmkfstools descriptor recovery process or creating a dummy VM and replacing the name of flat VMDK in its descriptor.

[vrurg]

With the extra header cut off I have managed to re-create the descriptor using vmkfstools and used that description to run the disk through vmware-vdiskmamanger on my MacBook. The good thing is that the fixed disk can be attached to a VirtualBox VM. The bad things is that it's geometry is totally wrong and GPT is broken. Though I tried -t 0 with vmware-vdiskmanager whereas it more likely to be 6 (thin, ESX 3.x and above).

[continuum]

You dont need to try anything obscure - if this

Number  Start (sector)    End (sector)  Size       Code  Name

   1              40            1063   512.0 KiB   A501  gptboot0

   2            2048        25167871   12.0 GiB    A502  swap0

   3        25167872       419428351   188.0 GiB   A504  zfs0

Is the correct partitiontable -  then this is the start of the flat.vmdk.

and this vmdk descriptor should work:

# Disk DescriptorFile

version=1

encoding="UTF-8"

CID=12345678

parentCID=12345678

isNativeSnapshot="no"

createType="vmfs"

# Extent description

RW 419430400 VMFS "rename-as-needed-flat.vmdk"

# The Disk Data Base

#DDB

ddb.adapterType = "lsilogic"

ddb.geometry.cylinders = "26108"

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

Filesize of the flat.vmdk is supposed to be 419430400 sectors or 200gb.

I hope you cut at the same offset ???

The header you sent is very strange - if this was created by a program you should report a bug.

Ulli

[vrurg]

Yes, my cut file starts with FC 31 C0 8E too:

00000000  fc 31 c0 8e c0 8e d8 8e  d0 bc 00 0e be 1a 7c bf  |.1............|.|

00000010  1a 06 b9 e6 01 f3 a4 e9  00 8a 80 fa 80 72 0b 8a  |.............r..|

00000020  36 75 04 80 c6 80 38 f2  72 02 b2 80 e8 c8 00 b6  |6u....8.r.......|

00000030  01 bb 00 0e be 88 07 e8  a3 00 66 81 3e 00 0e 45  |..........f.>..E|

00000040  46 49 20 75 0d 66 81 3e  04 0e 50 41 52 54 75 02  |FI u.f.>..PARTu.|

I tried the same approach with the descriptor. My version was somewhat more verbose (as vmkfstools generated it), but otherwise similar. I.e. I tried both the default geometry suggested by vmkfstools and the values you use in your descriptor (in other words: those found in the restored file header). VMWare Fusion never complains and uses the disk. Direct boot is not possible and I decided to use a workaround by installing a fresh FreeBSD 12.1 and  using the problematic image as a secondary drive. When the system complained about the disk geometry, I have adjusted the size after RW directive in the descriptor. It seemingly fixed the geometry problem but zfs pool remained faulty and unrecoverable. Changing of cyl/head/sectors in the descriptor didn't help.

Then I tried the same on a ESXi host where it also done cutting off the header by the time. Same problem with booting from the drive. But when I tried to change the size in RW to the expected value 419430400 (which I also got from gdisk utility, BTW) ESXi rejected the vmdk with the familiar message of the file not being a virtual disk.

In fact, the actual disk size is really smaller that required: 213678817344 bytes or 417341440.125 sectors. Considering that the original VMDK was thin-provisioned and corresponding configuration variable can be found in the restored file header, it shouldn't be a problem.

BTW, thanks a lot for your assistance! Even though I haven't succeed yet, but your information was certainly very useful.

[continuum]

Useful trick I learned from my recovery work ...

Often when I recover a flat.vmdk manually my restored flat.vmdk is slightly smaller than the size given in the descriptor.

ESXi then complains : file is not a virtual disk - and the message is appropriate.

Then the ugly workaround is to slightly reduce the size in the descriptor until it matches the flat.vmdk

The clean fix is to add blank space at the end of the file with dd.

By the way - nowadays the geometry values in the descriptor are no longer required.

You can skip them completely.

I use the standard values that ESXi would use too:

That means

This 2 parameters use fixed values - they are always 255 and 63

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

The value for

ddb.geometry.cylinders

can be calculated by

<extent-size in sectors> / 16065  rounded down = ddb.geometry.cylinders

In your case 419430400 / 16065 rouded down = 26108

I never came across a vmdk that complained about a bad geometry - but that is probably because most of the times I deal woth recovered Windows-VMs.

If I run into a geometry problem I would use a Linux-liveCD with Testdisk and let it fix the geometry.

But first of all I would make sure the complete flat.vmdk is not smaller then declared by the value for <size-in-sectors> given in the original descriptor.

[continuum]

... forgot to mentiom ...

> and GPT is broken.

That suggest that your flat.vmdk is indeed to short.

Add blank space at the end of the file - boot into Linux with gdisk, gparted or testdisk and let testdisk or gdisk

fix the GPT by copying the master copy of the GPT to the backup GPT-copy which should sit at the end of disk.

[continuum]

> In fact, the actual disk size is really smaller that required: 213678817344 bytes or 417341440.125 sectors.

> Considering that the original VMDK was thin-provisioned and corresponding configuration variable can be found in the restored file header, it shouldn't be a problem.

Wrong:

- the parameter for thin provisioning in the descriptor is just for your information.

In doubt or after a recovery - just remove it.

And keep in mind that during your initial creation of this VMDK some tool decided where to store the backup GPT.

This tools follow strict rules on where to store the backup GPT - and when you allowed that tool to use the full disk it will make the partitions as large as allowed.

So do NOT assume that thin provisioned vmdks dont care if they are too small.

The guestOS does not know if it is using a thin provisioned vmdk !!!

[vrurg]

The more I fail the more get convinced that the problem is a bug in bareos which resulted in backing up only the delta of a previously taken snapshot. After another round of experimenting with the descriptor I had to re-cut off the headers and tried the following descriptor with the -flat file after padding it with zeroes:

# Disk DescriptorFile

version=1

encoding="UTF-8"

CID=12345678

parentCID=12345678

isNativeSnapshot="no"

createType="vmfs"

# Extent description

RW 419430400 VMFS "real-flat.vmdk"

# The Disk Data Base

#DDB

ddb.adapterType = "lsilogic"

ddb.geometry.cylinders = "26108"

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

ddb.longContentID = "0828bc8d4fd9090a1a3a14486ca3963c"

#ddb.thinProvisioned = "1"

ddb.virtualHWVersion = "11"

Now Fusion rejects the disk with the following message: "The called function cannot be performed on partial chains." Not sure if this is something in the descriptor or in the flat file itself which resulted in the message. But if it's something in the flat file then I'm unlikely to get the problem fixed in any "legitimate" way.

[continuum]

I love interesting problems - if it does not go away I would like to visit you and have a look myself - but not tonight ...

[vrurg]

Shall I contact you by skype tomorrow?

[vrurg]

Further updater on the situation.

The following descriptor works as expected:

# Disk DescriptorFile

version=1

encoding="UTF-8"

CID=fffffffe

parentCID=ffffffff

isNativeSnapshot="no"

createType="vmfs"

# Extent description

RW 419430400 VMFS "LFLat-real-flat.vmdk"

# The Disk Data Base

#DDB

ddb.adapterType = "lsilogic"

ddb.geometry.cylinders = "26108"

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

ddb.longContentID = "d3ccce696e2f3ce86fb9bb70fffffffe"

ddb.uuid = "60 00 C2 97 5e fd 86 15-52 80 f7 bf 05 00 ee 12"

ddb.virtualHWVersion = "11"

ddb.thinProvisioned = "1"

ddb.uuid.image="042d4405-aa6f-49e0-bbb7-5b6298cb30c6"

ddb.uuid.modification="f9cb167b-4715-45da-a1ec-3df7e7f695a5"

ddb.uuid.parent="00000000-0000-0000-0000-000000000000"

ddb.uuid.parentmodification="00000000-0000-0000-0000-000000000000"

A temporary installation of FreeBSD see the disk as a secondary SCSI, partitions are properly recognized. zdb tool can now see the first 2 of 4 zfs labels. Inside the flat file I managed to find the other two labels near the end of the disk.

Another thing I realized is that after all it's not a difference image because I found files inside of it which were not touched for years. Perhaps they don't even exists any more and what I saw was leftovers of ZFS file writing strategy. In either case, those blocks are most certainly were not updated for long enough not be included into a difference file.

What it currently looks like to me is that a likely bug in bareos vmware plugin caused it to skip some number of blocks while restoring the disk. This would explain why it was shorter than expected. But it also means that when I padded it with zeroes at the end it didn't help placing the other two zfs labels into the correct locations. At this point I'm currently stuck with no further ideas as to where to go.