Snapshot Chain broken, VM can´t start, vmsd.file is corrupted

[haze_x]

Hey,

our ESX-Host crashed yesterday and since then one VM on it doesn´t start. The error messages vSphere gives are:

"Cannot power on VM. a general system error occurred:

Error message from ESX-Host: Cannot open the disk "xxxx-000002.vmdk" or one of the snapshot disks it depends on."

and

"Cannot power on VM. a general system error occurred:

Error message from ESX-Host: Cannot open the disk "xxxx_5-000004.vmdk" or one of the snapshot disks it depends on."

For each disk I have one flat.vmdk and multiple delta.vmdk files. The Snapshots don´t show up in the snapshot manager and I can´t consolidate them either (through vSphere or SSH).

I already searched Google and the forum here and found similar problems, but nothing helped in my case

Unfortunately the backup, that should have happened over the weekend  failed with a similar error.

The .vmsd file is empty, is there any way to recreate it?

Can I somehow merge the delta and flat files?

Can simply renaming the delta.vmdk files be helpful? I have a feeling that´s not the case

The .vmdk descriptor files are also missing.

I don´t have much experience with vmware disk management, so I would be thankful for every suggestion.

Thanks

[continuum]

Do NOT rename any files.

Provide a filelisting using ssh.

ls -lah > files.txt

[haze_x]

Okay, here is the filelist.

[continuum]

Ok - looks like you lost all vmdk-descriptorfiles.
Please attach KFIA SP OnBoarding_1.vmx to your next reply.
Is it correct to assume that the VM used 2 virtual disks ?
The vmdk-filenames look like there were more than 2 vmdks.
By the way - next time you run into a similar problem do NOT start the VM so many times if it runs into an error.
Instead start just once or twice and then create a backup of the old vmware.log files.
If you do not do that starting the VM more than 5 times destroys all useful evidence that was contained in the older log-files.

[haze_x]

Yes, the VM used 2 virtual disks.

The multiple vmdks are from snapshots i guess?

Yea starting the VM that many times was not good, a bit of miscommunication with my colleague there.

[a_p_]

It should be possible to recreate the descriptor files.

However, in order to see whether the virtual disks have been thin, or thick provisioned, please run ls -lisa > files.txt and attach the files.txt to a reply post.

Also let us know how much free disk space you currently have on that datastore, to determine if it is save to delete the snasphots.

André

[continuum]

Hi Andre
just curious ... in my experience it is not necessary to define
ddb.thinProvisioned = "1".
It should be fine to set the size to 100gb / 20gb.
If the vmfs-metadata for the flat and delta vmdks is healthy the system will set that parameter on first start appropiatly.
If the vmfs-metadata is damaged - a fixed descriptorfile will not make a difference :smileycry:

[a_p_]

If the vmfs-metadata for the flat and delta vmdks is healthy the system will set that parameter on first start appropiatly.

To be honest, I wasn't aware of this. Probably because I never tried it so far.

In this case go ahead, and create the descriptor files.

André

[continuum]

@ Andre
The order of snapshots for KFIA SP OnBoarding_1.vmdk is inconclusive judging from here:

-rw-------    1 root     root       84.7G Jan  9  2018 KFIA SP OnBoarding_1-000001-delta.vmdk

-rw-------    1 root     root        7.7G Jul 17  2014 KFIA SP OnBoarding_1-000002-delta.vmdk

-rw-------    1 root     root       48.2M Jan  9  2018 KFIA SP OnBoarding_1-000003-delta.vmdk

-rw-------    1 root     root       37.8G Aug 15 13:22 KFIA SP OnBoarding_1-000004-delta.vmdk

-rw-------    1 root     root      100.0G Jun 19  2014 KFIA SP OnBoarding_1-flat.vmdk
My tip would be to use either

KFIA SP OnBoarding_1-000004.vmdk

KFIA SP OnBoarding_1-000001.vmdk

KFIA SP OnBoarding_1-000003.vmdk

KFIA SP OnBoarding_1-000002.vmdk

KFIA SP OnBoarding_1.vmdk

or

KFIA SP OnBoarding_1-000004.vmdk

KFIA SP OnBoarding_1-000003.vmdk

KFIA SP OnBoarding_1-000001.vmdk

KFIA SP OnBoarding_1-000002.vmdk

KFIA SP OnBoarding_1.vmdk

Do you have any suggestion ?

[a_p_]

From the file sizes, I'd say 4, 3, 1, 2, flat.

To be sure we would need the exact time stamps, i.e. the output of ls -elisa.

Anyway, due to the small size of "3", there's not really a high risk if the order is incorrect. Just create another snapshot prior to powering on the VM, so that you can revert to the current state in case it's necessary.

André

PS: I just tested with setting ddb.thinProvisioned to an incorrect value. Powering on didn't modify the setting, but the vmware.log file shows that the disks are recognized with their correct provisioning type anyway.

Note: I tested this on ESXi 5.0, because the VM's configuration file shows HW version 8.

[continuum]

If I have to create a descriptor from scratch I typically do not set
ddb.thinProvisioned at all.
That seems to work well - never had any problems.

@ Julian
please provide the output of
ls -elisa

[haze_x]

Hi guys, here is the output of ls -elisa, hope it helps.

[a_p_ · Accepted solution]

Please extract re-created descriptor files from the attached archive, and upload them to the VM's folder on the datastore.

Prior to powering on the VM, take another snapshot.

Since I assume that you don't want/need to revert to an older snapshot, I'd suggest you run "Delete All" from the Snapshot Manager after confirming that the VM works as expected.

Please note that deleting the snapshots may require up to about 90GB of temporary disk space, so please ensure that you have sufficient free disk space.

André

[haze_x]

Thank you both for your help!

The VM is working flawless now.

Big thanks for your help guys, the server in question is very important for our developers.

[continuum]

@ Andre

well done :smileylaugh:

Ulli

[a_p_]

@ Ulli

Let's call it "Teamwork".:smileycheck:

André