Veeam created 256 snapshots VM unresponsive, needs consolidation

[LarsDalgaard]

Hi

We are running an old esxi 6.0 and unfortuantely something has tricked us

We now have a VM running on a snapshot with the number 256

In the vmware log, it opens all the snapshots and tries to boot, but it now has been running for over an hour and has not completed the boot yet

2022-04-12T07:36:44.887Z| Worker#1| I120: DISKLIB-LINK : Opened '/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB-000256.vmdk' (0xa): vmfsSparse, 251658240 sectors / 120 GB.
2022-04-12T07:36:44.891Z| vmx| I120: DISKLIB-CBT : Initializing ESX kernel change tracking for fid 532625.
2022-04-12T07:36:44.891Z| vmx| I120: DISKLIB-CBT : Successfuly created cbt node 82091-cbt.
2022-04-12T07:36:44.891Z| vmx| I120: DISKLIB-CBT : Opening cbt node /vmfs/devices/cbt/82091-cbt
2022-04-12T07:36:44.891Z| vmx| I120: DISKLIB-LIB : Opened "/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB_1.vmdk" (flags 0xa, type vmfs).
2022-04-12T07:36:44.892Z| vmx| I120: DISK: Disk '/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB_1.vmdk' has UUID '60 00 c2 96 38 74 9e ce-15 ac c9 2d ab f4 f1 27'
2022-04-12T07:36:44.892Z| vmx| I120: DISK: OPEN '/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB_1.vmdk' Geo (13054/255/63) BIOS Geo (0/0/0)
2022-04-12T07:36:44.892Z| vmx| I120: UTIL: Change file descriptor limit from soft 4096,hard 4096 to soft 4211,hard 4211.
2022-04-12T07:36:44.925Z| Worker#1| I120: DISKLIB-VMFS : "/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB-000255-delta.vmdk" : open successful (14) size = 302239744, hd = 598165. Type 8
2022-04-12T07:36:44.925Z| Worker#1| I120: DISKLIB-DSCPTR: Opened [0]: "DDB-000255-delta.vmdk" (0xe)
2022-04-12T07:36:44.925Z| Worker#1| I120: DISKLIB-LINK : Opened '/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB-000255.vmdk' (0xe): vmfsSparse, 251658240 sectors / 120 GB.
2022-04-12T07:36:44.941Z| Worker#1| I120: DISKLIB-VMFS : "/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB-000254-delta.vmdk" : open successful (14) size = 1510199296, hd = 663702. Type 8
2022-04-12T07:36:44.941Z| Worker#1| I120: DISKLIB-DSCPTR: Opened [0]: "DDB-000254-delta.vmdk" (0xe)
2022-04-12T07:36:44.941Z| Worker#1| I120: DISKLIB-LINK : Opened '/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB-000254.vmdk' (0xe): vmfsSparse, 251658240 sectors / 120 GB.
2022-04-12T07:36:44.951Z| Worker#1| I120: DISKLIB-VMFS : "/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB-000253-delta.vmdk" : open successful (14) size = 17027072, hd = 663703. Type 8
2022-04-12T07:36:44.951Z| Worker#1| I120: DISKLIB-DSCPTR: Opened [0]: "DDB-000253-delta.vmdk" (0xe)

..... and so on....

Is there any way I can consolidate that many snapshots?

They are not visible in the snapshot manager

Can something be done via SSH?

The backup is not even complete so we can not restore from it

Thanks 🙂

[continuum]

Suggested procedure has been tested hundred of times and is safe. ...
Power off VM.
Consolidate first 100 snapshots via vmkfstools into a new directory.
Consolidate next 100 snapshots via vmkfstools ....
Consolidate remaining snapshots ....

Option A: do it on the fly with a few very short downtimes
Option B: do it offline: this is the faster option

Dont do this if you are unexperienced with editing vmdk-descriptorfiles !!!

Ulli

[LarsDalgaard]

Hi

I have no option as to try and do it anyways

What is required to edit the descriptor files?

[continuum]

I highly recommend that we do it together - feel free to call me via skype then we should be back in save waters in a few hours.
We will need enough free space to store the basedisk / or basedisks in a new directory. Expect thick provisioned requirements ....

Ulli - skype: sanbarrow
If you are in germany - send a PM with your phonenumber

About editing descriptorfiles: I recommend to strictly use the embedded editor of WinSCP - it has the lowest failure-rate especially for newbies.

[continuum]

Here is the plan:
- power off VM
- vmkfstools -i snapshot000090.vmdk newdir/basedisk.vmdk
- when done edit snapshot000091.vmdk so that it points to newdir/basedisk.vmdk
- vmkfstools -i snapshot000182.vmdk newdir/basedisk2.vmdk
- when done edit snapshot000183.vmdk so that it points to newdir/basedisk2.vmdk
- vmkfstools -i snapshot000255.vmdk newdir/basedisk3.vmdk
- clean up and rename back to standard nameing conventions

[LarsDalgaard]

Can I call you by phone I only have access to teams

I´m in Denmark.. no problem

\Lars

[continuum]

check your PMs
I expect you have Teamviewer, Putty and WinSCP installed on your adminhost
Please dont use a Mac-adminhost - thats too painful ...

Ulli

[continuum]

we just started the consolidation at snap000090.

Ulli

[continuum]

started consolidation with snapshot000180

Ulli

[continuum]

all done successfully

Ulli

[LarsDalgaard]

Everything is back to normal 🙂

Thanks to Ullis effort, and superb guidance in helping me with this matter.