VMware Cloud Community
LarsDalgaard
Contributor
Contributor

Veeam created 256 snapshots VM unresponsive, needs consolidation

Hi

We are running an old esxi 6.0 and unfortuantely something has tricked us

We now have a VM running on a snapshot with the number 256

In the vmware log, it opens all the snapshots and tries to boot, but it now has been running for over an hour and has not completed the boot yet

2022-04-12T07:36:44.887Z| Worker#1| I120: DISKLIB-LINK : Opened '/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB-000256.vmdk' (0xa): vmfsSparse, 251658240 sectors / 120 GB.
2022-04-12T07:36:44.891Z| vmx| I120: DISKLIB-CBT : Initializing ESX kernel change tracking for fid 532625.
2022-04-12T07:36:44.891Z| vmx| I120: DISKLIB-CBT : Successfuly created cbt node 82091-cbt.
2022-04-12T07:36:44.891Z| vmx| I120: DISKLIB-CBT : Opening cbt node /vmfs/devices/cbt/82091-cbt
2022-04-12T07:36:44.891Z| vmx| I120: DISKLIB-LIB : Opened "/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB_1.vmdk" (flags 0xa, type vmfs).
2022-04-12T07:36:44.892Z| vmx| I120: DISK: Disk '/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB_1.vmdk' has UUID '60 00 c2 96 38 74 9e ce-15 ac c9 2d ab f4 f1 27'
2022-04-12T07:36:44.892Z| vmx| I120: DISK: OPEN '/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB_1.vmdk' Geo (13054/255/63) BIOS Geo (0/0/0)
2022-04-12T07:36:44.892Z| vmx| I120: UTIL: Change file descriptor limit from soft 4096,hard 4096 to soft 4211,hard 4211.
2022-04-12T07:36:44.925Z| Worker#1| I120: DISKLIB-VMFS : "/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB-000255-delta.vmdk" : open successful (14) size = 302239744, hd = 598165. Type 8
2022-04-12T07:36:44.925Z| Worker#1| I120: DISKLIB-DSCPTR: Opened [0]: "DDB-000255-delta.vmdk" (0xe)
2022-04-12T07:36:44.925Z| Worker#1| I120: DISKLIB-LINK : Opened '/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB-000255.vmdk' (0xe): vmfsSparse, 251658240 sectors / 120 GB.
2022-04-12T07:36:44.941Z| Worker#1| I120: DISKLIB-VMFS : "/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB-000254-delta.vmdk" : open successful (14) size = 1510199296, hd = 663702. Type 8
2022-04-12T07:36:44.941Z| Worker#1| I120: DISKLIB-DSCPTR: Opened [0]: "DDB-000254-delta.vmdk" (0xe)
2022-04-12T07:36:44.941Z| Worker#1| I120: DISKLIB-LINK : Opened '/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB-000254.vmdk' (0xe): vmfsSparse, 251658240 sectors / 120 GB.
2022-04-12T07:36:44.951Z| Worker#1| I120: DISKLIB-VMFS : "/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB-000253-delta.vmdk" : open successful (14) size = 17027072, hd = 663703. Type 8
2022-04-12T07:36:44.951Z| Worker#1| I120: DISKLIB-DSCPTR: Opened [0]: "DDB-000253-delta.vmdk" (0xe)

..... and so on....

Is there any way I can consolidate that many snapshots?

They are not visible in the snapshot manager

Can something be done via SSH?

The backup is not even complete so we can not restore from it

Thanks πŸ™‚

Reply
0 Kudos
10 Replies
continuum
Immortal
Immortal

Suggested procedure has been tested hundred of times and is safe. ...
Power off VM.
Consolidate first 100 snapshots via vmkfstools into a new directory.
Consolidate next 100 snapshots via vmkfstools ....
Consolidate remaining snapshots ....

Option A: do it on the fly with a few very short downtimes
Option B: do it offline: this is the faster option

Dont do this if you are unexperienced with editing vmdk-descriptorfiles !!!

Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
LarsDalgaard
Contributor
Contributor

Hi

I have no option as to try and do it anyways

What is required to edit the descriptor files?

Reply
0 Kudos
continuum
Immortal
Immortal

I highly recommend that we do it together - feel free to call me via skype then we should be back in save waters in a few hours.
We will need enough free space to store the basedisk / or basedisks in a new directory. Expect thick provisioned requirements ....

Ulli - skype: sanbarrow
If you are in germany - send a PM with your phonenumber

About editing descriptorfiles: I recommend to strictly use the embedded editor of WinSCP - it has the lowest failure-rate especially for newbies.

 


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
continuum
Immortal
Immortal

Here is the plan:
- power off VM
- vmkfstools -i snapshot000090.vmdk newdir/basedisk.vmdk
- when done edit snapshot000091.vmdk so that it points to newdir/basedisk.vmdk
- vmkfstools -i snapshot000182.vmdk newdir/basedisk2.vmdk
- when done edit snapshot000183.vmdk so that it points to newdir/basedisk2.vmdk
- vmkfstools -i snapshot000255.vmdk newdir/basedisk3.vmdk
- clean up and rename back to standard nameing conventions


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
LarsDalgaard
Contributor
Contributor

Can I call you by phone I only have access to teams

IΒ΄m in Denmark.. no problem

\Lars

Reply
0 Kudos
continuum
Immortal
Immortal

check your PMs
I expect you have Teamviewer, Putty and WinSCP installed on your adminhost
Please dont use a Mac-adminhost - thats too painful ...

 

Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
continuum
Immortal
Immortal

we just started the consolidation at snap000090.

Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
continuum
Immortal
Immortal

started consolidation with snapshot000180

 

Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
continuum
Immortal
Immortal

all done successfully

 

Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

LarsDalgaard
Contributor
Contributor

Everything is back to normal πŸ™‚

Thanks to Ullis effort, and superb guidance in helping me with this matter.

 

Reply
0 Kudos