coastalitsuppor
Enthusiast
Enthusiast

Snapshot won’t delete

Jump to solution

Hi all,

wandering if someone can please assist.

The backup program creates a snap shot as per usual, however if we consolidate says complete but nothing happens. If you try remove a snapshot manually single, all, children etc keeps failing. I have tried through powercli also and same see error below. We have had to stop backups as our data store is about to run out of space. It’s creating 400GB snapshots and we have about 14 we need removed. Even if we remove every snap shot and start fresh don’t mind.

FF398ADD-42F6-4533-B887-9ED510203896.jpeg

 thank you

Labels (3)
Tags (3)
0 Kudos
1 Solution

Accepted Solutions
continuum
Immortal
Immortal

> Discussing snapshots problem without details is a bit risky .

When the details are available it is so much easier ...
We just needed less than an hour via Teamviewer to fix the problem.

Ulli

Do you need support with a recovery problem ? - send a message via skype "sanbarrow"

View solution in original post

18 Replies
NathanosBlightc
Commander
Commander

First, check the detailed information of the *.vmsd file in this VM's directory and compare the snapshots that you couldn't remove, are they mentioned there or not? ...

Then try consolidation operation again while you directly connected to the ESXi host (not via the Center server or executing Power-CLI) then give back the result of this, please.

Please mark my comment as the Correct Answer if this solution resolved your problem
ashilkrishnan
VMware Employee
VMware Employee

Hi @coastalitsuppor,

If Veeam is a VM within vSphere environment, go to edit settings of it and ensure it does not have the disk of the problematic VM attached to it. If yes, detach it.

Create a test snapshot within vSphere and initiate delete all to consolidate the snapshots

continuum
Immortal
Immortal

Discussing snapshots problem without details is a bit risky .
Send all descriptor-vmdks, the vmsd and latest 2 vmware.logs.

Do the snpshot files appear in /dev/disks/deltadisks/ ?

When you use automatic backup-tools you MUST also use daily monitoring because if you run out of space with a sesparse snapshot it probably gets corrupted.

Do you need support with a recovery problem ? - send a message via skype "sanbarrow"
coastalitsuppor
Enthusiast
Enthusiast

Hi all,

Appreciate the feedback. Please see attached latest log and vsrm file as only allows me to upload the ZIP. The screen shot below you can see in the data store we actually have about 24 snapshots attached.

Screen Shot 2020-12-02 at 8.31.48 am.png

 

0 Kudos
continuum
Immortal
Immortal

Rule number one when troubleshooting snapshots: DONT USE DATASTORE-BROWSER
When troubleshooting snapshots you need to see the content of the directory as it really is - not the dumbed down oversimplified version that you see in the datastorebrowser.
Easiest tool is WinSCP
With WinSCP you will also see the vmdk-descriptorfiles that I asked for ...

from the vmware.log:

SNAPSHOT: SnapshotGenerateDeleteDisks Failed to fetch disk tree: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot (21)
SNAPSHOT: SnapshotDeleteNode failed: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot (21)
SNAPSHOT: Snapshot_Delete failed: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot (21)

from the vmsd:
snapshot0.disk0.fileName = "SERVER 2019 DC-000004.vmdk"
snapshot2.disk0.fileName = "SERVER 2019 DC-000004.vmdk" !!! vmdk appears twice = vmsd is bad

This means you have a bad vmsd - and the result is Veeam can not clean up after each backup = you will run out of free space soon.
Because of the bad vmsd you also cant use the Consolidation function.

Suggested fix : created empty sub-directory "deleteme-later" and move the vmsd-file into it.

Ulli

 

Do you need support with a recovery problem ? - send a message via skype "sanbarrow"
coastalitsuppor
Enthusiast
Enthusiast

Hi Ulli,

 

Thanks for the updated and noted for the DS browser. Appreciate the advice.

 

So from understanding below, if just create a blank folder in the DS and move the vmdd file into it, that should allow me to consolidate or delete the snap shots? Guessing the VM would have to be shutdown or restarted after moving the file to rebuild a new one?

Thank you appreciate you advice and time!

Mark

0 Kudos
coastalitsuppor
Enthusiast
Enthusiast

Hi,

This is what's listed in the dev/disks as you asked also.

 

Screen Shot 2020-12-02 at 11.33.59 am.png

0 Kudos
continuum
Immortal
Immortal

short answer via phone

the suggest fix is a complete list, dont do anything else.

No need to reboot  - a running vm does not need a vmsd.

If winscp does not allow to move the vmsd - rename it to invalid.vmsd , wait half a minute and then move to deleteme-later dir.

Next Veeam backup should be a full one. 

snaphotmanager in the gui will be confused and display nothing. you should not use  it when veeam jobs are running so thats no problem either.

When you want to consolidate just use create new and delete all.

Ulli

Do you need support with a recovery problem ? - send a message via skype "sanbarrow"
0 Kudos
coastalitsuppor
Enthusiast
Enthusiast

Ok thank you. Have renamed and can see its cleared the previous snapshot names. Then can go back and manually delete all those old ones? Tried pressing consolidate but nothing happens and the 26 existing snapshots are still there

0 Kudos
a_p_
Leadership
Leadership

Did you already check whether one of the VM's .vmdk files (virtual disks) is still mounted to one of the backup servers, or proxies?
I've seen issues like this a few times when backup jobs failed unexpectedly, and the backup server was not able to do the cleanup (unmont the virtual disk).

André

0 Kudos
continuum
Immortal
Immortal

> Tried pressing consolidate but nothing happens and the 26 existing snapshots are still there

I told you that snapshotmanager will be confused and that you should not use it.
So instead create a new snapshot and then use the delete-all function.

@a_p_ 
Andre - if I remember right the error message would be different and would complain about locked vmdks - not about "already in use"

It would help to see the content of /dev/deltadisks ...

Do you need support with a recovery problem ? - send a message via skype "sanbarrow"
0 Kudos
coastalitsuppor
Enthusiast
Enthusiast

Anyone able to give me some help? The DC wont turn on now. Anyone on Skype even able to assist???

0 Kudos
coastalitsuppor
Enthusiast
Enthusiast


Failed - Unable to enumerate all disks.

Errors


•Unable to enumerate all disks.
•A required file was not found

0 Kudos
scott28tt
VMware Employee
VMware Employee

@coastalitsuppor 

Moderator: Please note the following:

1. Moved this thread to the Backup & Recovery Discussions area

2. Please do not create duplicate threads for the same issue - the second thread you created has been archived

0 Kudos
continuum
Immortal
Immortal

Sorry - last night I was too tired to help via skype ...

Please try to provide more details - we dont know anything about your DC - is that the same VM that we discussed already ?
Quoting error messages without details does not help either.
If you receive an error message - it will appear in the vmware.log.
In the vmware.log you will see details that are necessary to deal with the problem.

If you read your last post - the obvious question is "which file ?"
Please answer all obvious questions when you post a question - that will help to find a solution faster.

Ulli

Do you need support with a recovery problem ? - send a message via skype "sanbarrow"
0 Kudos
continuum
Immortal
Immortal

> Discussing snapshots problem without details is a bit risky .

When the details are available it is so much easier ...
We just needed less than an hour via Teamviewer to fix the problem.

Ulli

Do you need support with a recovery problem ? - send a message via skype "sanbarrow"

View solution in original post

coastalitsuppor
Enthusiast
Enthusiast

Thank you to Sanbarrow (Ulli) who managed to get this fixed. Was very prompt, very helpful and really knows his stuff was very impressed!!

0 Kudos
hamidja
Enthusiast
Enthusiast

Hi

one of advantages of cloning is to consolidate all snapshots.sometimes old snapshots get stuck when we need to consolidate them.in this situation you need to create a clone from VM .your clone is consolidated.but parent stay in untouched.now you can use clone VM and remove parent.

hope this help you

0 Kudos