VMware Cloud Community
coastalitsuppor
Enthusiast
Enthusiast
Jump to solution

Snapshot won’t delete

Hi all,

wandering if someone can please assist.

The backup program creates a snap shot as per usual, however if we consolidate says complete but nothing happens. If you try remove a snapshot manually single, all, children etc keeps failing. I have tried through powercli also and same see error below. We have had to stop backups as our data store is about to run out of space. It’s creating 400GB snapshots and we have about 14 we need removed. Even if we remove every snap shot and start fresh don’t mind.

FF398ADD-42F6-4533-B887-9ED510203896.jpeg

 thank you

Labels (3)
Tags (3)
Reply
0 Kudos
1 Solution

Accepted Solutions
continuum
Immortal
Immortal
Jump to solution

> Discussing snapshots problem without details is a bit risky .

When the details are available it is so much easier ...
We just needed less than an hour via Teamviewer to fix the problem.

Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

View solution in original post

19 Replies
NathanosBlightc
Commander
Commander
Jump to solution

First, check the detailed information of the *.vmsd file in this VM's directory and compare the snapshots that you couldn't remove, are they mentioned there or not? ...

Then try consolidation operation again while you directly connected to the ESXi host (not via the Center server or executing Power-CLI) then give back the result of this, please.

Please mark my comment as the Correct Answer if this solution resolved your problem
ashilkrishnan
VMware Employee
VMware Employee
Jump to solution

Hi @coastalitsuppor,

If Veeam is a VM within vSphere environment, go to edit settings of it and ensure it does not have the disk of the problematic VM attached to it. If yes, detach it.

Create a test snapshot within vSphere and initiate delete all to consolidate the snapshots

continuum
Immortal
Immortal
Jump to solution

Discussing snapshots problem without details is a bit risky .
Send all descriptor-vmdks, the vmsd and latest 2 vmware.logs.

Do the snpshot files appear in /dev/disks/deltadisks/ ?

When you use automatic backup-tools you MUST also use daily monitoring because if you run out of space with a sesparse snapshot it probably gets corrupted.


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

coastalitsuppor
Enthusiast
Enthusiast
Jump to solution

Hi all,

Appreciate the feedback. Please see attached latest log and vsrm file as only allows me to upload the ZIP. The screen shot below you can see in the data store we actually have about 24 snapshots attached.

Screen Shot 2020-12-02 at 8.31.48 am.png

 

Reply
0 Kudos
continuum
Immortal
Immortal
Jump to solution

Rule number one when troubleshooting snapshots: DONT USE DATASTORE-BROWSER
When troubleshooting snapshots you need to see the content of the directory as it really is - not the dumbed down oversimplified version that you see in the datastorebrowser.
Easiest tool is WinSCP
With WinSCP you will also see the vmdk-descriptorfiles that I asked for ...

from the vmware.log:

SNAPSHOT: SnapshotGenerateDeleteDisks Failed to fetch disk tree: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot (21)
SNAPSHOT: SnapshotDeleteNode failed: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot (21)
SNAPSHOT: Snapshot_Delete failed: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot (21)

from the vmsd:
snapshot0.disk0.fileName = "SERVER 2019 DC-000004.vmdk"
snapshot2.disk0.fileName = "SERVER 2019 DC-000004.vmdk" !!! vmdk appears twice = vmsd is bad

This means you have a bad vmsd - and the result is Veeam can not clean up after each backup = you will run out of free space soon.
Because of the bad vmsd you also cant use the Consolidation function.

Suggested fix : created empty sub-directory "deleteme-later" and move the vmsd-file into it.

Ulli

 


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

coastalitsuppor
Enthusiast
Enthusiast
Jump to solution

Hi Ulli,

 

Thanks for the updated and noted for the DS browser. Appreciate the advice.

 

So from understanding below, if just create a blank folder in the DS and move the vmdd file into it, that should allow me to consolidate or delete the snap shots? Guessing the VM would have to be shutdown or restarted after moving the file to rebuild a new one?

Thank you appreciate you advice and time!

Mark

Reply
0 Kudos
coastalitsuppor
Enthusiast
Enthusiast
Jump to solution

Hi,

This is what's listed in the dev/disks as you asked also.

 

Screen Shot 2020-12-02 at 11.33.59 am.png

Reply
0 Kudos
continuum
Immortal
Immortal
Jump to solution

short answer via phone

the suggest fix is a complete list, dont do anything else.

No need to reboot  - a running vm does not need a vmsd.

If winscp does not allow to move the vmsd - rename it to invalid.vmsd , wait half a minute and then move to deleteme-later dir.

Next Veeam backup should be a full one. 

snaphotmanager in the gui will be confused and display nothing. you should not use  it when veeam jobs are running so thats no problem either.

When you want to consolidate just use create new and delete all.

Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
coastalitsuppor
Enthusiast
Enthusiast
Jump to solution

Ok thank you. Have renamed and can see its cleared the previous snapshot names. Then can go back and manually delete all those old ones? Tried pressing consolidate but nothing happens and the 26 existing snapshots are still there

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Did you already check whether one of the VM's .vmdk files (virtual disks) is still mounted to one of the backup servers, or proxies?
I've seen issues like this a few times when backup jobs failed unexpectedly, and the backup server was not able to do the cleanup (unmont the virtual disk).

André

Reply
0 Kudos
continuum
Immortal
Immortal
Jump to solution

> Tried pressing consolidate but nothing happens and the 26 existing snapshots are still there

I told you that snapshotmanager will be confused and that you should not use it.
So instead create a new snapshot and then use the delete-all function.

@a_p_ 
Andre - if I remember right the error message would be different and would complain about locked vmdks - not about "already in use"

It would help to see the content of /dev/deltadisks ...


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
coastalitsuppor
Enthusiast
Enthusiast
Jump to solution

Anyone able to give me some help? The DC wont turn on now. Anyone on Skype even able to assist???

Reply
0 Kudos
coastalitsuppor
Enthusiast
Enthusiast
Jump to solution


Failed - Unable to enumerate all disks.

Errors


•Unable to enumerate all disks.
•A required file was not found

Reply
0 Kudos
scott28tt
VMware Employee
VMware Employee
Jump to solution

@coastalitsuppor 

Moderator: Please note the following:

1. Moved this thread to the Backup & Recovery Discussions area

2. Please do not create duplicate threads for the same issue - the second thread you created has been archived


-------------------------------------------------------------------------------------------------------------------------------------------------------------

Although I am a VMware employee I contribute to VMware Communities voluntarily (ie. not in any official capacity)
VMware Training & Certification blog
Reply
0 Kudos
continuum
Immortal
Immortal
Jump to solution

Sorry - last night I was too tired to help via skype ...

Please try to provide more details - we dont know anything about your DC - is that the same VM that we discussed already ?
Quoting error messages without details does not help either.
If you receive an error message - it will appear in the vmware.log.
In the vmware.log you will see details that are necessary to deal with the problem.

If you read your last post - the obvious question is "which file ?"
Please answer all obvious questions when you post a question - that will help to find a solution faster.

Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
continuum
Immortal
Immortal
Jump to solution

> Discussing snapshots problem without details is a bit risky .

When the details are available it is so much easier ...
We just needed less than an hour via Teamviewer to fix the problem.

Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

coastalitsuppor
Enthusiast
Enthusiast
Jump to solution

Thank you to Sanbarrow (Ulli) who managed to get this fixed. Was very prompt, very helpful and really knows his stuff was very impressed!!

Reply
0 Kudos
hamidja
Enthusiast
Enthusiast
Jump to solution

Hi

one of advantages of cloning is to consolidate all snapshots.sometimes old snapshots get stuck when we need to consolidate them.in this situation you need to create a clone from VM .your clone is consolidated.but parent stay in untouched.now you can use clone VM and remove parent.

hope this help you

Reply
0 Kudos
DizZy_64
Contributor
Contributor
Jump to solution

Hi!

I got almost the same problem (see attachments):

from the VMWare log after vim-cmd vmsvc/snapshot.removeall 37 :

SNAPSHOT: SnapshotDiskTreeAddFromSnapshot: Trying to add snapshot SRV-E3-Snapshot574.vmsn to disk /vmfs/volumes/58beb247-340b1d1a-fc73-1402ec905020/SRV-E3/SRV-E3_2.vmdk which already has snapshot SRV-E3-Snapshot573.vmsn.
SNAPSHOT: SnapshotGenerateDeleteDisks Failed to fetch disk tree: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot (21)
SNAPSHOT: SnapshotDeleteNode failed: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot (21)
SNAPSHOT: Snapshot_Delete failed: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot (21)

from the vmsd:

snapshot0.uid = "573"
snapshot0.disk0.fileName = "SRV-E3_2.vmdk"
snapshot0.disk1.fileName = "SRV-E3_3.vmdk"
snapshot0.disk2.fileName = "SRV-E3_4.vmdk"

snapshot1.uid = "574"
snapshot1.disk0.fileName = "SRV-E3_2.vmdk"
snapshot1.disk1.fileName = "SRV-E3_3.vmdk"
snapshot1.disk2.fileName = "SRV-E3_4.vmdk"

I suppose the snapshot1.diskx.filename should be SRV-E3_x-000001.vmdk.

I see two options:

1) Should I try to manually correct the vmsd file and retry the vim-cmd vmsvc/snapshot.removeall 37?

2) Should I, as suggested in your solution,move the vmsd file?

In case of option 2, if I understand correctly the next steps are:

1) Create a new snapshot (vim-cmd vmsvc/snapshot.create which will create a new vmsd

2) Attempt a new vim-cmd vmsvc/snapshot.removeall 37

Is this correct? Do I have to shutdown the server? (It's my production Exchange server...)

Reply
0 Kudos