Cannot remove snapshot

[Blitz_Fandango]

The information:

• ESX 3.5 U4

• Virtual Center 2.5

• VCB Backups using VEAM

The Problem:

I decided to look in on my backup jobs this morning and discovered a bit of a problem. One of my virtual machines (a SQL database server) had two unresolved snapshots. Both were labeled as "Consoidate Helper". So this usually means they were put in place during a snapshot consolidation job. Usually caused by the VCB backup jobs resolving snapshots over the course of the backup job. All normal aside from the fact that these should have been removed automatically.

Due to these consolidation snapshots remaining, the LUN where the disk files reside was begining to become full. I had caught this issue with roughly 5GB of free space left. As the situation was going to become a problem very soon I elected to consolidate the snapshots (remove all snapshots from the snapshot manager). The snapshot removal process timed out within virtual center. Not entirely surprising as large snaphots can take a good deal of time to consolidate. The snapshot files consist of more than 80GB. That's just the snapshot files. Not the original vmdk files.

The snapshot removal process appears to have failed entirely. I'm left with 80+GB of snapshot files and vmware insisting that no snapshots exist. The snapshot manager says there are no snapshots. 'vmware-cmd pathtovmx hassnapshots' reports no snapshots. 'vmwarep-cmd pathtovmx remove snapshots' reports there are no snapshots to remove. And yet, the files still exist and data is going to them. I apparently have no way to remove the snapshots.

Has anyone experienced this and found a way to resolve it? My only idea at the moment is to take down the VM and copy it to a new VM. Not a great solution. This is a production system. I've also seen some posts that seem to indicate that having the VM powered off may allow the consolidation to happen. Any experiences here with VMs that refuse to admit they have snapshots?

I've migrated all the other VMs off this particular LUN and now have around 120GB free. So I've got a little time to play with things before this turns into a major issue. But I'm not quite sure where to go from here. Database backups will be happening everynight. So that's going to chew up space. And, as luck would have it, there's a major upgrade of the software going on this weekend and this will also chew up a lot of space. So, I need some ideas on where to go from here.

Thanks for any advice/experience.

[NTurnbull]

Hi, as you said large snapshots can take some time to delete, how long has it been since you deleted them?

Thanks,

Neil

[Blitz_Fandango]

It's been roughly 3 hours since I initiated the snapshot removal. A bit longer than I'm used to. Is there any way to verify the process? I'm not sure what process would be showing in ps if there were a disk consolidation in process.

[ShanVMLand]

Generally, Snapshot deletion will take time, it depends on size of your VMDK. This KB helps to monitor snapshot deletion process.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100756...

Shan

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

[Blitz_Fandango]

Thanks for the link. Unfortunately it doesn't help much. I'm now 4 hours deep since initiating the snapshot removals. There doesn't seem to be much happening. Your link does seem to confirm that there is no real way to validate the consolidation process. The files are there. They don't seem to be going anywhere soon.

Any more ideas?

[Blitz_Fandango]

Further information:

Daily backups have begun. So VCB has initiated an entirely new set of snapshot files. My hope is that the consolidation of these new snapshots will get rid of everything. But my hope is not very high. We'll see what happens.

[Blitz_Fandango]

Apparently luck was on my side today. As it turns out, the VCB backup was able to create a new snapshot and almost remove it. I was once again left with a consolidation helper snapshot left behind. When instructed to remove this, it happily complied and all snapshots now appear to be gone.

Crisis averted. For now.

Does anyone else have issues with VCB snapshots not clearing properly? I suppose I should move this question over to a VCB forum. Just curious.

[AndreTheGiant]

Have a look at this thread:

Andre

[AWarwick]

Some information for you. I had to remove a snapshot for one of my customers. It had grown to over 100GB. It took 24 hours to delete. Even though VC tells you it timed out, it didn't. It has to roll back the snap into the original vmdk file. Also, in the last 2 hours the VM became unresponsive and went offline. We decided to wait and all was well when it finished.

Alan