Snapshots sometimes fail to delete

[LondonD]

HI

Sometimes the VM snapshots do not get deleted because off errors like:

- Unable to access file, unspecified filename

- Unable to access file, unspecified filename since it is locked

- The virtual disk is either corrupted or not a supported format

This only happens on the deletion of the snapshots, and happens on 1-2 VMs per day.

As far as I can tell it is not always the same VMs, but I'm still monitoring it

The Snapshots is created by NetBackups VCB Backup method and backs up 100%

I looks like the Snapshot deletion has a retry attempt that fails with the same error

   -  the object has already been delete or has not completely been created

What happens is the Consolidate helper does not show in Snapshot manager, and i need to go

through all the vmdk's (RVTools) to look for Orphaned/rogue snapshots and then manually delete them

Q: Why would the files be locked or snapshot removal fail?

Help will be appreciated

Thanks

[a_p_]

Welcome to the Community,

the first place to look at is NetBackup. Check whether there are any fixes or updates available and if the version you are using supports your version/build of vSphere.

André

[LondonD]

Thanks Andre

I look after the Virtual environment and we have a backup team looking after the backups

As far as I understand NetBackup is on ver 7.0 with a whole lot EB's (Engineering Bundles) applied.

I know there is a Symantec call open to resolve the issue but we need to work together to solve the problem.

So from my side I need to know what the possibilities are for the snapshot deletion failures are.

For example - wrong code, high I/O, VCB not releasing locks in time, ect

I have seen this happen with other backup products like Veeam which is not VCB based.

and remember it only happens on 1-2VMs per day which equates to less than 1% of the enviroment.

Denzel

[a_p_]

It ain't easy to troubleshoot without going through all the log files (ESXi as well as NetBackup). Depending on the number of VM's backed up at the same time on the same host, there could be a load/timing issue where the deletion of the snapshot is triggered before the backup process terminated (released the locks). Another possibility - although not very likely - is an overload on the backup server!?

I would start checking the log files and put together all the time stamps of all actions for one of the VM's which had issues.

André

[LondonD]

Thanks for the advise Andre

I will log a Support Call and send them the logs to analize.

[continuum]

you say you got a 1% failure rate ? - thats very good in my experience - for VCB I would even call it excellent.

I guess I am a bit biased as I only see cases where it fails and I have to clean up 😉

If I had to give a list of possible reasons this would start with ...
- obscure VSS issues inside the guest prevent snapshots

- stale locks

- stupid users

- the way the vmsd-file is implemented is too fragile

...

[PduPreez · Accepted solution]

Here is a KB you can read: http://kb.vmware.com/kb/1010310

There are a few reasons why this would happen

- Busy VM failing to commit snapshot

- Background tasks locking VM

- Backup application not releasing VM files.

I also agree that a 1% failure ratio is not bad, the problem is to manage it. This happens on all backup applications.

It looks like you manually need to search for the VMs where the snaps did not get deleted.

That needs to be addressed by the backup software to give you notification if a Snapshot could not be deleted.

I know some other backup Software gives you a notivication when the snapshot commit failed, so it is possible.

regards

Please award points if this is helpful/correct

[LondonD]

The Support Call managed to sort out the problem with the specific VM.

Turned out that a Host NIC was locking the -flat.vmdk File.

I agree with the notifications that would make it allot easier to manage, I will suggest this to the Backup team

Thanks for everyone responses