Veeam, snapshots, consolidation etc...

[Jeremyyy]

I am helping a buddy out and running into roadblocks for his problem.

I'm not a Veeam guy and stay away from the snapshots when I can so with that....

I am seeing a ton of ghost? missing in action snapshots or phantom open snaps when i browse the datastore but when in snapshot manager there is nothing there.

I have read taking a snapshot and checking the Quiesce Guest file system, then deleting the open snap. However this works 75% of the time I run into a VM that I have to power the system down clone the machine and wait.... then power on the clone and delete the troublesome VM. Time to time I get this error

"An error occurred while consolidating disks: msg.fileio.lock."

I am wondering if this is the case of Veeam or snapshots in general for any other snap based backup.

Personally I am a user of BackupExec and just doing file level backups.

I am adding a screenshot from a datastore of the problem I am seeing.

Any thoughts would be great.

[ch1ta]

I would say that the problem with orphaned snapshots has to do with VMware, rather than with Veeam, as the latter plays only "requestor" role, instructing VMware to create/remove VM snapshot.

The problem typically occurs when the VM original disk is still locked by another process at the time the remove (consolidate) snapshot command is received by the host. Hence, the unique identifier of snapshot is removed from the VMware database before the actual snapshot is deleted. As the result, the snapshot exists, but is no longer shown in Snapshot Manager.

Another reason might be high disk I/O latency at the time of consolidation process. See the following link for further explanations:  Best Practices For Virtual Machine Snapshots in VMware Environments.

In order to remove orphaned snapshots, you can create another snapshot, then perform a “delete all” operation in Snapshot Manager.  All snapshots, including the ones not listed in snapshot manager will be consolidated. Also, you can use VeeamONE to be informed about orphaned snapshots' presence.

Cheers.

[Wh33ly]

Mostly indeed this problem occurs when files are still locked, see similar behavior with different solutions.

Problem: VMDK disk is still hot-added to a VEEAM proxy and isn't properly released after the backup was done. Target VM cannot consolidate because the file is in use by another VM.

Solution: Check the VEEAM proxy if you use hot-add and check if you see more disks then normal. Probably you notice a few disks which point to another VMDK

Problem: File lock by ESX host, sometimes I see a file lock on a VM which isn't properly released some how. This can cause a consolidation fail.

Solution: You could use this article to investigate if there is a file lock and who the owner of the file lock is. Sometimes it's possible to find the mac-address of the host and target the "locker of the file".

VMware KB: Investigating virtual machine file locks on ESXi/ESX

Sometimes I can't find both and as last resort I restart the services of all the hosts in that particular cluster which resolves it always.

VMware KB: Restarting the Management agents on an ESXi or ESX host

After the lock is gone, do another consolidation and you see only the VMDK's are left which you expected to see in the first place:)

Hope this will help you solve the problem or get you starting searching for your solution

[Jeremyyy]

thanks guys!

I did notice one machine (proxy) with 15 random VMDKs attached to it. I just removed them from the VM.

Im not sure how it should really look but he has replicas setup which had a number of span, i deleted them all who needs those backups =p

At least now there is no errors/alerts about this issue.

I know a ton of people say VeeamOne to keep an eye out for these things so Ill have him install it and see if it helps.

thanks again for your ideas and thoughts.

[ReinerHeinz]

Have you tried restarting your ESX hosts?  sometime file locks can do funny things with phantom files turning up in the vm directory on the datastore.