Hi,
we're using Veeam to backup the vms of our ESX Server ( ESX 4.0.0 Build 398348 ).
After a backup there is always a VM that still has a snapshot named "Consolidated Helper-0".
I can delete these snapshot trough the snapshot-manager, but the snapshots still exist in the Datastore of the vm.
The Support of Veeam said that they can't do anything to solve this problem because this snapshot is created from the esx
after the commit of the Veeam snapshot starts.
Does anybody have/had the same problem and a solution for this issue?
Regards
Benjamin
wht happened , did it work?
Can you confirm, that - after deleting the snapshot in the snapshot manager - the VM's HDD settings (either in the GUI or in the vmx file) show the base disk rather than a snapshot disk!? Are there any error messages in the VM's vmware.log? If changed block tracking is enabled, you could try to delete all the .ctk files in the VM's folder.
What build of ESX 4.0 are you currently running? (EDIT: forget this question, it's mentioned in your post )
Is there only one snapshot file in the datastore?
André
In the vmx file this file is set as disk:
scsi0:0.fileName = "srv1-ramsperger-000013.vmdk"
For the second hdd this file is set:
scsi0:1.fileName = "srv1-ramsperger_1-000013.vmdk"
This is the only error message I have found in the logs:
Changed Block tracking is enabled for both disks but I think deleting the ctk files is to risky at the moment.
I attachd a Picture we you can see the snapshot files in the datastore
check this out:
This looks like one or more files are locked and cannot be write-accessed when deleting the snapshots.
What you should do is to
For the above I assume the disks are thick provisioned (250GB and 60GB) and you do have some free disk space on the datatsore.
André
Thanks to both of you . I will try it this evening and update the status
Although I'm not 100% sure, I think there's still a stuck process running on the host which locks the base disk and prevents the snapshot consolidation.
Aug 14 02:11:59.934: vmx| AIOMGR: AIOMgr_OpenWithRetry: File '/vmfs/volumes/4a2f8bff-52313308-a9ac-003048c865f3/srv1-ramsperger/srv1-ramsperger-flat.vmdk' locked (try 0)
Aug 14 02:12:00.235: vmx| AIOMGR: AIOMgr_OpenWithRetry: File '/vmfs/volumes/4a2f8bff-52313308-a9ac-003048c865f3/srv1-ramsperger/srv1-ramsperger-flat.vmdk' locked (try 1)
Aug 14 02:12:00.537: vmx| AIOMGR: AIOMgr_OpenWithRetry: File '/vmfs/volumes/4a2f8bff-52313308-a9ac-003048c865f3/srv1-ramsperger/srv1-ramsperger-flat.vmdk' locked (try 2)
Aug 14 02:12:00.837: vmx| AIOMGR: AIOMgr_OpenWithRetry: File '/vmfs/volumes/4a2f8bff-52313308-a9ac-003048c865f3/srv1-ramsperger/srv1-ramsperger-flat.vmdk' locked (try 3)
Aug 14 02:12:01.137: vmx| AIOMGR: AIOMgr_OpenWithRetry: File '/vmfs/volumes/4a2f8bff-52313308-a9ac-003048c865f3/srv1-ramsperger/srv1-ramsperger-flat.vmdk' locked (try 4)
Aug 14 02:12:01.438: vmx| DISKLIB-VMFS : "/vmfs/volumes/4a2f8bff-52313308-a9ac-003048c865f3/srv1-ramsperger/srv1-ramsperger-flat.vmdk" : failed to open (Failed to lock the file): AIOMgr_Open failed. Type 3
Aug 14 02:12:01.438: vmx| DISKLIB-LINK : "/vmfs/volumes/4a2f8bff-52313308-a9ac-003048c865f3/srv1-ramsperger/srv1-ramsperger.vmdk" : failed to open (Failed to lock the file).
Aug 14 02:12:01.438: vmx| DISKLIB-CHAIN : "/vmfs/volumes/4a2f8bff-52313308-a9ac-003048c865f3/srv1-ramsperger/srv1-ramsperger.vmdk" : failed to open (Failed to lock the file).
Aug 14 02:12:01.438: vmx| DISKLIB-LIB : Failed to open '/vmfs/volumes/4a2f8bff-52313308-a9ac-003048c865f3/srv1-ramsperger/srv1-ramsperger.vmdk' with flags 0x20a (Failed to lock the file).
Aug 14 02:12:01.438: vmx| SNAPSHOT: SnapshotConsolidateOpenDisks failed: Failed to lock the file.
I think this could be a stuck backup process. Do you have a chance to reboot the ESXi host? This would most likely resolve the issue.
André
Dear Glshnu,
I have run into such problems a lot of times my self. Irrespective of the backup method which i select , be it veeam or TSM sometimes we run into this issue where the thirdparty backup tool creates snaps during the backup but does not successfully delete it. I read through ur post. I recommend u do the following, the only condition is that you should have disk free space in either the present datastore or in someother one that is accessible from this esx.
what we are going to do is that we will start from the last snapshot, ie the one the vmdk file is presently pointing to and write all the data in the present snapshot and all intermediate ones , and finally the base diskinto a new disk file. then you can create a new vm point to this new disk file and power it on and see if all is well, if its good u can now get rid of the problematic vm by going into console and manually deleting all disk files.
steps: 1. open ssh connection to esx/console
2. power down the vm in question
3. run : vmkfstools -i /vmfs/volumes/"datastore name where snapshots reside"/"name of last vmdk descritor file" /vmfs/volumes/"datastore name where there is enough freespace"/new.vmdk
if your last snap file is old-000035.vmdk and it resides in a dstore with name datastore1, and assume you have enough space in this datastore and we will name the new disk file new.vmdk the cmd will be:
vmkfstools -i /vmfs/volumes/datastore1/old-000035.vmdk /vmfs/volumes/datastore1/new.vmdk
4.once this completes create a new vm , point it to new.vmdk
5. power on new vm and see if all data exists. if yes you can just delete the entire vm folder of the old vm.
6. and IMPORTANT next time after every backup check if there are any undeleted snaps.
Hi,
In addition to the above method.....if you are aprehensive of using cmd line....just perform a V to V conversion using converter standalone or the one in VC. this will consolidate the disks and then you can get rid of old files.
TM
Hi timemachine,
thanks a lot . I will try it this evening
wht happened , did it work?
Hi timemachine,
sorry for my late response (was very busy the lst days).
It worked perfectly Thank you very much !!!!