VMware Cloud Community
GLSHNU
Contributor
Contributor
Jump to solution

[Help] Consolidated Helper

Hi,

we're using Veeam to backup the vms of our ESX Server ( ESX 4.0.0 Build 398348 ).

After a backup there is always a VM that still has a snapshot named "Consolidated Helper-0".

I can delete these snapshot trough the snapshot-manager, but the snapshots still exist in the Datastore of the vm.

The Support of Veeam said that they can't do anything to solve this problem because this snapshot is created from the esx

after the commit of the Veeam snapshot starts.

Does anybody have/had the same problem and a solution for this issue?

Regards

Benjamin

Reply
0 Kudos
1 Solution

Accepted Solutions
timemachine
Enthusiast
Enthusiast
Jump to solution

wht happened , did it work?

View solution in original post

Reply
0 Kudos
12 Replies
a_p_
Leadership
Leadership
Jump to solution

Can you confirm, that - after deleting the snapshot in the snapshot manager - the VM's HDD settings (either in the GUI or in the vmx file) show the base disk rather than a snapshot disk!? Are there any error messages in the VM's vmware.log? If changed block tracking is enabled, you could try to delete all the .ctk files in the VM's folder.

What build of ESX 4.0 are you currently running? (EDIT: forget this question, it's mentioned in your post Smiley Wink)

Is there only one snapshot file in the datastore?

André

Reply
0 Kudos
GLSHNU
Contributor
Contributor
Jump to solution

In the vmx file this file is set as disk:

scsi0:0.fileName = "srv1-ramsperger-000013.vmdk"

For the second hdd this file is set:

scsi0:1.fileName = "srv1-ramsperger_1-000013.vmdk"

Both files are snapshot files.

This is the only error message I have found in the logs:

Aug 10 15:17:31.067: mks| MKS: Base polling period is 10000us
Aug 10 15:17:32.011: mks| VNCENCODE 9 encoding mode change: (1024x768x24depth,32bpp,4096bytes/line)
Aug 10 15:17:32.170: mks| VNCENCODE 9 encoding mode change: (1024x768x24depth,32bpp,4096bytes/line)
Aug 10 15:17:33.170: mks| SOCKET 9 (140) recv error 104: Connection reset by peer
Aug 10 15:17:33.170: mks| SOCKET 9 (140) destroying VNC backend on socket error: 1
Aug 10 15:17:33.171: mks| MKS: Base polling period is 1000000us
Aug 10 15:17:39.568: mks| MKS: Base polling period is 10000us
Aug 10 15:17:40.410: mks| VNCENCODE 10 encoding mode change: (1024x768x24depth,32bpp,4096bytes/line)
Aug 10 15:17:40.602: mks| VNCENCODE 10 encoding mode change: (1024x768x24depth,32bpp,4096bytes/line)
Aug 10 15:22:20.020: vcpu-0| VIDE: Truncating 0x5a from 65520 bytes to 32768

Changed Block tracking is enabled for both disks but I think deleting the ctk files is to risky at the moment.

I attachd a Picture we you can see the snapshot files in the datastore

Reply
0 Kudos
john23
Commander
Commander
Jump to solution

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

This looks like one or more files are locked and cannot be write-accessed when deleting the snapshots.

What you should do is to

  • temporarily disable Veeam backup (to make sure it does not try to create a snapshot while doing the following steps)
  • delete all the ctk files (this has no effect on the VM itself, only an incremental/differential backup will take more time on the next run)
  • open the snapshot manager, create a new snapshot and then run "Delete All"
  • if the snapshots are not being deleted, please check the vmware.log file in the VM's folder for errors (e.g. access denied)

For the above I assume the disks are thick provisioned (250GB and 60GB) and you do have some free disk space on the datatsore.

André

Reply
0 Kudos
GLSHNU
Contributor
Contributor
Jump to solution

Thanks to both of you Smiley Happy . I will try it this evening and update the status

Reply
0 Kudos
GLSHNU
Contributor
Contributor
Jump to solution

I Have tested your ideas this weekend.

But sadley none helped.

The deltas are still there an the vm is still refering to those deltas in the vmx file.

I've watched in the log if there is an info but couldn't find any.

I have attached the log to this post

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Although I'm not 100% sure, I think there's still a stuck process running on the host which locks the base disk and prevents the snapshot consolidation.

Aug 14 02:11:59.934: vmx| AIOMGR: AIOMgr_OpenWithRetry: File '/vmfs/volumes/4a2f8bff-52313308-a9ac-003048c865f3/srv1-ramsperger/srv1-ramsperger-flat.vmdk' locked (try 0)
Aug 14 02:12:00.235: vmx| AIOMGR: AIOMgr_OpenWithRetry: File '/vmfs/volumes/4a2f8bff-52313308-a9ac-003048c865f3/srv1-ramsperger/srv1-ramsperger-flat.vmdk' locked (try 1)
Aug 14 02:12:00.537: vmx| AIOMGR: AIOMgr_OpenWithRetry: File '/vmfs/volumes/4a2f8bff-52313308-a9ac-003048c865f3/srv1-ramsperger/srv1-ramsperger-flat.vmdk' locked (try 2)
Aug 14 02:12:00.837: vmx| AIOMGR: AIOMgr_OpenWithRetry: File '/vmfs/volumes/4a2f8bff-52313308-a9ac-003048c865f3/srv1-ramsperger/srv1-ramsperger-flat.vmdk' locked (try 3)
Aug 14 02:12:01.137: vmx| AIOMGR: AIOMgr_OpenWithRetry: File '/vmfs/volumes/4a2f8bff-52313308-a9ac-003048c865f3/srv1-ramsperger/srv1-ramsperger-flat.vmdk' locked (try 4)
Aug 14 02:12:01.438: vmx| DISKLIB-VMFS  : "/vmfs/volumes/4a2f8bff-52313308-a9ac-003048c865f3/srv1-ramsperger/srv1-ramsperger-flat.vmdk" : failed to open (Failed to lock the file): AIOMgr_Open failed. Type 3
Aug 14 02:12:01.438: vmx| DISKLIB-LINK  : "/vmfs/volumes/4a2f8bff-52313308-a9ac-003048c865f3/srv1-ramsperger/srv1-ramsperger.vmdk" : failed to open (Failed to lock the file). 
Aug 14 02:12:01.438: vmx| DISKLIB-CHAIN : "/vmfs/volumes/4a2f8bff-52313308-a9ac-003048c865f3/srv1-ramsperger/srv1-ramsperger.vmdk" : failed to open (Failed to lock the file).
Aug 14 02:12:01.438: vmx| DISKLIB-LIB   : Failed to open '/vmfs/volumes/4a2f8bff-52313308-a9ac-003048c865f3/srv1-ramsperger/srv1-ramsperger.vmdk' with flags 0x20a (Failed to lock the file).
Aug 14 02:12:01.438: vmx| SNAPSHOT: SnapshotConsolidateOpenDisks failed: Failed to lock the file.

I think this could be a stuck backup process. Do you have a chance to reboot the ESXi host? This would most likely resolve the issue.

André

Reply
0 Kudos
timemachine
Enthusiast
Enthusiast
Jump to solution

Dear Glshnu,

I have run into such problems a lot of times my self. Irrespective of the backup method which i select , be it veeam or TSM sometimes we run into this issue where the thirdparty backup tool creates snaps during the backup but does not successfully delete it. I read through ur post. I recommend u do the following, the only condition is that you should have disk free space in either the present datastore or in someother one that is accessible from this esx.

what we are going to do is that we will start from the last snapshot, ie the one the vmdk file is presently pointing to and write all the data in the present snapshot and all intermediate ones , and finally the base diskinto a new disk file. then you can create a new vm point to this new disk file and power it on and see if all is well, if its good u can now get rid of the problematic vm by going into console and manually deleting all disk files.

steps:  1. open ssh connection to esx/console

2. power down the vm in question

3. run :   vmkfstools -i /vmfs/volumes/"datastore name where snapshots reside"/"name of last vmdk descritor file"  /vmfs/volumes/"datastore name where there is enough freespace"/new.vmdk

if your last snap file is old-000035.vmdk and it resides in a dstore with name datastore1, and assume you have enough space in this datastore and we will name the new disk file new.vmdk the cmd will be:

                vmkfstools -i /vmfs/volumes/datastore1/old-000035.vmdk /vmfs/volumes/datastore1/new.vmdk

4.once this completes create a new vm , point it to new.vmdk

5. power on new vm and see if all data exists. if yes you can just delete the entire vm folder of the old vm.

6. and IMPORTANT next time after every backup check if there are any undeleted snaps.

Reply
0 Kudos
timemachine
Enthusiast
Enthusiast
Jump to solution

Hi,

In addition to the above method.....if you are aprehensive of using cmd line....just perform a V to V conversion using converter standalone or the one in VC. this will consolidate the disks and then you can get rid of old files.

TM

Reply
0 Kudos
GLSHNU
Contributor
Contributor
Jump to solution

Hi timemachine,

thanks a lot Smiley Happy . I will try it this evening

Reply
0 Kudos
timemachine
Enthusiast
Enthusiast
Jump to solution

wht happened , did it work?

Reply
0 Kudos
GLSHNU
Contributor
Contributor
Jump to solution

Hi timemachine,

sorry for my late response (was very busy the lst days).

It worked perfectly Smiley Happy Thank you very much !!!!

Reply
0 Kudos