VMware Cloud Community
ptimme01
Contributor
Contributor

Consolidation Helper Snapshot

in our esx3.0 snapshot manager, we see a consolidation helper snapshot.

what is this? how does it appear?

how do we get rid of it (when we vmotion throws an error due to the vm having a snapshot).

0 Kudos
16 Replies
wunderon
Hot Shot
Hot Shot

That is a snapshot created (probably) by Vmware Consolidated Backup. If the pre-backup/post-backup scripts get out of wack, a snapshot can be left as a residual. I've had this happen a couple of times, although only early in testing..

0 Kudos
migueltorres
Contributor
Contributor

We had one of those today, and the problem was that we didn't know it, so the Snapshot created delta files and they grew until the LUN ran out of space and the server crashed. To delete just go to the snapshot manager and click on delete snapshot, this will commit all the changes to the original file and give you back all the space.

The problem is I can't find the reason why these are created. We do not use consolidated backup.

0 Kudos
admin
Immortal
Immortal

These are almost always created when there is an error during the clean-up faze after a VCB backup, I've had many of them, always with the same VMs. You get an error "unable to remove snapshot...attempting to clean up snapshot database" if you're running it from the command line. The trouble is once they're there you can only remove them in the VI client when the VM is shut down, if you try with the VM powered on you get 'unable to remove a file since it is locked'. Very annoying! I've got an SR open on it but not response as yet....I'll let you know when I get one.

I did read a suggestion on here that removing the snapshot from the command line using vmware-cmd would allow you to remove the snapshot without powering down the VM, but I haven't had a chance to try that yet.

I have seen these 'consolodate helper' snapshots get created without using VCB also, I had a VM give an error when trying to delete a snapshot, again a file locking problem, and after the failure the snapshot was renamed to the dreaded 'consolodate helper' and I was stuck in the above un-removable snapshot situation again.

0 Kudos
bluepenguin
Enthusiast
Enthusiast

Hi,

We have experienced the same problem:

If the vmdk-files of one VM are located in 2 different directories or on 2 different LUNs and have the same filename, you cannot commit (delete) the snapshot while the VM is powered on, you have to power off the VM. The Consolidate Helper Snapshot is an interim snapshot that ESX creates in order to do a non-disruptive commit of the data.

Example:

SCSI0:0 is in /vmfs/volumes/lun[b]0[/b]/myvm/myvm.vmdk

SCSI0:1 is in /vmfs/volumes/lun[b]1[/b]/myvm/myvm.vmdk

We have an SR open on this, VMware told us that it is not best practice to put disk files in different locations. (Which is true, but the bug is still there)

Ciao,

Frank

0 Kudos
admin
Immortal
Immortal

Aha, interesting! This explains why I've been seeing this problem on a few VMs but not others. Thanks for your post.

I've got VMs where the OS drive is on RAID1, and the DB drive on RAID5, and I know see these are the ones I've been having trouble with.

I've also got an SR open with VMware on this, I'll point them in the direction of this thread and hopefully we can get it fixed in the next point release.

0 Kudos
conyards
Expert
Expert

Did the open SR here result in any progress or workarounds?

https://virtual-simon.co.uk/
0 Kudos
bluepenguin
Enthusiast
Enthusiast

Hi,

Yes, VMware says that the problem will be fixed in version x.y.z. If you like, open another SR, the more SRs VMware recieves, the faster the problem gets fixed.

To work around the problem, rename the disk files so each filename is unique.

Ciao,

Frank

0 Kudos
conyards
Expert
Expert

Hi thanks for the response,

I have renamed the vmdks using vmkfstools -E \[oldname] \[newname] and updated the VM in a number of ways from;

removing/adding the disk back in from VC

editing the VMX file to reflect the new location

and removing/adding the disks and removing the VM from VC and adding it back in.

When I run my scripts on a VM with renamed disks the damn script hangs after successfully creating the snapshot....

am I missing a step here?

edit: as you can see from this thread I've been having great fun with VCB...

http://www.vmware.com/community/thread.jspa?threadID=64693&tstart=0

Message was edited by:

conyards

https://virtual-simon.co.uk/
0 Kudos
ITQPG
Contributor
Contributor

I think the answer to the SR record by vmware is a bit strange, to say the least...

"VMware told us that it is not best practice to put disk files in different locations"

In a large environment it is very common to have different speed in SAN disk access since FC disks are more expensive than FATA (Fibre Attached Technology Adapted) disks.

In our environment (HP blades connected to EVA8000 storage) all Virtual servers with data disks have there OS vmdk on FC (fast) disks an there DATA disks on FATA disks.

If VMware wants to be BIG they have to think BIG and listen to there partners and users of there software.

Those best practice answers are not what we are waiting for in this forum.

0 Kudos
analyst7
Contributor
Contributor

I found that this was only happening for me on about 3 virtual machines. All three were multiple disk machines that were migrated from esx 2.5.3. They also all had Undoable mode enabled on them at one time in the 2.5x world which I had to convert to "persistent" before migrating to a vmfs3 lun for the vi3 upgrade.

To eliminate the issue of the Consolidated Helper snapshot being created and giving me grief on the vi3 side of things I needed to edit the .vmx file and remove "persistent" from the scsi0:(disk).mode = "persistent". I simply left it with empty quotes (""). I have not had any issues on these machines since.

0 Kudos
Mike_Deardurff
Enthusiast
Enthusiast

Hi,

Simply put, Im having the same problem as most of you. I know im coming in on this thread fairly late, but i was wondering if there have been any updates as to a fix from VMware for this "consolidate helper" issue.

In my case, my scripts within Commvault for VCB are (i guess) not working properly and im getting these residual snapshots and consolidate helper snaps for my VM's. Unfortunately, i will have to power off to delete them if im going to get rid of them, what a pain!

Anyone else here an update or a better fix for this darn thing.

Thanks

-Mike

0 Kudos
botanic74
Contributor
Contributor

Hi,

I'm trying to make VCB work for a VM running on one virtual disk, and it doesn't work.

We use NetBackup 5.1 MP5 to launch the backup process, and I have an error telling me that a snapshot already exists ... and yes, there is the Consolidate Helper snapshot !

NetBackup is configured to launch simultaneously several file-level backup (not file and image level backups at a time !) with the NEW STREAM directive ...

This directive permits to launch several VCB backup process at a time in the same Netbackup Policy, not only one at a time ...

I tried to launch several jobs in multiple Netbackup Policies, and it works ... so I think that it comes from the multiple streaming management in Netbackup.

I hope this will help VMWare to find the origin of the problem.

0 Kudos
riconeo
Contributor
Contributor

This should solve the problem.

1) SSH in the ESX where the problematic VM resides

2) Type the command below to restart the services:

service vmware-vpxa stop

service mgmt-vmware restart

service vmware-vpxa start

3) Try deleting the snapshots again, you should be able to do it now.

http://ricotroubleshooting.blogspot.com/
0 Kudos
Heinrich928
Contributor
Contributor

I'v had the same problem, and your suggestion did it! problem solved

It's a 3.5 U4 in a Cluster

Thank you very much for posting this solution

heinrich

0 Kudos
Beansontoast
Contributor
Contributor

Does anyone know what impact running these commands may have ?

We have another six busy production VM's sharing the same datastore as the offending VM (with the Consolidated Helper snapshot).

0 Kudos
ElioteArg
Contributor
Contributor

Hello,

I'm having a similar problem with the Consolidated Helper. The fact is that I do not have any backup tool and I do not know why the Consolidated Helper snapshot appeared. Besides, I cannot power off, neither power on the virtual machine. I cannot delete the snapshot, I cannot unregister th vm and I do now know what else to do. I tried to look for the pid associated to the process of this vm, but it doesn't exist. I tried powering off the machine via command line, and it says that the operation is not allowed. I think a have a Zombie VM.

Does any one how to solve this issue???

0 Kudos