VMware Cloud Community
asp24
Enthusiast
Enthusiast
Jump to solution

not able to remove snapshots!

I have a big problem. I was trying the Vmware Data Recovery solution, but had some problems with IO errors on the destination SMB, so several times this was aborted etc.

Now 2 days later I see that I have multiple snapshots on several servers. They do not show in snapshot manager, but they show up when I click edit settings and click on the disks (and when I ssh to the host and "ls" the vmfs dir.

I have tried creating a new snapshot and then remove it, and then it is removed from the snapshot manager. But it is still there!! I have tried vmware-cmd "vmx-file" removesnapshots , and it tells me that snapshots is removed. hassnapshots also say no.

I have tried to add a snapshot and remove it from the GUI (one that had no snapshot) and this works.

Ideas??

Reply
0 Kudos
1 Solution

Accepted Solutions
peppert
Enthusiast
Enthusiast
Jump to solution

after some digging, in my instance the base .vmdk file for each VM and all of the deltas except for the newest include the following line in their .vmdk file:

ddb.deletable = "false"

I was able to remove these snapshots and restore performance by doing the following:

1) power down the VM, back up all of the files someplace safe

2) in esx console, cd to the vmfs path of the affected VM

3) use nano to set ddb.deletable = "false" -> ddb.deletable = "true" in the root .vmdk (usually .vmdk). Do the same for any other base disk .vmdk files you have if you have more than one virtual hard drive.

4) run the following shell script to comment out the 'ddb.deletable = "false"' lines in all of the snapshot delta vmdk files. Answer y for each file to overwrite if prompted:

for y in *-000???.vmdk; do sed 's/ddb.deletable = "false"/#ddb.deletable = "false"/g' $y > temp; mv temp $y; done

5) create a snapshot (either in virtualcenter client, or via vmware-cmd /vmfs/full/path/vmname.vmx createsnapshot foo foo 0 0 in esx console)

6) remove all snapshots (either in virtualcenter client, or via vmware-cmd /vmfs/full/path/vmname.vmx removesnapshots)

7) power the vm back on, make sure all is well.

I am uncertain if upgrading from 3.5->4 inserted the initial ddb.deletable parameter which then propogated to all deltas, or what happened exactly. I have also not tried (and at the moment can't recommend) modifying vmdk files and snapshots without first powering off the VM.

I'm leery of testing VMware Data Recovery again, but will do so once I have modified all of the VMs we run and report if the problem has been solved permanently. I have 2 clusters that are exhibiting this behavior and one which is not, so in theory the product is capable of working Smiley Happy

Good luck out there.

View solution in original post

Reply
0 Kudos
20 Replies
Steve_Tron
Enthusiast
Enthusiast
Jump to solution

Hi,

Given that you have added and removed additional snapshots from the affected VMs and that they don't show up in snapshot manager they could be just orphaned. Have you tried browsing the datastore and deleting the snapshot files after first verifying that the vm's in question are not using the snapshot files as their disks i.e. right click vm, settings etc. and are using the original vmdk files.

Reply
0 Kudos
asp24
Enthusiast
Enthusiast
Jump to solution

The files are in use (confirmed). I will try to storage migrate / clone and see if that helps.

Reply
0 Kudos
asp24
Enthusiast
Enthusiast
Jump to solution

when trying to storage migrate I get an "the virtual machine has virtual disk in linked-cloned mode that prevents migration". But there are no such usage here?? I have not found anythink in the VM directory, nor in the vmx file that points to that..

Reply
0 Kudos
Steve_Tron
Enthusiast
Enthusiast
Jump to solution

Hi,

Can you try this

Execute vmware-cmd /vmfs/volumes/<datastorename>/<vmname>/<vmname>.vmx removesnapshots on the Service Console.

Regards

Reply
0 Kudos
asp24
Enthusiast
Enthusiast
Jump to solution

I did try that. the result is output "removesnapshots() = 1" , but the VM still uses the latest delta file (main vmdk date/time not changing)

Reply
0 Kudos
Steve_Tron
Enthusiast
Enthusiast
Jump to solution

Hi,

When you create a new snapshot on a affected VM does the the VM use this file if so what happens when after taking the snapshot you issues the removesnapshot command again from the console.

Regards

Reply
0 Kudos
asp24
Enthusiast
Enthusiast
Jump to solution

I created a new snapshot, and the delta "extension" went from 000009.vmdk to 000010.vmdk. I ran the removesnapshots command, and the snapshot is gone from snapshot manager. But it is not gone from the VM (and this one is the new used vmdk for the VM)

VERY strange!! I think I have to try to remove the snapshot using a 3.5 host and see if that works.

Reply
0 Kudos
Steve_Tron
Enthusiast
Enthusiast
Jump to solution

Hi,

So after you removed the snapshot you made using the manager the VM reverted back to the snapshot it was using before?

Regards

Reply
0 Kudos
Steve_Tron
Enthusiast
Enthusiast
Jump to solution

Hi,

One trick to try is remove the vm from virtual centre then re-add it see if it picks up the snapshot in snapshot manager

Regards

Reply
0 Kudos
asp24
Enthusiast
Enthusiast
Jump to solution

It did not pick it up (added to an OTHER virtualcenter that have access to the same san)

I think the linked-clone error message when trying to storage-migrate is the key to finding out what is happening..

Reply
0 Kudos
Steve_Tron
Enthusiast
Enthusiast
Jump to solution

Hi,

What was the linked clone error?

Reply
0 Kudos
asp24
Enthusiast
Enthusiast
Jump to solution

when trying to storage migrate I get an "the virtual machine has virtual disk in linked-cloned mode that prevents migration"

I have not configured any linked clones.

Vmware Data Recovery might have created one, but I find no pointers to any.

Reply
0 Kudos
jasoncllsystems
Enthusiast
Enthusiast
Jump to solution

complicated one - http://malaysiavm.com/blog/manual-commit-snapshots-delta-file-to-vmdk-flat-file/

Regards,

CLL SYSTEMS http://www.cllsystems.com

MALAYSIA VMWARE COMMUNITIES

http://www.malaysiavm.com

http://www.malaysiavm.com
Reply
0 Kudos
asp24
Enthusiast
Enthusiast
Jump to solution

Very strange!

VCenter reports provisioned storage as 132GB and not-shared and used storage as 69,35 GB (thick provisioned files)

I have tried changing the CID's in the VMDK files, created a new VM and attaching the files etc. It looks like the removesnapsshots utility now checks for Vcenter / esx snapshot "findings" ??

Reply
0 Kudos
peppert
Enthusiast
Enthusiast
Jump to solution

after some digging, in my instance the base .vmdk file for each VM and all of the deltas except for the newest include the following line in their .vmdk file:

ddb.deletable = "false"

I was able to remove these snapshots and restore performance by doing the following:

1) power down the VM, back up all of the files someplace safe

2) in esx console, cd to the vmfs path of the affected VM

3) use nano to set ddb.deletable = "false" -> ddb.deletable = "true" in the root .vmdk (usually .vmdk). Do the same for any other base disk .vmdk files you have if you have more than one virtual hard drive.

4) run the following shell script to comment out the 'ddb.deletable = "false"' lines in all of the snapshot delta vmdk files. Answer y for each file to overwrite if prompted:

for y in *-000???.vmdk; do sed 's/ddb.deletable = "false"/#ddb.deletable = "false"/g' $y > temp; mv temp $y; done

5) create a snapshot (either in virtualcenter client, or via vmware-cmd /vmfs/full/path/vmname.vmx createsnapshot foo foo 0 0 in esx console)

6) remove all snapshots (either in virtualcenter client, or via vmware-cmd /vmfs/full/path/vmname.vmx removesnapshots)

7) power the vm back on, make sure all is well.

I am uncertain if upgrading from 3.5->4 inserted the initial ddb.deletable parameter which then propogated to all deltas, or what happened exactly. I have also not tried (and at the moment can't recommend) modifying vmdk files and snapshots without first powering off the VM.

I'm leery of testing VMware Data Recovery again, but will do so once I have modified all of the VMs we run and report if the problem has been solved permanently. I have 2 clusters that are exhibiting this behavior and one which is not, so in theory the product is capable of working Smiley Happy

Good luck out there.

Reply
0 Kudos
peppert
Enthusiast
Enthusiast
Jump to solution

just confirming that this works on powered-on VMs as well via service console commands from the host the VM is registered to - still strongly recommend backups before you go this route.

i have a couple of VMs with 'ddb.deletable = "true"' set in their base .vmdk which do not exhibit the weird snapshot behavior. if i had to guess, i'd say that the delta .vmdk files inherit the ddb.deletable flag. no idea on what initially set them to ddb.deletable=false initially, however.

Reply
0 Kudos
peppert
Enthusiast
Enthusiast
Jump to solution

Two final updates on this issue:

If you are averse to fixing this via console (or don't run esx) you can rectify by powering each VM off and cloning it via virtualcenter server. The new clone will have single, merged vmdks.

After fixing them all, I let VMWare Data Recovery take a stab at another backup. It changed ddb.deletable in the root vmdk from true to false when it created the snapshot, backed up successfully, attempted to delete the snapshot, claimed that the deletion was successful. I checked the directory and lo and behold the same behavior is there - delta file still present after "successful" backup.

Not sure what's specifically broken in my implementation. I have 2 2-host clusters with VMDR broken, both with NFS shared storage (NetApp FAS3020c in prod and an infrant readynas pro on the test cluster), and a third cluster where VMDR is working on another netapp fas3020c via NFS. I can't discern any difference between them. Guess I'm going back to NetApp snapshots and rsync for offsite backups. I'll miss dedup.

Problem is occuring in the 4.0.0 release of VMDR (VMware-VSZ-DR-4.0.0-164675-R1.0G68-2.iso)

Reply
0 Kudos
kenthansen
Contributor
Contributor
Jump to solution

Hi,

I can confirm the problem with VMDR exists on our system too - we're running on 2 NetApp filers (FAS920 and FAS270) and one OpenFiler server. We're using NFS on all the boxes.

Your procedure to remove the snapshots work like a charm! (even while the VM are running - I've removed 10 by now). Thanks!

Guess it's bye-bye to VMDR until next release/patch... 😕

Regards, Kent

Reply
0 Kudos
peppert
Enthusiast
Enthusiast
Jump to solution

Just confirming this is resolved for me in DR 1.0.1 release. They have a handy python script in the knowledge base that fixes the snapshot descriptions, though you still have to manually create and delete all snapshots to get esx/esxi to purge them.

Reply
0 Kudos