universalknr
Contributor
Contributor

Recommended way to remove a VM Snapshot?

Jump to solution

I just recently tried to delete a snapshot of a VM via the VM snapshot manager.  The VM was 80GB in size with a snapshot of 50GB existing, I gracefully shutdown the OS/VM prior to removing the snapshot.  Once I started the removal, all was well until we got to 95% (the 95/99% issue we all know).  After doing some research, I finally decided to cancel the process and restart the management services via the host.  This resolved the issue and the snapshot appeared to be removed.

A couple days later I was working on another task and ended up experiencing the "An internal error occurred...Details:  Object reference not set..." error, this took quite some time to fix but eventually was resolved.  This showed me that the snapshot I thought removed previously, did not actually get removed;  i have now lost trust in the VMware snapshot manager!

As I am going to have to go through the process of removing snapshots from numerous VM's, over the next couple of weeks.

I want to avoid having to experience this type of related issue again, I want to ensure that the VM snapshot is completed removed and as quick as possible.

Is there any preferred way of going through the snapshot removal process?  Maybe via the CLI or a VMware or Third-Party utility/tool?

Is there a "verification" process that can be taken to ensure that the snapshot was actually removed?  It appears that I cannot trust the GUI/vSphere Client when it comes to verifying the snapshot removal. 

0 Kudos
1 Solution

Accepted Solutions
proden20
Hot Shot
Hot Shot

Consider the snapshot manager to be a reliable reference point in environments with fast, healthy storage infrastructure and snapshots that don't grow very large (50GB is very very large.)

Like a.p. said, VMware's snapshot handling is super-reliable, so don't panic when snapshot manager isn't displaying what you'd expect.  ESXi is very good at keeping your data intact via snapshot redo logs.

To answer your questions:

"Is there any preferred way of going through the snapshot removal process?  Maybe via the CLI or a VMware or Third-Party utility/tool?"

- The first step is to prevent snapshots from growing to very large sizes, and keeping ample free space on your datastores for operational overhead.  As far as the preferred method, continue using Snapshot Manager.  If you suspect the VM is still running from a snapshot, you will see those files in the datastore browser.  In the past, we needed to use the ESXi CLI to safely merge and clean up snapshot trees.  VMware introduced a GUI version of this method in 5.x.  It is called "Consolidation."  See this excellent KB article:

VMware KB: Consolidating snapshots in vSphere 5.x

"Is there a "verification" process that can be taken to ensure that the snapshot was actually removed?  It appears that I cannot trust the GUI/vSphere Client when it comes to verifying the snapshot removal. "

Snapshot Manager and the "Needs Consolidation" field are your first indication.  Gaining an understanding of the snapshot files stored with the VM will give you a greater comfort level with the internals.  I recommend this article:

VMware KB: Understanding virtual machine snapshots in VMware ESXi and ESX

View solution in original post

0 Kudos
12 Replies
a_p_
Leadership
Leadership

>>> ... i have now lost trust in the VMware snapshot manager!

I think the issue was that you were not patient enough. For a 50GB snapshot it might take some time to merge all the delta data from the snapshot into the parent disk. Snapshot removal is actually very reliable, at least if you e.g. don't run out of disk space in case of thin provisioned virtual base disks.

André

proden20
Hot Shot
Hot Shot

Consider the snapshot manager to be a reliable reference point in environments with fast, healthy storage infrastructure and snapshots that don't grow very large (50GB is very very large.)

Like a.p. said, VMware's snapshot handling is super-reliable, so don't panic when snapshot manager isn't displaying what you'd expect.  ESXi is very good at keeping your data intact via snapshot redo logs.

To answer your questions:

"Is there any preferred way of going through the snapshot removal process?  Maybe via the CLI or a VMware or Third-Party utility/tool?"

- The first step is to prevent snapshots from growing to very large sizes, and keeping ample free space on your datastores for operational overhead.  As far as the preferred method, continue using Snapshot Manager.  If you suspect the VM is still running from a snapshot, you will see those files in the datastore browser.  In the past, we needed to use the ESXi CLI to safely merge and clean up snapshot trees.  VMware introduced a GUI version of this method in 5.x.  It is called "Consolidation."  See this excellent KB article:

VMware KB: Consolidating snapshots in vSphere 5.x

"Is there a "verification" process that can be taken to ensure that the snapshot was actually removed?  It appears that I cannot trust the GUI/vSphere Client when it comes to verifying the snapshot removal. "

Snapshot Manager and the "Needs Consolidation" field are your first indication.  Gaining an understanding of the snapshot files stored with the VM will give you a greater comfort level with the internals.  I recommend this article:

VMware KB: Understanding virtual machine snapshots in VMware ESXi and ESX

View solution in original post

0 Kudos
admin
Immortal
Immortal

check this link ..i hope it will help you..must watch.. http://www.youtube.com/watch?v=A8KOh4CzmQc

0 Kudos
universalknr
Contributor
Contributor

@proden20 - Thanks for the thorough response.  I am going to go ahead and check out those 2 articles that you provided.  In your reply, you actually touched based on exactly what I am seeing - "if you suspect the VM is still running from a snapshot, you will see those files in the datastore browser."  At the current moment, the VM is still running from the snapshot, but it is not showing that in the snapshot manager GUI.

I have not yet read your article, but will next.  However, this is for an ESXi Host running 4.1 - is the method still the same?

When you say the "Needs Consolidation" field, what are you referring to exactly?

0 Kudos
universalknr
Contributor
Contributor

Thanks for this!  I watched it and it did help with my understanding of the back-end portion of the snapshot.

0 Kudos
universalknr
Contributor
Contributor

I see you guys give hints that 50GB snapshot is considered large, with that being said - what is considered a limit for how large you generally let your snapshots get to (before removing/committing)?

0 Kudos
a_p_
Leadership
Leadership

Basically a snapshot should be used for e.g. critical system changes/updates to be able to revert in case something goes wrong, but never as a backup. That means a snapshot should only exist as long as really needed, because it will not only consume disk space but also have impact on the VM's performance. The size of the snapshot can grow and it doesn't matter in regards of deleting a snapshot, it will just take a longer time (this highly depends on the storage you use).

The snapshot information you see in the snapshot manager is what's stored in the VM's .vmsd file and if this is incorrectly updated, corrupted or whatever, the Snapshot Manager may not show any snapshots anymore. A great tool to monitor snapshots and other things is RVTools (RVTools - Home) which provides an excellent overview of the environment.

André

universalknr
Contributor
Contributor

Once again, thanks for your awesome and prompt reply!

How does the snapshot actually impact the VM's performance (if you don't mind me asking)?

I actually have RVTools, I just never put time into utilizing it.  I will have to give it a go!

Thanks to everyone!

0 Kudos
proden20
Hot Shot
Hot Shot

OK...

In ESXi 4.1 you'll need to perform consolidation manually.  You may have luck with getting an automatic consolidation out of a clone or Storage vMotion operation to a different datastore, but this functionality was intermittent between versions, I don't remember specifics.  Simple enough to try though.  "Consolidation" refers to asking the hypervisor to merge your snapshots and clean up the files when the Snapshot Manager isn't showing you snapshots that you expect to see.  It works great, VMWare has gone to great lengths to ensure your data stays intact after consolidation (I always consolidate with the VM powered-down.)  As far as snapshot size goes, it personally try not to let them grow above 5GB and not let them exist longer than 24 hours.

VMware KB: Consolidating snapshots in ESX/ESXi 3.x and 4.x

0 Kudos
universalknr
Contributor
Contributor

Thanks.

With all this being said, is there a way to maybe setup some Alerting or Automatic Deletion/Clean-up for a VM snapshot?  This way it's automated or at least, I can receive an alert that states something along the lines of "VM is 98% close to meeting you specified threshold".

0 Kudos
proden20
Hot Shot
Hot Shot

As far as alerting goes, you sure can (see below.)  I don't have a 4.1 vCenter around right now, but take a look at scheduled tasks.  There may be a snapshot removal task available.  If not, dig around on the interwebz, I'm sure someone is doing it with PowerCLI.

VMware KB: Configuring VMware vCenter Server to send alarms when virtual machines are running from s...

0 Kudos
a_p_
Leadership
Leadership

The performance impact comes from the fragmented data in the snapshot/delta files. Snapshot .vmdk files basically consist of metadata (the header, a grain directoriy and grain tables) and the user data (grains). Each of these grains contain 512 bytes, and the grains are appended to the delta file as data is modified. You may want to take a look at http://www.vmware.com/support/developer/vddk/vmdk_50_technote.pdf (“ESXi Host Sparse Extents") if you are interested in how the delta files look like.

André

0 Kudos