ESXi 4 - How to get rid of snapshots manually?

[RaWie]

Hi All,

on an ESXi 4 I've been run out of disk space. Reason was that there have been several snapshots for one virtual machine. After moving other virtual machine into another datastore the snapshots have been commited by use of snapshot manager.

The snapshots disappeared from the snapshot manager's hierarchy as expected.

So far, so good.

But, the snapshot files did not disappear from the datastore. And: The files are still needed to start the VM as they are somehow concatenated according to configuration file and startup-log. In the first vmdk-file's config-data it seems to contain some info regarding the disk geometry etc. in it?!

In addition, the size of the 1st disk could not become changed by the vSphere-Client settings-dialogue as it is for the 2nd disk.

How proceed to combine the snapshots or what to do to get rid of unused snapshots and delete files? At the end, I like to have only one vmdk-file for the 1st disk. (Without any reference to old but never used snapshots.)

Any hint and advice is very appreciated.

Thanks and best regards

Ralf

[henshawp]

Hi Ralf

I had to remove a number of snapshots from a VM and this was my experience..

Look at my post here:

http://phenshaw.com/

In short, it is best to remove using CLI.. but before you remove you will want to ensure the following:

- VMTools is upto date on the VM

- The snapshot chain is consistent

- If space is a concern, you can remove each snap one by one to save space.

Have a read of my article and let me know what you think. (This was on ESX 4) but I sure you can use Remote CLI to do the same.

Thanks

Paul

[mittim12]

In my experience large snapshots can sometimes take awhile to complete.  Are you sure it has finished up?     You can always create a test snapshot and go through snapshot manager and delete all.  That should clean everything up.  

[a_p_]

What is the exact version/build of ESXi you are running. I'm asking this, because there were major changes regarding snapshot deletion in ESX(i) 4.0 Update 2. If the hints of the other posters don't work, please post a list of files (either a screen shot of the datastore browser window, showing all files and details or the output of ls -alis from the command line) and attach the latest vmware.log file.

André

PS: Discussion moved from VI: VMware ESXi™ 3.5 to VMware ESXi™ 4

[RaWie]

Hi André,

just a quick response regarding the ESXi version. The FujitsuPrimergy Server is running ESXi 4.0.0, 398348. Not sure, if this is the Update version you mentioned.

br

Ralf

[a_p_]

Build 398348 belongs to ESX 4.0 Update 3, so that's ok.

André

[J1mbo]

Might be worth putting in something to generate alerts on snapshots - I have a health check script with this functionality on my blog.

[RaWie]

Hello Paul,

thx for your reply. The snapshots look concatented correctly and the chain of snapshots seems not broken.

But, the problem seems missing diskspace!

Pls see the following picture to have an actual view into the directory:

The free space on disk is only 19 GB!

Following your docs this is insufficiant to take a new snapshot and run the snapshot manager to remove snapshots.

All actual data is in the m01sbsdc020-000004-delta.vmdk, the disk information is in vm01sbsdc020.vmdk

What are the old snapshots needed for?

Is there a way to make the latest delta-file the actual flat disk manually?

What happens during snapshot managers "commit all" with the data. I guess, the actual contend of the latest virtual disk is not touched?

Thx for comments and br

Ralf

[DSTAVERT]

Considering the size of the snapshots this has been running for some time.

You are in good hands so I won't become another cook. Don't do anything on your own. Listen to what André has to say.

In the mean time please read up on Snapshots.

Best Practice http://kb.vmware.com/kb/1025279
Understanding http://kb.vmware.com/kb/1015180

[a_p_]

Considering the ESX version/build you are running and looking at the files, it should be no big issue to commit/delete the snapshots. Just one important questions remains: Is the virtual disk with the snapshots thin or thick provisioned? If thin, then stop here and report back!

If it a thick provisioned disk (should be 70GB in this case) the additional disk space needed while the delete job is running depends on the guests disk activity. To be safe you could delete the snapshots with the VM powered off. Since the snapshot manager does not show any snapshots, you need to create a new snapshot and then click the "Delete All" button. Due to the size of the snapshots, the process may take several hours. Make sure there is no other application running (e.g. backup) which will try to create another snapshot during this time.

If the GUI times out, don't panic. The process will continue to run on the ESXi host. Once finished, the delta files should be gone.

Regarding your questions:

What are the old snapshots needed for?

Usually you take a snapshot if you are going to install a new application and want to have the option to revert to the previous state if the installation fails. Another usual use case for snapshots are backup application, which backup the VM image based.

Is there a way to make the latest delta-file the actual flat disk manually?

No, The disks are used as a chain. The snapshots only contain deltas (modified data blocks) and are not useful without their parent disks.

What  happens during snapshot managers "commit all" with the data. I guess,  the actual contend of the latest virtual disk is not touched?

Correct. All the modified data blocks from all snapshots are committed to the base disk. Starting with ESXi 4.0 Update 2, VMware modified the way how the delta files are merged to prevent from running into "out of disk space" issues. That's why I asked for the build in one of my previous posts.

André

[RaWie]

Hello André et al,

thx so far. Understanding your valuable comments I now know that I misunderstood the ESXi snapshots. Thought, they are similar to the ones from vmware server. But they are not as they function totally different.

OK - actual status:

The guests disk in question is appr. 75 GB whereas the latest delta now is appr. 70 GB. (It is eating up my hosts disk space)

The host actually left only 19 GB free space for the datastore1 where the VM resides.

What I learned as well: Need to download and install the vSphere CLI for Windows to be able to run some vmware-cmd commands.

Understanding you correctly, I should:

1st: stop the VM

2nd: take a new snapshot - wich may take less diskspace as the system is switched of and has no changes to keep track on?!

3rd: Commit all snapshots - even if there is only the last one in the list

4th: Let the system run for a while as it may take hours to put everything back into the initial .vmdk-file

Is that ok to proceed or do you recommend something different?

Thanks a lot and

br

Ralf

[a_p_]

The steps are correct. However you did not answer the question, whether it is a thin or thick provisioned virtual disk. With a thin provisioned disk this could end up in a disaster! Either take a look at the VM's HDD properties and check the configured disk size and whether it shows thin or thick or - even better - attach the header vmdk files (the small vmdk files) and post a screen shot of the datastore browser window showing all files and details.

André

[RaWie]

Sorry, André,

have not been aware of the small note in the configuration window and thought thin or thick was related to disk size ...

Anyhow, I am a lucky man. As you can see from screenshot, it is a "thick" provisioned disk, even if settings are greyed out:

The datastore of the VM looks like following screenshots wich represents very actual data:

Actual figures of the VM can also be taken from this SCP-Screenshot:

Please find attached four .vmdk-files wich will build the chain of these snapshots.

If I do not misunderstand the function of the "commit all" command, all changes stored inside the delta-files are consolidated and at the end, I get a resulting vm01sbsdc020.vmdk-disk-file representing the actual content of the VM. All other disk-file could than be deleted and changes are commiteted when the VM shuts down latest?

Thanks for your comments.

br

Ralf

[a_p_ · Accepted solution]

Something strange is going on here. Although the disk is shown as "Thick" provisioned in the VM's properties, the vmdk file shows ddb.thinProvisioned = "1" which would explain the size of ~26GB, shown in the datastore browser!? In this case you would not be able to commit the snapshots, because you only have 19GB free disk space and you would temporarily need at least ~45-50GB.

Could you do another check please and run an ls -lisa on the command line to verify the size of the vm01sbsdc020-flat.vmdk.

Do you have another VMFS datastore with at least 70GB (better more) free disk space? In this case we could go the safe way and clone the virtual disk without touching the original vmdk files.

André

PS: Added "... because you only have 19GB free disk space and you would temporarily need at least ~45-50GB." to clarify why the snapshot cannot be deleted with thin provisioned base disk.

[RaWie]

Hi André,

sorry for the late response. It took a while, And lot has happened.

Encouraged through your comments but unfortunatelly prior to your last post I have been pusged by users and management to run a snapshot and delete it. Just made it twice as the first did not remove the files. The first run only increased the original disk up to 72 GB.

At the end, system won't start anymore, as there is was no more free space in datastore.

Moved all stuff into a NAS without success, VM won't start.

The snapshots I took was taken from a VM switched off. My thought was that nothing really changed beteween the old situation and after the two snapshots.

I decided to change the entries back to the values prior to the last snapshots in the .vmx-file and give it a try.

Luckily, the VM starts up and is running so far without problems. Performance is poor, but that is about running a VM with datastore inside a NAS, isn't it?

But now I have app. 200 GB free space. And that is much more than all vmdk-files together.

hopefully, it helps.

Try now fiddelling out the CLI-command to give yoiu the result from the lately requested command.

br

Ralf

[RaWie]

Hi, André,

just to let you know what happened so far.

After moving the VM into a NAS with more free space, I could run the "snapshot/Commit all snapshots" commands.

As I said earlier, system won't start. (vmx-file still points to the snapshot-files.)

To make it starting, I changed vmdk-settings back to the ones before I ran the snapshots. (My thoughts here: As I took snapshot with system down, nothing changed the content of the disks/vmdk-files!)

OK! System started up without problems.

But something strange happend as now the second disk became thin provisioned?

I stopped the system! Thereafter I changed the provision-flag in vmx-file manually from "1" to "0" to let the HDD become thick provisioned again.

I started the system again and it worked!

Taking and committing snapshot went fine except that it did not delete the snapshot-files.

I stopped the VM again and checked/changed the vmx file to let it point to the initial flat-files for the HDDs.

Additionally, I moved the bunch of snapshot-files into another directory by use of WinSCP.

At next start my VM runs without problems. To verify the mechanism, I took again a snapshot and after that, commiting all would remove the snapshot from inside snapshot manager's hirarchy as well as the taken snapshot files themselves.

In properties dialogue, both HDD are now thick provisioned.

I won't say that changing settings is a common solution, but If you have a clone and free diskspace in datastore, it might be worth a try?

Again, thanks a lot and

br

Ralf