1.5TB in Datastore but only 850GB Provisioned, Need to increase that by 100GB

[TDSSupport]

I am sure this has been asked and I am sure there are answers and KB's on this but for whatever reason my search terms are not returing specific enough instructions so sorry up front if you think I should have found the answer on my own.

Anyway ESXi 4, 1.35TB in datastore1, 868GB provisioned - 788GB Used, 475GB Free. I have a server with Snapshots that I need to clear but I am afraid there is not enough room to do this live Delete. I would do a Delete with the Guest turned off but it is a production box and with no idea how long the merge/delete would take that just isn't an option. The Guest VM is around 150GB Used with 3 Disk setup, 100, 200, 200 so my thinking was the simple fix would be to use some of that 475GB Free and slide it over into the Provisioned space, say 150GB but I give, how the heck do I do this?

I want to stay within the VSphere client or use a limited amout of linux command line tools and want to do this live if possible, one would think you can.

Another option would be to shrink the hard drives assigned to the VM as the 200GB has barely 20GB being used, the 100GB about 40GB used. Leaving the OS boot alone if I shrank those to drives while doing the Snapshot Delete would that help and be an easier path?

The Snapshots are OLD so let's assume they will require the full amout of space possible to merge. One of them was from when the OS was loaded some two years ago, long before the databases had any data or who knows what else (this isn't my server, I took it over).

[a_p_]

You explained a lot,but the important part is missing. Are the virtual disks Thin or Thick provisioned? In case of thick provisioning there's nothing to be afraid of and you should be fine running "Delete All" from the Snapshot Manager. In case of thin provisioned disks, please post a screen shot of all the VM's files in the Datastore Browser, showing full names, extensions sizes, time stamps.

André

[TDSSupport]

Thick. This is ESXi 4.0.0 Build 208167 so I don't think i want to do to a Delete all from what I read. Updating ESXi will be on the list once I have these drives shrunk and I am getting consistant backups both in the Guest OS and the VM itself using Veeam.

[a_p_]

You are right, "Delete All" with ESXi before Update 2 may cause disk space issues with multiple snapshots. I misread "Anyway ESXi 4, 1.35TB" as ESXi 4.1, sorry about this. Do all the snapshots show up in the Snapshot Manager? If yes, you could delete the snapshots 1-by-1, always selecting the snapshot closest to the base disk. This would merge the snapshot data into the base disk without requiring additional disk space except for the last snapshot in the chain.

I only see snapshots for the first (200GB) virtual disk. Are the other disks set to independent!?

André

[TDSSupport]

ok that makes sense but let me make sure my thinking is inline with yours.

When you make a Snapshop you are actually running the Snapshot. As such if I have a base image and then three snapshots the base plus 2 are static and merging them through the Delete Snapsot won't increase disk space, it would reduce it as it merges Snapshot 1 into the Base disk. Snapshot 2 will result in the same as it merges those changes. Lastly you have the active Snapshot which at this point should have more than enough room since the prior two snapshots have been merged and deleted.

Also the performance hit on any "deleting" should not be that great since the merge of the first two snapshots do not include the active snapshot, I/O with slight CPU overhead expected but guest VM impact should be marginal at best, yes?

[a_p_]

Please take a look at http://kb.vmware.com/kb/1015180 which explains the snapshots in VMware products. Snapshots are used in chains where only the last chain link is used for writes and the other ones are only accessed in read-only mode. Each snapshot only contains changed data blocks.

With a snapshot chain like "Base -> Snap1 -> Snap2 -> Snap3" writes go to Snap3 only. When you delete any of the snapshots the data is merged into its direct parent. With the Base disk being thick provisioned you will not need additional disk space if you first delete Snap1 and then Snap2 since the base disk is already full provisioned and will not increase in size anymore. Deleting the last snapshot in the chain will require additional disk space (with the VM powered on only) because VMware temporarily creates a "Consolidate-Helper" snapshot. However, as you already mentioned, with every snapshot you delete (i.e. merge into the base disk) the snapshot/delta file will be deleted.

The issue with ESX 4.0 before Update 2 was that the "Delete All" process merged the files in a reverse order (i.e. Snap3 into Snap2, then Snap2 into Snap1 and finally Snap1 into the base disk) which increased each delta file and the delta files were only delete after the final merge into the base disk. This often cause out-of-diskspace issues in the past.

If you want, please post (attach) the VM's .vmsd file so I can take a look at it to see whether it contains the proper snapshot chain.

André

[TDSSupport]

Well first delete went fine and the space change was marginal. The next Snapshot Delete, also from years ago, completed and datastore size dropped a reasonable amount but now I have a Consolidate Helper-0 which the KB's say happens where there was not enough space which doesn't seem right. I now have just the Snapshot I took of the server when I took over the site which "was" the Snapshot being used.

Root (Base)

-----My Snapshot

-----------Consolidate Helper - 0

----------------------You are Here

Edit: Backup Exec 2010 is running the Guest OS and I did not shut it down prior to "Deleting" so it fired off at 5:45PM at which time the Snapshot Delete was probably in the last bit of processing as it was at 95% at 5:00. So at this point if you agree that BE was the likley problem here then I will process the next Snapshot on Thursday during business hours so if there is a problem I am not locked out of the building. That would leave just the Consolidate Helper which should delete pretty easily/quickly. I'll also pause and BE jobs this time just in case, my error in not catching that.

[a_p_]

I'm a little bit unsure about the Consolidate-Helper and the sizes of both remaining snapshots!? Which .vmdk file is currently the one mentioned in the VM's .vmx file (000003.vmdk or 000005.vmdk) and how much free disk space do you have now?

If you want, post the VM's .vmsd file (or drop me a PM with its contents if you don't want to post it publicly) so I can verify whether this represents the correct snapshot chain.

André

[a_p_ · Accepted solution]

The .vmsd file you PM'd me is partly corrupt, i.e. it has multiple entries with the same uid, pointing to the same parent uid but different parent disk names!? But anyway, if you can confirm that the current virtual disk's name in the .vmx file is "...-000005.vmdk", it should work to delete/merge the first (your) snapshot from the Snapshot Manager and once only the one last snapshot is there, then use the "Delete All" button to delete/merge it. This will ignore the .vmsd entries and should also clean up the file.

André

[TDSSupport]

Not sure how to find the active disk since in the properties of the running VM it only shows part of the file path in the window and can't be edited while it's running. I can shut it down but have to schedule that and to be honest I was hoping to get all these merged and then to do a Veeam backup before rebooting the guest VM just in case.

EDIT: Maybe I should hit up a Veeam Backup now anyway?

BTW In looking deeper at the Datastore I see now why so much space is being used. There are 4 other machine folders listed of which only one of those is a VM which ever gets mounted and used. One of those Folders in the Datastore is for an old server that was Decom'd way back in 2010 and I guess the prior techs VM'd it during the conversion to the new server as it has multiple Snapshots and is taking up about 250GB of the Datastore. It has no function and hasn't ben ran since 7/2010 so at some point I probably need to import it, merge/Delete the Snapshots and then pull that data out of the satastore and onto a USB drive or two. Almost like prior techs forgot about it and never finished cleaning up their work area.

In any case this help explains why the Datastore was reporting far more use than I was accounting for. Although I saw that folder I didn't even consider that it could have 250GB in it because I knew how big that physical server was when it was running years ago and at best I expected it to be using 80GB. This site was my site when I worked for another shop, they left that shop and started using a Managed Services Provider but once they found out I wasn't working for the guy I use to work for they called me and asked me to take it back over so that's how I know some of the history.

[a_p_]

You can either download and check the VM's .vmx file or - much easier - install RVTools (http://www.robware.net/) which gives you a great overview. You can find the active .vmdk file for each VM in the "vDisk" tab and the tool will also show you any zomies (VM's which are not in the inventory) in it's "vHealth" tab.

André

[TDSSupport]

Nice tool. Sent a link to a screen cap of the drives/disk. Under vHealth I only show what I expected for the old decom'd server, 4-5 Possible Zombie files but that's not related and not an issue. For the VM in question it shows just two CD's connected and my Snapshot I took, not the Consilidate one, as active.

[a_p_]

Ok with the information you provided the current snapshot chain looks like this:

<vmname>.vmdk - <vmname>-000003.vmdk - <vmname>-000005.vmdk

I'm confident that deleting the snapshots as mentioned before will work. So go ahead and "Delete" your own snapshot, this will merge <vmname>-000003.vmdk into the base disk. Once done run "Delete All" with the last remaining snapshot <vmname>-00005.vmdk.

André

[TDSSupport]

Outside the topic but I thought I would report that my attempt at doing a VeeamZip backup is not showing promise. After 26 hours Veeam shows it's at 0% and processing, VSphere shows "Create Virtual Machine Snapshot In Progress". I don't see anything in the Datastore which would seem to say it is in fact creating such a snapshot and the output folder for the backup hasn't changed the 8k initial file it created since it started.

[TDSSupport]

Delete one by one then a Delete All seems to have done the trick in cleaning out the snapshot images without increasing datastore use during the process. Process was SLOW thus why it's taken days to respond. Made sure I disabled backups during the Delete then would go back in and force a backup within the VM before doing the next Snapshot Delete. The bar graph of percent complete is more or less useless as it would take 30 minutes to get to 95% then sit at 95% for hours and hours.

I would like you to check over the config file one more time to make sure it's no longer corrupt or contains orphaned entries.

Thanks for the help.