Snapshot storage management questions

[MikeAW2010]

I'm fairly new to ESXI and I'm trying to grasp how snapshots work in this environment.

From what I understand so far, it seems like they work almost like GIT repositories, where a master snapshot is created first, and then a concurrent snapshot is dependent on the master snapshot and the concurrent snapshots (child snapshots) are dependent on each snapshot taken before it making kind of a chain of dependencies. It appears that when ESXI boots a VM, it actually uses the most recent snapshot along with the vdisk and not just the vdisk, meaning the most recent snapshot must be fully intact.

Here is my question. We like to take a snapshot of a certain VM once a week but over time we do not want it to fill up and over encumber our storage capacity. My first thought was to delete any snapshot older than 2 months, but upon doing this I was no longer able to boot into the VM. I did manage to repair it by loading the original vdisk image which was months out of date, and rebuilt everything from there, however; what is the best way to manage snapshots if you seemingly cannot delete the oldest snapshot without affecting the current snapshot?

Thanks.

[a_p_]

Snapshot in VMware products work as chains, i.e. each chain-link is required, and only the last one in the chain is the with write access.

Manually deleting one of the snapshots in the chain will break the VM, so do not do that.

Deleting a snapshot using the Snapshot Manager will merge this snapshot's data into its direct parent.

Regarding keeping snapshots for a long time. It's not recommended to maintain VMware snapshots on production VMs.

What may be an option - if you want to keep different states - is to see whether your storage system offers snapshot functionality.

André

[MikeAW2010]

Thank you for the above info. So that pretty much confirms my suspicions.

We are using a PowerCLI script in order to automatically take and delete the snapshots. Would PowerCLI also have the same effect as snapshot manager in the respect of merging a snapshots data into its direct parent? Also, what would happen if the 'oldest' snapshots were deleted first rather than the newest? For example:

Snapshot Main > Snapshot A > Snapshot B > Snapshot C > Snapshot E > Snapshot F > Snapshot Current

In Scenario A: - Lets say that the Main, A and B snapshots were more than 2 months old and deleted by a PowerCLI script. What would happen to Snapshot's E thru the Current Snapshot? Would they be merged properly and would the machine still be able to boot properly?

In Scenario B: - Lets say that the Main snapshot were PRESERVED, and snapshots A and B were deleted, would C thru the Current be merged properly?

----

As for storing the snapshots, how do I tell ESXI where I want to store them if I don't keep them in a production VM?

[a_p_]

Let me just modify your example slightly:

Base VM > Snapshot A > Snapshot B > Snapshot C > Snapshot E > Snapshot F > Snapshot Current

All snapshots only contain data blocks which have been modified, while that specific snapshot was active, i.e. the last one in the chain.

As mentioned before, deleting a snapshot (the correct way) will merge the stored deltas into its direct parent.

If you are going to delete "Snapshot C", its delta will be merged into "Snapshot B".

If you are going to delete "Snapshot A", its delta will be merged into the base virtual disk.

Deleting Snapshot A and B together will merge the deltas into the base disk.

As for storing the snapshots, how do I tell ESXI where I want to store them if I don't keep them in a production VM?

Since all snapshots are in use, they should be stored together with the VM's base virtual disk. Storing snapshots on a lower performance tier will negatively impact the VM's performance. Btw. running VM's with multiple snapshots may also impact performance.

André

PS: I don't know your PowerCLI script, so I can't comment on this.

[MikeAW2010]

I see, so correctly deleting Snapshot B or C should 'not' cause problems to E, F, or the Current correct?

For ESXI, the Main VM is not considered a snapshot as well correct?

I'll post my script here:

Connect-VIServer -Server X.X.X.X -User username -Password *********

$date = Get-Date

New-Snapshot -VM VM_Name -Name "snapshot-$date"

Get-VM VM_Name | Get-Snapshot | Where-Object {$_.Created -lt (Get-Date).AddMonths(-2)} | Remove-Snapshot -Confirm:$false  

It was intended to create a new snapshot when it is run every week, and delete any snapshot more than 2 months old to conserve storage space. Is this method workable or am I approaching this wrong?

[continuum]

The only reason to create automated weekly snapshots I can think of is sneaky delayed sabotage - and in that case you should not talk about it in public.

Seriously - do you want to keep snapshots as a form of backups ?

VMs with 5 or 6 snapshots need maintenance - they should never be regarded as normal.

[MikeAW2010]

It's a bit more complicated. The VM is acting as a repository for debian packages for other VM's to pull via Apt. Our intent was to create a snapshot for each state of the VM before adding new packages to the repository  incase we needed to roll back changes in the event one of the packages we serve is faulty. The idea would be to revert the repository VM to a state of which the last known functional packages are the current versions and the clients would pull those package from the repository in the event we have to revert them.

Packages are added daily hence the weekly snapshots.

[a_p_]

In such a case, snapshots may be acceptable.

What may be worth looking at, is whether replacing the last line in your script would make sense.

Get-VM VM_Name | Get-Snapshot | Where-Object {$_.Created -lt (Get-Date).AddMonths(-2)} | Remove-Snapshot -Confirm:$false

deletes old snapshots, one after the other.

Get-VM VM_Name | Get-Snapshot | Where-Object {$_.Created -lt (Get-Date).AddMonths(-2)} | Select-Object -Last 1 | Remove-Snapshot -Confirm:$false -RemoveChildren

I didn't test this, but if this works as I expect, it should delete all old snapshot in a single run.

Maybe LucD​ can have a look at this.

André

[LucD]

The Select -Last 1 will only remove 1 snapshot.

To remove the snapshots in chronological order, I would add a Sort-Object on the Created property.

Something like this perhaps

Get-VM VM_Name | Get-Snapshot |

Where-Object {$_.Created -lt (Get-Date).AddMonths(-2)} |

Sort-Object -Property Created |

Remove-Snapshot -Confirm:$false -RemoveChildren

[a_p_]

The intention behind the select -last 1 was to get the newest snapshot out of the ones filtered before, and then use the Remove-Snapshot with the "-RemoveChildren" option to delete this one, and all older snapshots. In this case the Sort-Object would however be required.

Get-VM VM_Name | Get-Snapshot |

Where-Object {$_.Created -lt (Get-Date).AddMonths(-2)} |

Sort-Object -Property Created |

Select-Object -Last 1 |

Remove-Snapshot -Confirm:$false -RemoveChildren

Does that make sense?

André

[NathanosBlightc]

I explain some about snapshot mechanism here: Virtual Machine Snapshot Details Investigation - Part 1

I hope it can be helpful for you

[MikeAW2010]

If I'm understanding properly, this would select the newest snapshot that is 2 months old or older, then delete it and any child snapshot related to it correct?

So for example:

Base VM > Snapshot A > Snapshot B > Snapshot C > Snapshot E > Snapshot F > Snapshot Current

If Snapshot A thru C were 2 months old, or older, and Snapshot C was the most recent of those Snapshots, Snapshot C would be selected. Snapshot E and F however would be child snapshots in comparison to C correct? If Snapshot C were dependent on A and B, but Snapshots E and F were less than 2 months old, what else would it be deleting? Just trying to grasp what you're inferring is all.

[a_p_]

I think I made a mistake here with the -RemoveChildren option, and mixed up children with parents, so the changed script is not what you want.

What should work for you is LucD's version without the -RemoveChilderen option. This will delete the snapshots from the oldest to the newest, thus always merging the deltas into the base disk, which avoids the requirement for additional disk space (if the base disk has been thick provisioned).

André

[MikeAW2010]

Thanks for the replies.

I have another problem I'm running into.

I'm attempting to configure a cronjob (Ubuntu Server 16.04 environment) to run the script in order to automate snapshot creation / deletion. Initially it was handled via an ansible play but we want to completely automate it. During testing, I'm finding that cron seems to not execute the script and no new snapshots are being created, yet if I run the cron contents manually over shell, it works fine.

#test

*/2 * * * * /home/[username]/scripts/repository/repository-snapshots

#! /snap/bin/pwsh

Connect-VIServer -Server [IP-Address] -User [Username] -Password ********

$date = Get-Date

New-Snapshot -VM [VM-name] -Name "[VM-name]-$date"

Get-VM [VM-name] | Get-Snapshot |

Where-Object {$_.Created -lt (Get-Date).AddMonths(-2)} |

Sort-Object -Property Created |

Remove-Snapshot -Confirm:$false -RemoveChildren

If I run

/home/[username]/scripts/repository/repository-snapshots

directly from the cmd, it works fine. But under the cron, no new shapshots are created. I can tell its running because I get this in my syslogs:

StartupStop:PowershellConsoleStartup.WinStop.Informational] PowerShell console is ready for user input

at the same intervals my cron is configured to start.

Any idea on this?

Thanks.

[LucD]

I had the same.
In my case, the snap installation caused it.
The path used in processes started by cron doesn't have the snap folder in the path variable.

Did a regular install, and the cron jobs worked

[MikeAW2010]

Basically I just need to reinstall Cron?

Edit: Reinstalled Cron via apt but still no success.

[LucD]

No, reinstalling cron didn't help me either.
I installed PS with the instructions Installing PowerShell Core on Linux
That way PS is installed in a folder that is in the path variable.

[MikeAW2010]

Thanks, Im remembering what happened now. I ran into this problem when I tried to install via apt:

The following packages have unmet dependencies:

powershell : Depends: libicu60 but it is not installable

Ubuntu-Server:~$ lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 16.04.6 LTS

Release: 16.04

Codename: xenial

Ubuntu-Server:~$ sudo apt-get install libicu60

Reading package lists... Done

Building dependency tree      

Reading state information... Done

Package libicu60 is not available, but is referred to by another package.

This may mean that the package is missing, has been obsoleted, or

is only available from another source

E: Package 'libicu60' has no installation candidate

This was even after an apt upgrade.

Any idea as to what could be causing this?

Thanks.

[LucD]

Can you try the suggestion mentioned in PowerShell issue Install fails on Ubuntu 18.04

The about the Universe repository.

[MikeAW2010]

seems to already be enabled:

~$ sudo add-apt-repository universe

'universe' distribution component is already enabled for all sources.