Exception remove-snaphot cmdlet

[len_rat]

Hi there!

I have a script that backups VMs but when it comes to the remove-snapshot part it stops. The progressbar indicates removing 0 percent. The snapshots are removed from the VMs but after that there seems to be no communication to the PowerCli that it had been removed and just gives this error:

I think i run against a timeout or something. After this issue other PowerCLi cmdlets are failing with the same error message. Looks like somehow the connection and working of PowerCLI is broken. After starting fresh powershell instance everything is fine again. I already updated to 11.0.0 and .net 4.7.2 but the issue remains.

Also i need to make note that we tried using CA signed certificate on VCenter but had gone back to a snapshot. To be sure I did a renew of certifcate on ESXi hosts.

[LucD]

Can you check that you have configured PowerCLI to ignore invalid certificates?
And if you suspect a timeout as the cause, you might want to increase the default 300 second timeout to, for example 600, or higher.

Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -WebOperationTimeoutSeconds 600 -Confirm:$false

[len_rat]

Wow it is LucD himself, would like to take this special moment to thank you for sharing you're knowledge!

I already have the setting to ignore the invalid certificate, the timeout I haven't tried yet. Because this issue only seems to concentrate to a particular datacenter i want to try some other things. For now i will remediate the hosts and see if it helps.

After that I could go with increasing the time out value and let you know. Thanks!

[len_rat]

Seems that the issue remains with remediating and rebooting hosts, even with the timeout value increased. Also the issues occur on other datacenters as well. The interesting thing I think is that the timeout only happens with Remove-Snapshot, the snaps are removed from VC but the PowerCli machine never get reported back on the snapshot removal status (looking at stuck progresss bars in ISE even after snaps are completely removed):

As opposed to VCenter

Also within a fresh powershell instance:

Other operations like new-snapshot do get an updated progress bar. The use of the -RunAsync (background) is a workaround to resolve my problem so that the PowerCLI does not hang itself. Any other ideas?

[LucD]

Which PowerCLI version are you using?

And which PowerShell verson?

Get-Module -Name VMware* -ListAvailable

$PSVersionTable | Out-Default

[len_rat]

[LucD]

The versions look ok, could be something on the vCenter os ESXi nodes.

Did you already check the vpxd logs to see if there is any further info?

[LucD]

Looking again at the PowerCLI versions, I think there is a problem.

The VMware.PowerCLI is normally the module you install from the PowerShell Gallery.
And that is always the latest version (currently 11.1.0), while your screenshot says 6.5.4.

On the other hand, there are other PowerCLI modules which do seem to be on version 11.0.0.
Looks like your installation is messed up.

I would suggest to remove (manually) all the VMware modules from that folder, and then do a fresh install.

PS: not sure what the VMware module is about, but I suspect it is a local module.
Perhaps better to move that to your personal (CurrentUser) modules folder (C:\Users\<User>\Documents\WindowsPowerShell\Modules) or the global (AllUsers) folder (C:\Program Files\WindowsPowerShell\Modules).
Since PS v4 we are not supposed to have anything local in the C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules folder anymore.

[len_rat]

After removing all VMware folders from the module home I reinstalled the PowerCli module:

Unfortunatly the issue perists. Looking at the VPXD log I cant make anything from that. For now I have a workaround in place where I defer the removal of snaps to a job with start-job so that module in the initial instance keeps working. This is not ideal. Any toughts on what to do next?

[LucD]

Your modules look ok, so that is fixed.

Since you say that the same Remove-Snapshot seems to work in a Start-Job, I'm wondering if your PowerShell/PowerCLI sessions might have an issue.
Did you already stop/start the environment from where you run the Remove-Snapshot, after the creation of the snapshots and before the removal of the snapshots?