Getting all Snapshots in a large environment with vRO fails regularly

[FlorianZimmer]

Hi all,

as part of automating the deletion of snapshots, I build a workflow that requests all VMs from the VCs and then infers all snapshots. This, however, regularly fails.

We are talking about an environment with multiple VCs in ELM and a few thousand VMs. All the scripts that I will go into work perfectly fine, complete in seconds and have never even once failed in a test environment with one VC with about 100 VMs. The Orchestrator is on version 8.9.0.

The built-in action "getPropertiesForAllSnapshotTask" executes for about 30 minutes and has about a 70% failure chance. The errors that have been observed are:

Unable to convert object, 'Stub: moRef = (ManagedObjectReference: type = VirtualMachineSnapshot, value = snapshot-***, serverGuid = null), binding = https://***:443/sdk' plugin exception reason : convertToResult() --> Finder 'VC:VirtualMachineSnapshot' : unexpected error 'ch.dunes.model.sdk.SDKFinderException: convertToResult() --> Finder 'VC:VirtualMachineSnapshot' : unable to invoke read method : 'description''

and

TypeError: Cannot read property "connectionState " from undefined (Dynamic Script Module name : getPropertiesForAllSnapshotTask#6)

I was able to mitigate the latter by writing my own version of getPropertiesForAllSnapshotTask and reduce the failure rate to about 40%.

It implements the check "if (typeof vm != 'undefined') {"... and also excludes vcls VMs  and templates with "if(vm.runtime.connectionState.value=="connected" && !vm.config.template && vm.name.match(/(vcls)\w*/g) == null){"

The former, however, I wasn't able to resolve, even after writing a completely new function to get all VMs as seen below and checking the properties of all VMs one by one instead of collecting all VMs (the built-in "getPropertiesForAllSnapshotTask" uses "getAllVirtualMachines" and then checks for snapshots) and then checking for snapshots.

for each (vCenter in vCenters){
System.log(vCenter.name);
var clusters = vCenter.getAllClusterComputeResources();
for each (cluster in clusters) {
System.log(cluster.name);
var vms = vCenter.allVirtualMachines
for each (vm in vms) {...

This, however, takes 18 hours (!!!) before always failing with a "TypeError: Cannot read property "template" from undefined (Dynamic Script Module name : MY_getPropertiesForAllSnapshotTask#17)"

Does anyone know whats going on here? I have already spent way too much time on this. I am suspecting that the workflow fails because there are so many VMs in such a large environment, but I have no idea as to how to proceed.