filosmith
Enthusiast
Enthusiast

getAllVirtualMachines problems

In troubleshooting another workflow that takes hours to complete, I created a simple workflow that takes an sdkConnection as an input, and does a getAllVirtualMachines().

VMs is an Array/VC:VirtualMachine here, output to an attribute:

VMs = sdk.getAllVirtualMachines();

System.debug(VMs.length + " VMs found.");

This logs the length of the array in a second or two, but then the VMs attribute remains undefined, with the vco server load average ramping up, and server.log spamming this for seemingly every VM:

2018-06-15 11:36:05.434-0500 [taskExecutor-8] ERROR {} [ObjectUpdateParser] assign disabledMethod -> VirtualMachine::vm-54649

2018-06-15 11:36:05.434-0500 [taskExecutor-8] ERROR {} [ObjectUpdateParser] Ignore error and continue with others updates

java.lang.RuntimeException: java.lang.IllegalStateException: Cannot find managedMethod with name: MountToolsInstallerImage

        at com.vmware.o11n.plugin.vsphere.pc.ObjectUpdateParser.applyProperyChange(ObjectUpdateParser.java:65)

        at com.vmware.o11n.plugin.vsphere.pc.ObjectUpdateParser.processObjectUpdate(ObjectUpdateParser.java:112)

        at com.vmware.o11n.plugin.vsphere.pc.CacheUpdate.processUpdateSet(CacheUpdate.java:110)

        at com.vmware.o11n.plugin.vsphere.pc.CacheUpdate$UpdateSetHandler.nextUpdateSet(CacheUpdate.java:171)

        at com.vmware.o11n.plugin.vsphere.pc.CacheUpdate.run(CacheUpdate.java:66)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:748)

It eventually threw this in the vco client:

[2018-06-15 10:56:06.041] [D] 2541 VMs found.

[2018-06-15 11:00:28.113] [E] Workflow:Get all virtual machines / Scriptable task (item1) : ch.dunes.model.type.ConvertorException: Unable to convert object, 'Stub: moRef = (ManagedObjectReference: type = VirtualMachine, value = vm-704346, serverGuid = null), binding =https://fqdn.com/sdk' plugin exception reason : convertToResult() --> Finder 'VC:VirtualMachine' : unexpected error 'ch.dunes.model.sdk.SDKFinderException: convertToResult() --> Finder 'VC:VirtualMachine' : unable to invoke read method : 'name''

Is this supposed to work? I think the error in the client is because a VM was deleted mid-way through the run?

vCO 7.3.0.5481809

vSphere Version 6.5.0.10000 Build 7119157

VC Plugin 6.5.0.7720419

Thank

9 Replies
tschoergez
Leadership
Leadership

Check out this discussion, seems a comparable situation:

Re: Slow looping through vCenter objects since upgrade from vRO 7.2 to 7.4

Can you somehow narrow down either the attributes you need, or the array of VMs you want to work with later?

filosmith
Enthusiast
Enthusiast

When searching for just 'name', like in the post you linked, it's doing the same thing. server.log is spamming this for every VM (it seems):

2018-06-21 06:38:55.233-0500 [taskExecutor-8] ERROR {} [ObjectUpdateParser] assign disabledMethod -> VirtualMachine::vm-564253

2018-06-21 06:38:55.233-0500 [taskExecutor-8] ERROR {} [ObjectUpdateParser] Ignore error and continue with others updates

java.lang.RuntimeException: java.lang.IllegalStateException: Cannot find managedMethod with name: UpgradeToolsFromImage_Task

0 Kudos
ectoplasm88
Contributor
Contributor

I'm wondering if you have a memory issue.  Have you modified the JVM settings at all on this orchestrator.  We've had to bump all of ours up substantially beyond the default memory settings.  Ours are all set to 6144MB, default is 2560 I think.

0 Kudos
filosmith
Enthusiast
Enthusiast

I doubled the amount of memory allocated to vRO, and it's still just doing the same thing.

0 Kudos
iiliev
VMware Employee
VMware Employee

These iilegal state exceptions look very suspicious, and could be one of the reasons for the slow performance (exceptions alone are much slower to throw/handle, especially if there are thousands of them).

I'll try to reproduce this on our lab. It could be somehow related to the version of vCenter, as I can't recall seeing such exceptions on vCenter 6.0.x, but I saw another report or two with similar kind of exceptions happening on vCenter 6.5.x.

rdowling2
Contributor
Contributor

Has anyone gotten to the bottom of this - seeing it since upgrading VCO from 7.2.1 to 7.5.0

0 Kudos
qc4vmware
Virtuoso
Virtuoso

I'm getting ready to upgrade to 7.5 in my lab today.  If I see this issue I'll let you know.  I can add quite large vm inventory to test against.

0 Kudos
qc4vmware
Virtuoso
Virtuoso

I did some very limited testing and I am seeing the described behavior in my 7.3.1 vRA/vRO environment but not in my lab which I just upgraded to 7.5 .

0 Kudos
mstefani77
Contributor
Contributor

Filo, did you ever get a resolution on this?  We're seeing a similar issue with that same error that loops through every VM in vCenter.  Once it happens once the vCenter plug-in cache seems to prevent it from happening again for 4 hours.  After 4 hours it happens again.  It's causing us other issues in vRA as XAAS won't load the form for us until it iterates through all the VMs throwing errors which in our case is like 10 minutes.

0 Kudos