9 Replies Latest reply on Jul 8, 2019 3:42 PM by mstefani77

getAllVirtualMachines problems

filosmith Jun 15, 2018 9:47 AM

In troubleshooting another workflow that takes hours to complete, I created a simple workflow that takes an sdkConnection as an input, and does a getAllVirtualMachines().

VMs is an Array/VC:VirtualMachine here, output to an attribute:

VMs = sdk.getAllVirtualMachines();
System.debug(VMs.length + " VMs found.");

This logs the length of the array in a second or two, but then the VMs attribute remains undefined, with the vco server load average ramping up, and server.log spamming this for seemingly every VM:

2018-06-15 11:36:05.434-0500 [taskExecutor-8] ERROR {} [ObjectUpdateParser] assign disabledMethod -> VirtualMachine::vm-54649
2018-06-15 11:36:05.434-0500 [taskExecutor-8] ERROR {} [ObjectUpdateParser] Ignore error and continue with others updates
java.lang.RuntimeException: java.lang.IllegalStateException: Cannot find managedMethod with name: MountToolsInstallerImage
        at com.vmware.o11n.plugin.vsphere.pc.ObjectUpdateParser.applyProperyChange(ObjectUpdateParser.java:65)
        at com.vmware.o11n.plugin.vsphere.pc.ObjectUpdateParser.processObjectUpdate(ObjectUpdateParser.java:112)
        at com.vmware.o11n.plugin.vsphere.pc.CacheUpdate.processUpdateSet(CacheUpdate.java:110)
        at com.vmware.o11n.plugin.vsphere.pc.CacheUpdate$UpdateSetHandler.nextUpdateSet(CacheUpdate.java:171)
        at com.vmware.o11n.plugin.vsphere.pc.CacheUpdate.run(CacheUpdate.java:66)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

It eventually threw this in the vco client:

[2018-06-15 10:56:06.041] [D] 2541 VMs found.

[2018-06-15 11:00:28.113] [E] Workflow:Get all virtual machines / Scriptable task (item1) : ch.dunes.model.type.ConvertorException: Unable to convert object, 'Stub: moRef = (ManagedObjectReference: type = VirtualMachine, value = vm-704346, serverGuid = null), binding =https://fqdn.com/sdk' plugin exception reason : convertToResult() --> Finder 'VC:VirtualMachine' : unexpected error 'ch.dunes.model.sdk.SDKFinderException: convertToResult() --> Finder 'VC:VirtualMachine' : unable to invoke read method : 'name''

Is this supposed to work? I think the error in the client is because a VM was deleted mid-way through the run?

vCO 7.3.0.5481809

vSphere Version 6.5.0.10000 Build 7119157

VC Plugin 6.5.0.7720419

Thank

1. Re: getAllVirtualMachines problems

tschoergez Jun 20, 2018 9:06 AM (in response to filosmith)

Check out this discussion, seems a comparable situation:
Re: Slow looping through vCenter objects since upgrade from vRO 7.2 to 7.4

Can you somehow narrow down either the attributes you need, or the array of VMs you want to work with later?
1 person found this helpful
Like Show 0 Likes(0)

Actions
2. Re: getAllVirtualMachines problems

filosmith Jun 21, 2018 4:41 AM (in response to tschoergez)

When searching for just 'name', like in the post you linked, it's doing the same thing. server.log is spamming this for every VM (it seems):

2018-06-21 06:38:55.233-0500 [taskExecutor-8] ERROR {} [ObjectUpdateParser] assign disabledMethod -> VirtualMachine::vm-564253
2018-06-21 06:38:55.233-0500 [taskExecutor-8] ERROR {} [ObjectUpdateParser] Ignore error and continue with others updates
java.lang.RuntimeException: java.lang.IllegalStateException: Cannot find managedMethod with name: UpgradeToolsFromImage_Task
Like Show 0 Likes(0)

Actions
3. Re: getAllVirtualMachines problems

ectoplasm88 Jun 21, 2018 9:43 AM (in response to filosmith)

I'm wondering if you have a memory issue. Have you modified the JVM settings at all on this orchestrator. We've had to bump all of ours up substantially beyond the default memory settings. Ours are all set to 6144MB, default is 2560 I think.
Like Show 0 Likes(0)

Actions
4. Re: getAllVirtualMachines problems

filosmith Jun 22, 2018 9:47 AM (in response to ectoplasm88)

I doubled the amount of memory allocated to vRO, and it's still just doing the same thing.
Like Show 0 Likes(0)

Actions
5. Re: getAllVirtualMachines problems

iiliev Jun 22, 2018 10:00 AM (in response to filosmith)

These iilegal state exceptions look very suspicious, and could be one of the reasons for the slow performance (exceptions alone are much slower to throw/handle, especially if there are thousands of them).

I'll try to reproduce this on our lab. It could be somehow related to the version of vCenter, as I can't recall seeing such exceptions on vCenter 6.0.x, but I saw another report or two with similar kind of exceptions happening on vCenter 6.5.x.
Like Show 1 Likes(1)

Actions
6. Re: getAllVirtualMachines problems

rdowling2 Apr 30, 2019 1:14 AM (in response to filosmith)

Has anyone gotten to the bottom of this - seeing it since upgrading VCO from 7.2.1 to 7.5.0
Like Show 0 Likes(0)

Actions
7. Re: getAllVirtualMachines problems

qc4vmware Apr 30, 2019 8:57 AM (in response to rdowling2)

I'm getting ready to upgrade to 7.5 in my lab today. If I see this issue I'll let you know. I can add quite large vm inventory to test against.
Like Show 0 Likes(0)

Actions
8. Re: getAllVirtualMachines problems

qc4vmware May 12, 2019 11:53 PM (in response to qc4vmware)

I did some very limited testing and I am seeing the described behavior in my 7.3.1 vRA/vRO environment but not in my lab which I just upgraded to 7.5 .
Like Show 0 Likes(0)

Actions
9. Re: getAllVirtualMachines problems

mstefani77 Jul 8, 2019 3:42 PM (in response to filosmith)

Filo, did you ever get a resolution on this? We're seeing a similar issue with that same error that loops through every VM in vCenter. Once it happens once the vCenter plug-in cache seems to prevent it from happening again for 4 hours. After 4 hours it happens again. It's causing us other issues in vRA as XAAS won't load the form for us until it iterates through all the VMs throwing errors which in our case is like 10 minutes.
Like Show 0 Likes(0)

Actions