VMware Cloud Community
qc4vmware
Virtuoso
Virtuoso

vRO 8 scriptable javascript task restarts after long run time. Possibly memory issue or timeout?

I'm trying to convert a vRO 7 server that has a large inventory over to vRA 8.  A workflow that works fine in vRO 7 seems to have a strange bug popping up in vRA 8.  There is a scriptable task that runs through all of the vm's generating a csv for another team.  It is running through around 4000 hosts and 77,000 vms.  It gets to around 3500 hosts worth of the inventory after about 4 hours running then seems to restart.  So I'm wondering if I am bumping up against a timer or memory issue?  Not sure on vRO 8 where to further debug it.  There is no error the scriptable task just restarts so its stuck in a loop never finishing.  I'm wondering if maybe I just need to add some additional memory to the appliance but I've not seen a process for doing so with vRO 8.  We always needed to with vRO 7.

Attaching an image of a current run.  Shows max time as all zeroes so I assume that means it can run forever.

qc4vmware_0-1667414452418.png

 

0 Kudos
4 Replies
xian_
Expert
Expert

Anything in /services-logs/prelude/vco-app/file-logs/*.log log files?

qc4vmware
Virtuoso
Virtuoso

So it looks like 57 minutes ago the scriptable task restarted... same workflow now shows that step with a run time of 57 minutes

qc4vmware_0-1667421217115.png

Logs show this you can see on the 3rd line where the log shows the scriptable task item11 kicking off again start time in logs was 15:53 and it restarts at 19:33 so it ran for 3 hours and 40 minutes:

2022-11-02 19:31:13.016 +0000 INFO Processing VM's on [3239 of 4371] esx-vdesk-lascolo1-d16.qualcomm.com

2022-11-02 19:31:14.215 +0000 DEBUG vm WIN10-V3-20H2-Qgov4 is not in state poweredOn... skipping

2022-11-02 19:33:06.025 +0000 INFO __item_stack:/item11

2022-11-02 19:41:32.584 +0000 INFO Generating csv and json output for 77002 vm's on 4371 ESX Hosts.

2022-11-02 19:41:32.586 +0000 INFO Processing VM's on [1 of 4371] svdlbnxa03.dlb.prod

0 Kudos
xian_
Expert
Expert

I'd open a support case on that

qc4vmware
Virtuoso
Virtuoso

It looks like vco services totally restart so I guess its really restarting that whole workflow since that is the default recovery behavior.  The logs show the last couple lines I logged then all the logging shows java parameters getting set then a couple of minutes later the ascii graphic showing the vco services have started.  Nothing showing why it crashed but I'm guessing memory.  Where do you know where the java heap dumps get deposited on these appliances?

2022-11-02T19:31:13.016Z INFO vco [host='vco-app-656b7664dc-k2z69' thread='WorkflowExecutorPool-Thread-2' user='ppoppletx@NA.QUALCOMM.COM' org='-' trace='-'] {|__SYSTEM|ppoppletx@NA.QUALCOMM.COM:QCcpuCoreThreadCount:AB8080808080808080808080808080809380808001319826223236aebf2a6a5a5:token=42d767ac-3dcc-42cf-b0f9-e212881249b3} SCRIPTING_LOG - [QCcpuCoreThreadCount (11/2/22 15:53:27)] Processing VM's on [3239 of 4371] esx-vdesk-lascolo1-d16.qualcomm.com
2022-11-02T19:31:14.215Z DEBUG vco [host='vco-app-656b7664dc-k2z69' thread='WorkflowExecutorPool-Thread-2' user='ppoppletx@NA.QUALCOMM.COM' org='-' trace='-'] {|__SYSTEM|ppoppletx@NA.QUALCOMM.COM:QCcpuCoreThreadCount:AB8080808080808080808080808080809380808001319826223236aebf2a6a5a5:token=42d767ac-3dcc-42cf-b0f9-e212881249b3} SCRIPTING_LOG - [QCcpuCoreThreadCount (11/2/22 15:53:27)] vm WIN10-V3-20H2-Qgov4 is not in state poweredOn... skipping
2022-11-02T19:31:54.534Z INFO vco [host='vco-app-656b7664dc-k2z69' thread='localhost-startStop-1' user='' org='' trace=''] {} com.vmware.o11n.security.SecurityProviderUtils - Using security provider :SUN
2022-11-02T19:31:54.543Z INFO vco [host='vco-app-656b7664dc-k2z69' thread='localhost-startStop-1' user='' org='' trace=''] {} com.vmware.o11n.security.SecurityProviderUtils - Loaded : SUN class: sun.security.provider.Sun
2022-11-02T19:31:54.543Z INFO vco [host='vco-app-656b7664dc-k2z69' thread='localhost-startStop-1' user='' org='' trace=''] {} com.vmware.o11n.security.SecurityProviderUtils - Loaded : SunRsaSign class: sun.security.rsa.SunRsaSign
2022-11-02T19:31:54.544Z INFO vco [host='vco-app-656b7664dc-k2z69' thread='localhost-startStop-1' user='' org='' trace=''] {} com.vmware.o11n.security.SecurityProviderUtils - Loaded : SunEC class: sun.security.ec.SunEC
2022-11-02T19:31:54.544Z INFO vco [host='vco-app-656b7664dc-k2z69' thread='localhost-startStop-1' user='' org='' trace=''] {} com.vmware.o11n.security.SecurityProviderUtils - Loaded : SunJSSE class: com.sun.net.ssl.internal.ssl.Provider
2022-11-02T19:31:54.544Z INFO vco [host='vco-app-656b7664dc-k2z69' thread='localhost-startStop-1' user='' org='' trace=''] {} com.vmware.o11n.security.SecurityProviderUtils - Loaded : SunJCE class: com.sun.crypto.provider.SunJCE

0 Kudos