Why reading a property of an ESXi host takes 3 minutes?

[lurims]

I have an environment for a client to check an annotation value on all ESXi hosts.  The environment comprises of a couple of thousands of ESXi hosts.  I am iterating an array of ESXi hosts. It is taking 3 minutes per host to check an annotation and it is taking days to complete the run of the workflow.  Is the single array of all ESXi hosts is the culprit?  If so, what is the best way do it?  Should I limit to cluster level? Any other better idea?

[iiliev]

It is hard to say what could be the problem without knowing more details about the environment/configuration, and the code you use.

Have you tried to access other attribute of the hosts? For example, how much time is it taking to iterate over all the hosts and print their names?

[lurims]

After getting each host, I am checking the attribute value using this property for every esxi host, which return customAttributes/Annotations on that ESXi host.

esxiHost.vimHost.customFieldsManager.field

[Hazenet]

Please provide a screenshot of your workflow.

I think the problem is about Serialization of inputs for each Workflow element.

If you have a very large array, like your "couple of thousands of ESXi hosts" array, it can take quite a bit of time for the vRO server to serialize this and write it to the database.

You can read more about this on Page 23 of the "vRealize Orchestrator Coding Design Guide" pdf which can be found: https://docs.vmware.com/en/vRealize-Orchestrator/7.0/vrealize_orchestrator_coding_design_guide.pdf

One way to work around this problem, is to minimize the number of workflow elements in your workflow, and especially don't do a "graphical loop" using workflow elements, but instead do the loop thru the very large array inside a single scriptable task or inside one action.

That way, the vRO server only needs to write the serialized data once, and then it can loop thru the very large array "at full speed" in pure JavaScript code.

[lurims]

Thanks.  I like that comment "don't do a "graphical loop" using workflow elements," - possibly this could be the issue.

[Hazenet · Accepted solution]

Im speaking from experience.

I once did a workflow set for a large bank, that needed to pull a lot of Advance Settings from about 2500 ESXi Hosts, spread across like 200 Clusters on 30 different vCenters.

And the "graphical loop" combined with a vRO backed by an external (not super fast) MS SQL DB resulted in very very poor performance and extremely long run times.

After I converted that "graphical loop" into a single scriptable element, it run time went down from "many many hours" to around 30 minutes or so.