Search VM in Orchestrator is slow

[jonathanvh]

Hi,

We have about 3000 VMs spread over 2 vCenters.

We created an Orchestrator workflow that searches a particular VM.

But this takes about 30 seconds.

We have used following search methods:

vms = System.getModule("com.vmware.library.vc.vm").getAllVMsMatchingRegexp(".*" + pattern + ".*");

and

var xpath = "xpath:name[starts-with(.,'"  + pattern + "')]";
vms = VcPlugin.getAllVirtualMachines(null , xpath);

But these are both too slow.

Is there an other method for searching VMs in multiple vCenters? (Without knowing in advance in which vCenter the VM is located)

Thanks in advance!

Jonathan

[bhoriuchi]

Yes if you have the inventory service. You can use the built in work flow "Get virtual machines by name" (available on 5.5 not sure about before) or the following modified code to get the first occurance. allVimServers gives you ALL your sdkConnection objects and vmName is a variable for the name you are looking for.

var vmObj;

var allVimServers = VcPlugin.allSdkConnections;

for (var i = 0; i < allVimServers.length; i++) {

     if (allVimServers[i].isInventoryServiceAvailable()) {

          vmObj = allVimServers[i].getAllVirtualMachines(null, "matches(name, '" + vmName + "')");

     }

     else {

          vmObj = allVimServers[i].getAllVirtualMachines(null, "xpath:matches(name, '" + vmName + "')");

     }

     if (vmObj != null) {

          vmObj = vmObj[0];

          break;

     }

}

return vmObj;

[jonathanvh]

Hi,

I just tried it, but this isn't faster.

The command allVimServers[i].isInventoryServiceAvailable()) isn't available in version 4.2.2

I'm still searching for a way to speed up the search.

We changed in the wrapper.conf file following settings:

From:

# Maximum Java Heap Size (in MB)

wrapper.java.maxmemory=2048

To:

# Maximum Java Heap Size (in MB)

wrapper.java.maxmemory=4096

But this didn't help either.

Anymore suggestions?

Jonathan

[bhoriuchi]

What version of vSphere are you running? I believe what makes the code I provided faster is that it can take advantage of the inventory service which I believe is available starting in vSphere 5. If you can, try deploying the vCO 5.5 appliance and seeing if it runs any faster.

[jonathanvh]

We are running Orchestrator Version 4.2.1 build 555 The search per SDK connection is in fact even slower then the original version...

[ericr999]

I know this is a old post. But this look exactly my issue. I was wondering if others found a solution ? In my case it takes over an hour to complete the first getAllVirtualMachines, then it seems to store some kind of cache for an amount of time. Then usually the next morning the cache seems to be gone and it takes another hour to create the initial cache.

I'm running VRO 7.0, already increased the RAM 8GB and adjusted the JVM.

Found out that the problem seemed to be related to getAllVirtualMachines. Also, I have about 3000 VM on multiple VC, and it takes more than an hour to complete. And its having an impact to other workflows. Already have my ticket open with the support, but I was wondering about other peoples experiences in the community.

Thanks!

[jacksonecac]

Have you tried the baked in workflow for search for vm, I believe it is located in the Library -> vCenter -> Virtualmachine folder. Once you find your vms using that workflow you could filter them after the fact using each vms properties. It was pretty fast as I recall.

[ericr999]

I'm seeing the same pattern, its using the same method anyway, which is getAllVirtualMachines.

And in the logs, I see this:

'BUR_ct20ro82_https___Wpvcpri20_443_sdk_Main_60f5d3eb useIS_ false'

2016-08-30 12:36:23.550+0000 [WorkflowExecutorPool-Thread-8] INFO {cz2409-6@whatever:Get virtual machines by name:wfid=BB808080808080808080808080808080A180808001322751030482b80adf61e7c[BB808080808080808080808080808080A180808001322751030482b80adf61e7c]:token=8a8f841b56b2aaeb0156db6d3ee00112} [PluginCacheManager] Put cache element 'vm-2569' (1858102620) into 'BUR_ct20ro82_https___Wpvcpri20_443_sdk_Main_60f5d3eb useIS_ false'

2016-08-30 12:36:23.550+0000 [WorkflowExecutorPool-Thread-8] INFO {cz2409-6@whatever:Get virtual machines by name:wfid=BB808080808080808080808080808080A180808001322751030482b80adf61e7c[BB808080808080808080808080808080A180808001322751030482b80adf61e7c]:token=8a8f841b56b2aaeb0156db6d3ee00112} [PluginCacheManager] Putting cache element 'vm-9154' (230398570) into 'BUR_ct20ro82_https___Wpvcpri20_443_sdk_Main_60f5d3eb useIS_ false'

2016-08-30 12:36:23.550+0000 [WorkflowExecutorPool-Thread-8] INFO {cz2409-6@whatever:Get virtual machines by name:wfid=BB808080808080808080808080808080A180808001322751030482b80adf61e7c[BB808080808080808080808080808080A180808001322751030482b80adf61e7c]:token=8a8f841b56b2aaeb0156db6d3ee00112} [PluginCacheManager] Put cache element 'vm-9154' (230398570) into 'BUR_ct20ro82_https___Wpvcpri20_443_sdk_Main_60f5d3eb useIS_ false'

Every few minutes I see a couple of new lines in the logs.

Pretty sure its some sort of cache being built up... but its taking forever!

[lavinka]

Hello,

Did you have a chance to look at following discussion. using WaitForUpdate method of PropertyCollector to get any notification ?

Thanks,

Savina

[jacksonecac]

I would recommend either splitting out multiple vRO instances and chaining them together or having some kind of tagging / naming convention to help filter down your search subset. At the end of the day its searching a database so the better the query, the more efficient the results are. I also have had trouble with the baked in postgresql database, so if you are running production workloads, which it sounds like you are, I recommend a clustered solution with an external DB on the backend.

[ericr999]

The issue was related to some sort of cache managed by the vCenter Plug-in in vRO. I was at version 6.0.2, and the support requested that I either migrate to vRO 7.1 or upgrade to the version of the plugin 6.5.0, and it fixed the problem.

[jauling]

Is upgrading to 7.1 + 6.5.0 plugin the only way to alleviate the pain of traversing multiple vCenters? We run vCO 5.5.3 in production, and I am in the process of evaluating vRO 6.0.4, but this little update is peaking my interest.

We've been dealing with this slowness for years now, and haven't had much luck in improving the situation. We've basically chalked it up to "this is how its gonna be". FWIW, we run 12 vCenters all plugged into a single vCO instance, with a total of 5000-6000 VMs. You can imagine what kind of latency there is when you search for a single VM across all the SDK connections! Its even more surprising when you use an equivalent vCenter API call across the same 12 vCenters, and the API calls complete in mere seconds.

[dvatov]

7.1 comes with vCenter plugin 6.5 which has rewritten workflow 'Get virtual machines by name'. The 7.1(6.5) implementation does not rely on the built-in actions to filter the VM's list but use vCenter property collector for efficiency. If you look at the implementation there is a base workflow that can be used to filter other entities besides VMs and use other property instead of 'name'. There is a limitation that the filter does not work for array properties but this logic is in the workflow and can be customized. I think that these workflows can run on 5.5.x also. Additional benefit of this is that if your search is restrictive and return only few (1-2) VMs out of 6000 only these VMs will be cached and this will keep the cache small, will save network traffic for synching it with vCenter and will put less load on the vCenter.

[stevejin]

[Moderator note: The author of this post is from DoubleCloud.  The FTC’s Endorsement Guides require disclosure of vested interests in social media posts.]

If the performance is critical, you may want to try the DoubleCloud vSearch. With Google like GUI and speed, you can search VMs with all sorts of queries like VM name, IP address, MAC, ..., and event UUID. Here is a screenshot showing that searching 2,000+ out of 40,000+ VMs inventory takes about 0.118 second. To use it with workflow, you can try the built-in REST APIs.

[lurims]

Have you tried this approach?  Get the SDK connections, vCenters, DataCenters, Clusters and then search for the VM pattern asynchronously.  Fire different searches using asynchronous workflows and collect the results.  Getting up to Clusters would be faster.  You may even fire these async workflows at vCenter level or a cluster level.

[jonathanvh]

I have an update from my side.

Our 2 vCenters are now version 6.5.

The number of VMs in total is around 10.000

We installed vRealize Orcherator 7.5

vCenter Plugin: VC 6.5.0.11809518

I tried several ways to get all the VMs back:

vms = VcPlugin.getAllVirtualMachines();

vms = VcPlugin.getAllVirtualMachines(["name"], null);

var xpath = "xpath:name[starts-with(.,'"  + "NAME" + "')]";
vms = VcPlugin.getAllVirtualMachines(["name"], xpath);

And on our Orchestrator it will only take 3.5 seconds.

[jonathanvh]

All,

Again an update to this thread.

I have no solution for this yet, but a workaround.

I created a workflow that will run each 10 minutes and write the file in JSON

This is what I write in the file:

jsonObj["vmName"] = vm.name;

jsonObj["vmObject"] = vm.vimId;

jsonObj["sdkId"] = vm.sdkConnection.instanceUuid;

jsonObj["sdkName"] = vm.sdkConnection.id;

And we have a Worklow that will search for the name of the VM in that file and return the VM I needed

var fw = new FileReader(vmObjectFileLocation);
System.debug ("Start reading file: " + vmObjectFileLocation);
fw.open();
fileContent = fw.readAll();
fw.close();
System.debug ("Stopped reading file");
jsonObj = JSON.parse(fileContent);
for each (object in jsonObj) {
     if (object["vmName"].match("^" + name)) {
          System.debug ("VM found in cache file: " + object["vmName"] + " - object: " + object["vmObject"]);
          vmId = object["vmObject"];
          sdkName = object["sdkName"];
          break;
     }
}
vm = Server.findForType("VC:VirtualMachine" , sdkName + "/" + vmId);

0.009 seconds instead of 3.5 seconds.

[2020-04-14 11:09:24.883] [D] Start reading file: /var/lib/vco/app-server/temp/allVMsObjectInfo.txt

[2020-04-14 11:09:24.885] [D] Stopped reading file

[2020-04-14 11:09:24.888] [D] VM found in cache file: TEST-VRO - object: vm-4328

[2020-04-14 11:09:24.902] [I] VM found via cache file: TEST-VRO

Not the solution I want, but the one I have.