vRO (vCO) HTTP-REST Workflow - Connection pool shut down

[VIR2AL3X]

Hi All - I recently upgraded to vRealize Automation 6.2 and am working with vRealize Orchestrator to create workflows.  I've created HTTP-REST workflows to request/release an IP address from our IPAM system (VitalQIP) and it seems to be failing with the following description/exception:

Connection pool shutdown (Workflow:Request-IP-from-QIP / Scripting (item3)#14)

Looking in the Schema>Scripting line #14 is in red below:

//prepare request

//Do not edit

var inParamtersValues = [subnet, hostname, ddns, mac, type, w2kdom, comment];

var request = restOperation.createRequest(inParamtersValues, null);

//set the request content type

request.contentType = "";

System.log("Request: " + request);

System.log("Request URL: " + request.fullUrl);

//Customize the request here

//request.setHeader("headerName", "headerValue");

//execute request

//Do not edit

var response = request.execute();

//prepare output parameters

System.log("Response: " + response);

statusCode = response.statusCode;

statusCodeAttribute = statusCode;

System.log("Status code: " + statusCode);

contentLength = response.contentLength;

headers = response.getAllHeaders();

contentAsString = response.contentAsString;

System.log("Content as string: " + contentAsString);

// Grep for IPaddress

var patt = ("IP Address.*");

var ipaddress = contentAsString.match(patt).toString();

ipaddress = ipaddress.split(/[\s,]+/);

ipaddress = ipaddress[2];

System.log ("IP Address := " +ipaddress);

I am unsure of how to get around this issue and am looking for some assistance as I am fairly new to vRO.  Thanks much!

[cdecanini_]

Could you please deploy a vCO 5.5.2 appliance and import your workflow to check if it works there.

If I am heading in  this direction this is because I have seen someone else who had the same problem and who did not before using vCO 6.0 from VRA 6.2.

If you confirm that you do not have an issue on 5.5.2 then I would advise to open a GSS ticket so a fix or work around can be provided to you.

Christophe.

[cdecanini_]

Also before checking with 5.5.2 double check the self signed cert on your vCAC environment. It may be expired.

[VIR2AL3X]

Thanks for the suggestions.  This is a new vRA 6.1 environment that has only been up for about a month and I upgraded to 6.2 a few days ago when it was released so I assume that the cert should still be valid.  But I will still check it out and report back.  I am going to deploy a 5.5.2.1 appliance as well and test again.  Will report back soon.  Thanks cdecanini

[VIR2AL3X]

cdecanini, apologies for the delay...holidays.  Anyhow - I have gone ahead and deployed the vCO 5.5.2.1 Appliance into my lab and reconfigured the Workflow exactly as previously done in vRO 6.0.

I am not getting the same error as before but am getting a new one seconds after starting the workflow:

"...Description: TypeError: Cannot call method "toString" of null (Workflow:Request-IP-from-QIP / Scripting (item3)#28)..."

Looking at script line #28 under Schema is the following line...

Here is a screen shot of the logs:

I am not sure how to proceed and am stuck.  Could you kindly assist please?

Thx much in advance.

Alex.

[Burke-]

Try removing the ( ) in that line.. The pattern should just be a string, right?

[VIR2AL3X]

Thx Burke- but unfortunately, removing the ( ) surrounding "IP Address.*" results in the same error.  If I instead remove the () at the end of Line#29 following .toString, then the error changes to "Cannot read property..." instead of "Cannot call method..."

[iiliev]

When you perform the pattern matching, you are assuming that the match will always succeed and you will be able to call toString() on a valid non-null object.

var ipaddress = contentAsString.match(patt).toString();

Apparently, the first subexpression contentAsString.match(patt) return NULL, and when you try to invoke toString() on NULL, you'll get the error 'cannot call method...'

So the solution is to split the above expression to something like:

var ipmatch = contentAsString.match(patt);
if (ipmatch == null) {
   // somehow handle the case when there is no match
} else {
  var ipaddress = ipmatch.toString();
  ... // the other statements as in the original code
}

Also, look at the logs where the contentAsString variable is printed. It seems that your HTTP call has returned an error The requested resource (/qip/ticadd.pl) is not available so it is kind of expected that the matching attempt will fail.

[VIR2AL3X]

Thx much Ilian Iliev.

Odd that it's returning a "Not Found" 404 error as I can access the URL just fine from any internal browser.

I appreciate your advice/feedback but this seems a little over my head as I am not a scripting expert.  I am trying to follow a guide (see attachment) to integrate vCO with Lucent's VitalQIP IPAM solution.  The directions in the guide were written by chiragpatel (using vCO 5.1) but I have no way of contacting him for assistance, and am unable to find any solutions on the web to make this work with our vCO 5.2 or vRA 6.0.  The workflows are simply to request or release an IP address from QIP.  Is there any way to get chiragpatel involved in this discussion for assistance?

[cdecanini_]

I have sent an email to Chirag. I do not know his availability to check on this but at least he will soon know about it.

Christophe.

[VIR2AL3X]

Excellent!  Thanks much cdecanini!  I hope he will be able to find time to assist and I appreciate all of the efforts given so far from all users

Alex.

[chiragpatel]

Alex,

From surface it seems that you need to tweak the REST call to what is being expected by your QIP implementation. You mentioned that your request goes through when you access URL from internal browser. I would take that request and make sure that the Parent host and Template URL setting during “Add a Rest Operation” are correct.

For example if you are using this (http://qipserver.yourdomain.com/cgi-bin/ipadd.pl?/subnet=192.168.1.0&hostname=test123) as a URL from your browser, I would set the Parent Host and Template URL to following.

Parent Host : http://qipserver.yourdomain.com/cgi-bin

Template URL : /ipadd.pl?/subnet={subnet}&hostname={hostname}

Again, this is just an example. You will have to match this to your setup.

[VIR2AL3X]

Hi chiragpatel! Thx so much for your response.  I have been trying to follow a guide you created (attached to an earlier post) and have run into these issues. So I need to understand what the correct template URL is then I guess for my setup.  I'd really like to go back to the initial post since we will be using the new vRO 6.0 that is integrated with vRA 6.2 which resulted in a "Connection pool shutdown" error.  These newer issues arose when using vCO 5.5.2.1 appliance.

I had simply done a copy/paste of the template URL from your guide into my setup.  Was this an incorrect move?

Apologies if I was misunderstood but I was not submitting a successful request via internal URL, I just meant that I can access my QIP just fine from a browser normally so I guess I need to understand the proper setup of the template URL if you can offer me any assistance and/or a bit of your time please.

My parent host URL to access QIP is simple. (example - http://qipdev.mydomain.com:port#/qip)

My template URL was a copy/paste from your guide: /ticadd.pl?subnet={subnet}&hostname={hostname}&ddns={ddns}&mac={mac}&type={type}&w2kdom={w2kdom}&comment={comment}

Was there a typo in that URL?  should it be /ipadd.pl?/subnet=....and not /ticadd.pl?subnet=.... - I notice some discrepancies from the guide based on your recent post.

This is the Template URL I used from your guide for the next workflow to release the IP: /ticdelete.pl?reqin={hostname}&request={request}&ticket={ticket}&ddns={ddns}

Is that incorrect as well?  I look forward to your reply, thanks again!

Alex.

[igaydajiev]

Hi,

"Connection pool shut down" was recently reported and we were able to track it down to an error in third part library (Apace HTTP Client).

http://stackoverflow.com/questions/25889925/apache-poolinghttpclientconnectionmanager-throwing-illeg...

You can test with the latest available version of httpclient library (4.4) or downgrade them.

This can be achieved by replacing "httpclient-4.3.5.jar" and "httpcore-4.3.2.jar" jars in {VCO_INSTALATION}\app-server\deploy\vco\WEB-INF\lib

[VIR2AL3X]

Hi igaydajiev thanks for this information.  I have found the files located in /var/lib/vco/app-server/deploy/vco/WEB-INF/lib

I've searched the downloads for these newer files (https://hc.apache.org/downloads.cgi) and noticed that http-client 4.4 is still in beta so the stable GA version is 4.3.6 which is still newer than the one installed (4.3.5).  Http-core is GA version 4.4 so I will replace these and see what happens on vRO 6.0.  I will report back soon after testing, but I fear that if this resolves the "Connection pool shut down" error I will still have the other errors I encountered above in vCO 5.5.2.1.

Edit: igaydajiev, after replacing the files and restarting the server/service I cannot launch vRO!

[igaydajiev]

you need to update the permissions of replaced files

[VIR2AL3X]

Can you elaborate a bit please?  I have gone ahead and restored the original files and having the same issue still.

[igaydajiev]

Regarding the version of httpclient I believe I read somewhere that this issue is fixed in 4.4 is and not sure if it has been backported.

Any way it worth trying. If the error persist you can try with 4.4 beta or downgrading the httpclient  to the version used in vCO 5.5.2

[igaydajiev]

since you have uploaded new files as user root the ownership of the files and access rights got changed. This is the reason why vCO server is not able to start.

From provided screenshot you can see that newly uploaded files has owner "root and Rights set to "rw" while other .jar files have owner vco and rights  "rwrr"

After uploading the files you need to set the correct owner/group and file access permissions.

[VIR2AL3X]

OK thanks much!  Let me give this a try and see what happens.  Is it recommended to restart the server afterwards or just the service?