vRealize Orchestrator REST API Hangs after a few delete commands

[omrsafetyo]

So I was working on a script to poll the VRO API for workflows, which would essentially track these until completion, and archive them to a database.  The logic is basically:

1. Look at vRO and get a listing of executions for a given workflow, and get the details (execution Id, parameters, completion time, completion status, last step, etc.)
2. Check the database table where I am storing these to see if this execution Id has been logged
3a. If the execution Id is already logged, update the entry.
3b. If the execution Id is not already logged, insert the new entry.

4. Check the status of the execution - if it is in a completed state of some form (failed, completed), delete that entry from VRO..

The script worked great for logging - and it worked perfectly fine right until the point that I started deleting workflows.

I am basically running:

DELETE https://{vro_server}:8281/vco/api/workflows/{workflow_id}/executions/{execution_id}/

I am running this using Invoke-RestMethod in Powershell.  I also noticed that in the relations.link property of the return JSON when retrieving a execution, there are some other verbs available:

https://{vro_server}:8281/vco/api/workflows/{workflow_id}/executions/{execution_id}/logs/

https://{vro_server}:8281/vco/api/workflows/{workflow_id}/executions/{execution_id}/remove/

https://{vro_server}:8281/vco/api/workflows/{workflow_id}/executions/{execution_id}/state/

I had tried playing with the /remove/ resource, but I tried GET, PUT, PUSH, and DELETE, and it threw a 404 on each.  So it's entirely possible I'm going about this the wrong way, and if so, feel free to stop me here.

So, first I tested the DELETE verb against the execution.  This worked on my test execution.  So I added the logic into the script, added a long sleep, turned on verbose output, and kicked it off and watched.  There was 2 successful deletes right at the beginning.  It moved to the third, and I noticed it took longer.  Eventually I got an error indicating that the operation timed out.  It failed on the 4th as well.  I killed the script, and after making a couple modifications, I kicked it off again.  This time, even getting my listing of executions failed with a time-out issue.  It then moved on to my next VRO appliance (I have 4, because they only support 5 vcenter SDK connections each), it successfully pulled in all my executions, and then just like the first appliance, there was 1 success, then failure.  Then, a second failure. Then a success. Another failure, another success.  And then it moved to all failures.  Now if I try hitting the REST API for either of these appliances, it fails.

What gives?  Did I do something wrong?  How do I start this sucker back up?

[omrsafetyo]

Okay, so I feel a little foolish here.  As an update, I got the VRO appliance restarted from Control Center.  I then tried to retrieve a workflow, and it still failed.

I decided to try a new Powershell session, and it worked.  So it seems the issue was actually with the PowerShell session, and not the REST API.  This is quite bizarre.

[omrsafetyo]

Interestingly enough, the behavior is still reproducible.  I had taken delete portion out.  I logged all my workflows successfully.  I put it back, and in my new PS window, I kicked the script off.  After 2 successful deletes on a particular appliance, it stops working. 

[omrsafetyo · Accepted solution]

So for what it's worth, it looks like this is an issue with Invoke-RestMethod, and not the API itself.

I ended up changing my code from Invoke-RestMethod to submit the DELETE command to the API, to using Invoke-WebRequest.  This would have given me the ability to find out some more information that Invoke-RestMethod hides (HTTP status code, response headers, etc.).

Once I switched over, I found that the behavior no longer persists.