Managing a long running process as a workflow component

[jbweber2]

I've read through https://blogs.vmware.com/orchestrator/2013/01/how-to-deal-with-long-running-external-processes.html and the content makes sense. Based on whats there I want to see if the solution I'm considering would work or if there is a better way to implement this process.

I've got a long running process which is monitored by doing periodic polling, and when it's poll its fairly simple to get the status of every outstanding action in one call saving doing polling per instance of the long running process.

I've got a workflow which needs to invoke this long running process as part of its execution. When the workflow calls out to invoke the long running process I will register the id of the long running process and of the workflow in a configuration element.

This workflow will this sit waiting on a user interaction.

Behinds the scenes a separate scheduled workflow will periodically get the status of all long running processes and then correlate with info in the configuration element. It will then respond to the waiting user interactions with the results of the long running process.

Is there a better way to implement this kind of process, or is there anything I've missed about how orchestrator works which may prevent this from working?

[tschoergez]

Hi!

This makes sense and is quite what I had in mind when writing the last paragraphs of the article.

So you keep only one "polling" workflow instance, no matter how many external things to check.

And using the User Interactions for the "workload" workflow will pause this workflow during the waiting time, so that it does not consume any resources during this timeframe.

You just have to check if you can programmatically answer user interaction from a workflow, but I think that's possible (some method for a WorkflowToken object).

Nice side effect BTW: If the polling workflow fails for whatever reason, having the workflows paused with a user interaction even allows a human admin to do the checks and answer the interactions.

In general for signaling between different workflows, this is worth a read: (But i does not contain any better way to do it right now 🙂

Re: How to model parallel workflows with dependencies?

Cheers,

Joerg

[jbweber2]

Doing a bit more investigation it doesn't appear that the scripting api exposes a way to answer a user interaction unless I'm missing something.

It appears that the user interaction is a WorkflowInput and you must answer the input with parameters matching the setup from the user interaction, but I don't see any methods which allow you to get the WorkflowInput from a WorkflowToken.

I was able to quite easily answer the interaction using the vCO REST API, but that feels counter intuitive to have to create a circular configuration to call out to the REST API from within vCO.

I think I could also expose the ability to access the user interactions from a small plugin instead of having to resort to using the REST API. At that point though is there any difference between doing the user interaction and the proposed trigger plugin from the other thread? Having the ability for a human to answer is maybe the greatest benefit I see from going that direction.

Have I missed something while exploring the api which would allow me to do this without accessing the rest api or creating a plugin?

[jbweber2]

I wasn't able to find any way to answer a user interaction with the default exposed api so I whipped up a small vCO  plugin to expose what I needed. The plugin its self is fairly raw, but with access to answer user interactions from javascript code the interaction pattern works as I was hoping.

Hopefully it either turns out I've missed a default way to do it, or this can be exposed in a future release of vCO because I think this pattern is quite powerful.

I'm also going to investigate creating a generic set of triggers just to learn a bit more about the vCO plugin api, but I think the user interaction is more powerful for general purpose development after playing with it over the weekend.

If anyone is interested you can find the code for the plugin on github below, comments are welcome!

https://github.com/jbweber/vco-toolbox-plugin

[tschoergez]

WOW!!!

I haven't had the time to test your plugin yet, but thanks for sharing! That is really powerful stuff.

Generic triggers would be the best solution I think.

Unfortunately, there are no real examples available how to build triggers for plugins "the modern way", so re-engineering solarsystem seems to be the only way so far...

I will try to get some more information about it.

Cheers,

Joerg

[thechaos]

Ich bin voraussichtlich ab dem 11.2.2014 wieder zu erreichen, E-Mails werden nicht weitergeleitet

[jbweber2]

I started to take a look at how I could create a generic plugin trigger system and it seemed straight forward until I dug into the example implementation in the SDK.

Creating the trigger is fairly straight forward, but I think I'm missing something on the watcher side of the equation.

It's unclear how creating the trigger ties into the "addWatcher" and "removeWatcher" methods. The example sort of makes sense, but all of the watcher managers are based on local data types so I don't see what happens when you're running vCO in a HA configuration or if you restart your instance how it continues to manage things. Are there any more examples out there or better descriptions of how this works other than the solar system in the SDK?

After digging into this the UserInteraction is much simpler to work with and seems to work across instance restarts as well just out of the box.

[mcfadyenj]

yes you are missing something 😉

you can answer a workflow from the api using the answerWorkflowInput()

you call the workflow using something like this.

WorkflowToken executingWorkflowToken = vcos.executeWorkflow(workflow.id, user, pass)                    ' # vcos = VSOWebControlService

this will return a workflowToken object which is your ticket to answering the workflow.

once you have a workflow token you need to monitor the status of the token.

string[] status = vcos.getWorkflowTokenStatus(token[] blah blah)

this will return an array with 1 value in it being a single string of

the first record in the array is the token status

string tokenStatus = status[0]

"waiting" "completed" "failed" etc

here you loop on the result.

while ("completed".equals(tokenStatus) == false

       && "failed".equals(tokenStatus) == false)

then the money shot of answering the token is here.

Create an array of WorkflowTokenAttributes containing the response criteria. Typically this is two items.

"Approved"

"comment"

vcos.answerWorkflowInput(token.id, array of attributes here, user, pass)

and your done.

[mcfadyenj]

I should add you can use the businessState property to control flow during these long waits etc.

It is pretty powerful out of the box and I haven't found much you cannot do with it. I am rolling 50 environments provision, test, undeploy, delete using this method takes a few hours but the waits are all internally handled no external entities are required. although I do appreciate the effort you have put in to achieve the external method.

I will certainly be testing that out.

[tschoergez]

Guys, this is a great discussion!

I am in process to get some more details and example how to implement triggers in a plugin, ETA within the next days (hopefully 🙂

Cheers,

Joerg

[jbweber2]

Very cool.

I've been trying to make an attempt at implementing triggers using the spring support package, but it's not gone too well so hopefully this will help close that loop.

[mcfadyenj]

one more thing,

if you are going to check out the native way. I have found that it seems to behave differently based on the source. sometimes you will get a "waiting" sometimes you will get a "waiting-signal".

I did find at some point which I have now lost that this require different responses.

waiting requires.

answerWorkflowInput

waiting-signal seems to believe its a third party system.

then it needs a customEvent sent as the response.

i have not found any real doco on this so this is all trial and error testing.

I have been toying with the concept of hand crafting AMQP messages as well using the AMQP .net client. I will keep you posted on this as I should have it sorted this week.

[mcfadyenj]

I thought of something else they may be useful to understand as for a while it confused me about this topic.

I was using blocking tasks and AMQP in this type of setup, one of the things I noticed was that using the answerWorkflowInput didnt seem to work with the blocking task.

although I never validated this I found that it appears the answerWorkflowInput will only work on 1 layer of workflows.

where I involved AMQP it seems that the correct workflow token was not tracked and therefore it would fail to answer as no waiting tokens were present.

not sure if this is a valid issue in the system you are implementing.

[jbweber2]

I've got the basics of a generic trigger system with code committed to the previously linked github repo.

The functionality of creating a trigger, attaching it to a waiting event, and then being able to signal that event to move on works as implemented (code is ugly, but it works). The triggers also appear to be durable across restarts of vCO which is nice. I can say after the time spent it would be nice to have some documentation related to how to do this with spring. I spent a lot of time with it not working because of what seems like a stupid bug of the "watch service" needing to be started. Surprised it's not handled by spring machinery by default.

I think the next order of business would be to create an inventory object related to the trigger which can be used to store state information about the status of said trigger. I was thinking I could do this with a configuration element as well, but I've not decided what would be better yet.