Long Running Workflows Fail with Authentication Errors After the Executing User Changes their Active Directory Account Password

[stevenbright1]

I have been encountering an issue for some time now where long running workflows (2+ weeks) will fail after the executing user changes their Active Directory account password and we either reboot the Orchestrator virtual appliance or restart the Orchestrator services on the appliance. We have configured the appliance to communicate with vCenter using a shared session and this functions without any problems (this password isn't changed nearly as often). Not updating the executing user's account password is not an option and is required every 60 days. Using a shared account with a different password policy to execute workflows is also not an option as an audit trail must exist that ties each action to an individual person. The workflows themselves are only accessing vCenter for information and thus the executing user's credentials aren't used for executing any portion of the workflow. We are currently using the vCenter Orchestrator virtual appliance version 5.1.1.

An example scenario is this:

1) User A executes long running workflow (workflow uses a waiting timer and checks vCenter for status information every 24 hours).

2) User A is required to reset their AD account password and does so.

3) Sometime later, the Orchestrator appliance/services are restarted.

4) All long running workflows executed by User A fail with the following error log entries (sensitive information removed):

2013-05-22 18:33:44.416-0400 WARN  [Execution] Unable to resume workflow:[Workflow Name], reason: Cannot login user : CN=LastName\, FirstName,OU=OU Name,DC=domain,DC=domain,DC=domain,DC=domain(reason : [LDAP: error code 49 - 80090308: LdapErr: DSID-0C090334, comment: AcceptSecurityContext error, data 52e, vece])

2013-05-22 18:33:44.433-0400 WARN  [Execution] User session fix disabled

2013-05-22 18:33:44.433-0400 ERROR [Execution] Unable to resume workflow execution [ff8080813d5bfd88013d8e725456016d], unable to create user session

[jvassev]

Hi Steven,

This is really vCO's expected behaviour. The credentials are saved as part of the workflow execution. At resume time the credentials are checked against the authentication provider (LDAP). As far as vCO is concerned, a changed password is indistinguishable from trying to authenticate a non-existent user.

You could try using vCO with SSO. SSO acts as mediator between vCO and the authentication provider and deals with tokens instead of passwords. The nice thing about tokens is that they can be renewed by vCO if necessary so even if the password is changed in the LDAP the token is still valid.

Beware that in SSO mode the password in not available. This could change some of your scenarios because you could no longer integrate with a 3rd party system that supports LDAP only. vCenter and vCloud can work in SSO mode though.

[stevenbright]

Jvassev,

How have others worked around this issue in the past (before SSO) for long executing lifecycle workflows?

Currently using the SSO server is not an option (unless we setup an SSO server just for Orchestrator, we are currently running vSphere 5.0) as we discovered an issue (later verified by VMware) with the way the SSO server serially handles authentication requests which allowed for search queries to cause the SSO server to stop responding until a search query completes (basically a DoS as repeated queries prevent any logons and many vSphere Web Client activities). Normally this wouldn't be an issue to for many environments, but in our AD, we have hundreds of thousands of objects. We can narrow the search scope down by specifying the User and Group OUs, but the problem we have is that we need to authenticate users from multiple OUs which only share the root of the AD forest as a common point. Thus, our searches were being executed across every object in AD and would cause the SSO server  to stop responding for up to 15 minutes every time a search query was executed. A SR was opened with GSS that was quickly escalated to a senior technical support engineer and to product engineering/developers, but the final response after a few months was that the issue would require large changes in the code base and that there wouldn't be any solution in the next minor release of 5.1 or the next major release (currently in beta).

[cdecanini_]

An option is to change the credential in the workflow with a system account right before the long wait. You will still have audit trails to know who was the user starting the workflow even if the end of the workflow is run with another account.

Not ideal but should work.