VMware Cloud Community
r0j
Enthusiast
Enthusiast
Jump to solution

[vRO] Workflow collisions on powershell host

Hello All,

Having random issues with workflows failing, related to our powershell host via vRA event subscription orchestrator tasks that create MS DNS entries using dnscmd .

This issue happens consistently when launching 3 or more catalog item requests simultaneously in vRA.

We are using the Add DNS Record workflow from PSCoE (on solution exchange), after switching over from using run script in guest, which was overly complex for such a simple task, and also was experiencing the same issue as above.

Turned on vmtools logging, switched powershell hosts, the symptoms are consistent.

It looks like there are collisions/contention issues with the powershell script copy to the host before execution.

We tried adding random timers to the workflow, which only reduces the problem but doesn't eliminate them altogether.

Has anyone seen this kind of behavior before?

Any feedback or alternative methods would be welcomed and appreciated.

Thanks and Cheers...

Reply
0 Kudos
1 Solution

Accepted Solutions
r0j
Enthusiast
Enthusiast
Jump to solution

We resolved this with a lock / wait / unlock on the workflow, not ideal but works 100% of the time now.

A support ticket is also opened to get any feedback from the developer, in case we are missing anything as to why it doesn't work out of the box.

Will post if anything productive comes from it.

View solution in original post

Reply
0 Kudos
4 Replies
daphnissov
Immortal
Immortal
Jump to solution

This is always a problematic method when it comes to concurrency, locking, and PowerShell. Honestly, a much more scalable, robust, and also fully supported method is the SovLabs DNS module which doesn't require any vRO coding or EBS subscriptions on your part, and you can use different connection methods to your MSDNS server. I've seen zero problems running this in high scale or high concurrency deployments.

Reply
0 Kudos
r0j
Enthusiast
Enthusiast
Jump to solution

I was afraid someone would say that...

Smiley Sad

We attempted to engage sovlabs on pricing, and cant seem to get a straight answer without them fully engaging.

Not ideal...

Regarding the DNS Record workflow from PSCoE on the solution exchange.

There is the ability to add multiple powershell hosts in the configurations / attributes for the workflow (we added three).

The issue here is the first host in the list is the only one that is called from the workflow, and the ability to round robin between them combined with some delay timers might resolve the issue.

Also, we have been considering moving DNS to the infoblox appliance, as it currently does our IPAM for the environment.

Motivation to move in that direction I suppose, although the in service people may have some strong objections to that.

Thanks for the feedback.

Cheers

Reply
0 Kudos
daphnissov
Immortal
Immortal
Jump to solution

You might want to speak with them regarding pricing anyhow. It's usually never as simple as asking a company, "how much for this thing?" and they give you the answer. It might be worth your time to check it out. I've had mixed luck with just about anything else when you start getting into concurrency issues, and PowerShell hosts are the worst about that (not just with DNS but with module invocation as well). They also have Infoblox integration for both DNS and IPAM, and I've used that fairly extensively again without any observed issues with as many as 500 deployments a day.

Reply
0 Kudos
r0j
Enthusiast
Enthusiast
Jump to solution

We resolved this with a lock / wait / unlock on the workflow, not ideal but works 100% of the time now.

A support ticket is also opened to get any feedback from the developer, in case we are missing anything as to why it doesn't work out of the box.

Will post if anything productive comes from it.

Reply
0 Kudos