I've asked this question multiple times, both on the forums, Import workflow via REST API , and of the vRO dev teams. The vRO dev team recognizes this is a serious issue, yet they refuse to actually do anything about it.
We're working on coming up with a new process using GitLab and each developer will get their own vRO dev server. The thought is to rely on package versions and do the branching in GitLab rather than in vRO, where branching isn't possible. It's fairly complicated and requires extra effort on behalf of the developer to true up their code, but it is a workable system and should prevent the problems we have today of developers stomping on each others code and accidentally pulling in low level workflows into production.
Nice to know that I'm not the only one who's in this situation. Maybe togheter we will be able to build a case and make them understand that this is really an issue.
So I'm thinking the exact same thing as you. Setting up a VRO instance in dev for each dev. But for this I would have to use the PostgreSQL database.
Right now we are using the SQL database because our production environnement uses a SQL one because of the load balancing.
So right now, how do you manage the code ? Have you developped some methods that can help you in some ways ?
Also, if I ever manage to get a dev vro for each developper, I don't know how I'll manage all of them to get connected to all external services. Like IPAM, vCenter, Cf-Engine/Ansible.
We currently don't have dev environnements for each of these systems, sometimes in my dev I have to work with a preprod of the vCenters, or with the production of Cf-Engine etc.
We switched to using the embedded postgres and moved away from clustered mode. We had too many split brain issues where each node would kick off a scheduled task, or neither, and the vRO team was unable to explain why this happened. We have a pretty busy vRO server and just make it larger (4 vCPU / 24gb RAM now), and it works pretty well as long as you modify all the right java params.
Right now we do our best to be diligent and use a central dev server in our testlab and push code via a package up to production and make sure to only import the code we intend to import. It ends up with unfortunate issues around accidentally pulling in downstream workflows or missing dependencies, but it isn't that bad. For reference, we have about 8 people regularly changing code and we have about 750 custom workflows.
We have a dedicated testlab with all its own services, and that is where we'll put the dev servers. Recommend you at least have a lab vCenter, even if it just has esxi-on-esxi hosts for testing. We have a lab IPAM as well (we use Infoblox). Having a lab is key - if we didn't have a lab we'd probably take down production systems all the time.
Would you care sharing your java params ? So far I've increased the memory in the vm, and I've ajusted the setenv.sh file for the initial jvm memory usage and its maximum value. But I haven't seen a huge gain in performance. So from my understanding, you run a single vro node in production ? That's interesting, I was also considering that in a future version I might have to go back to a single node with a postgres db.
On our side we are 3 dev and probably around 300 workflows.
Yeah, that's what I'm missing a dev environnement of all the components. But since all these environnements are maintained by other teams and sometimes they neglect to build a dev environnement, because they personally don't have the use for it or the ressources to maintain it. I guess I'll have to find a way to ask all the other teams to provide me with a dev environnement.
Thanks for sharing your situation!
Here's a very low-fi way to approach this. Not elegant, and does require some adherence to procedure by the team, but it's cost-effective and available today.
First, every developer keeps two clients open (I sometimes do more than that so I can inspect config elements + resource elements etc. while also looking at other workflows). In one client a dev identifies that he/she wants to make a change to a workflow. They open the workflow for editing in one client instance, then go to the other client instance and right-click the workflow --> Locking --> Lock. This prevents editing without unlocking it. They can still make modifications to it in the first client because it was open before the workflow got locked. But other devs would not be able to make mods without unlocking it first. This would be a signal that it was checked out and they should check with the other devs on the team before unlocking it.
Once you save the workflow from the window it was already open it, the lock will get overwritten and removed.
One catch is that it takes a few seconds from locking the workflow in a client to that workflow lock being recognized by other clients. During that window of time, another dev could open the workflow and begin editing not knowing that a lock was placed. If you want to be sure, you click the refresh button in the top before opening for editing.
This is clearly not a full proof fix, but if everyone pays attention and follows the process it could work at least a large percentage of the time. If you end up not making changes to the workflow, you would need to remember to unlock it after finishing changes, so I could see that as another issue. You might be able to run a report of locks every day or something to identify things that have been left locked.
It doesn't solve the problem of branching or if more than one dev needs to work on a workflow at a time, but it might help with the problem of overwriting each other's changes.
That's a nice workaround for the overwritting issue. It would be a good habit to use for now.
Its just sad that the product has been around for a long time now, and Git has been around for quite some time too, and yet in VRO there's no real strong support for versionning, ease of comparison between version and branching support.
But for now, I think you have a good suggestion. Can't wait to have something better though!
You might take a look at the vRealize Code Stream Management Pack For DevOps. Solution brief here. It can capture content from vRO servers (along with vRA, vROPS, and other content from other things) and put it in source control as well as help with release pipeline if you need to get it to multiple downstream servers. It might be a step closer to your goal, but again it's not going to solve the problem of devs stepping on each other. It also requires some licensing. I believe as of vRealize Automation 7.3, you can use the onboard Code Stream instance for the purpose of using this vRCS Management Pack for DevOps, and that's included in the license. Otherwise, you need to license Code Stream.
I have faced with the same problems, but I'm looking for the solution from the other point of view.
Woud be possible to import a package which contains workflows, actions, resources, etc, but with new IDs ?
If it is possible each developer could do it and work with the same code (project/package), and after make some changes, sinchronize with a repository (GIT). But for this we need an import option which let us to use new IDs for each import.
What do you mean by new ID ?
A new workflow ID ? Or a new versions number of the Workflows/Actions/CE etc ?
How do you sync with Git ? Do you use a plugin ?
With ID I mean workflow or Action ID. This ID is the reference which collide when you tried to import two times the same workflow or action, etc. if would be possible to import more that one time the same workflow more that one developer could be work simultaneously. But it is necessary another ID or a way to work with the same ID in different directores for the same workflow or action, etc.
Workflow IDs are globally unique. You cannot have multiple instances of the same workflow with the same ID in different directories on a single vRO server.
The best way to allow several developers to work simultaneously on given workflows is to export the package containing these workflows to the file system, put the files under source control. Each developer then can deploy his own vRO server, checkout the package sources from the source control, import them from the file system into the vRO, make some changes to the workflows, export them to the file system, and push the changes to the source control where the other developers can pick them.
If for whatever reason all developers must share the same vRO server instance, each of them can duplicate the orgininal workflow (thus creating a new workflow with a different ID but with the same content) and work on the copy. But this will require extra work when the merging/pushing the changes back to the original workflow, and they have to be carefull to not overwrite the original workflow ID with the ID of the copied workflow.
I am the VMW Lead Architect for the solution raised above - the codestream management pack. We will soon be releasing a new version that is integrated tightly with the LCM product and has native source control integration. GA around a month or so away. This allows multi developer support, code reviews and much more!
That is great news! And this will work with a VRO Standalone ? I don't have VRA Server at the moment.