I am having an issue where my vRO workflows fail if I attempt to perform two guest operations on a single machine at the same time. An example of how I can recreate the issue is start a guest script workflow that loops and attempt to use the built-in get processes from guest workflow. Below is the error. I am using VMTools 10.1.15 on the guest and it is running Server 2016.
I have been trying to narrow this down for some time and finally narrowed it down to two ops running at once. Is this a limitation? I have been researching and I have not seen anyone with this specific case. I have not discovered anything useful in the Logs (Windows, VMware, VMTools).
[2017-12-21 14:32:56.313] [E] Error in (Workflow:Get processes from guest / Scriptable task (item1)#8) The operation is not allowed in the current state.
[2017-12-21 14:32:56.320] [E] Workflow execution stack:
item: 'Get processes from guest/item1', state: 'failed', business state: 'null', exception: 'The operation is not allowed in the current state. (Workflow:Get processes from guest / Scriptable task (item1)#8)'
workflow: 'Get processes from guest' (C98080808080808080808080808080800180808001322751030482b80adf61e7c)
Forgot to mention, vSphere does not create a "Task" on the VM while the script is running. The error is something I have seen in vSphere before for failed VM tasks, so that was where I looked 1st.
Your issue sounds similar to an issue I was having. Are you using the Guest Script Manager to run the workflows by any chance? If so the issue is actually with how the timer is handling the sleep state. I got an engineering build which fixed it. If this sounds applicable let me know and I will provide you the support ticket I opened up for reference. It took me forever to explain and get them to recreate so you will lose a day of your life if you try a fresh case.
I am, yes this sounds exactly the same. Any information I can relay would be great.
The support team you opened up with was VMWare? I thought the Guest Script Manager was community driven. Or is the sleep itself vRO builtin?
This is good new though, I was worried I was alone in this.
If I modify the "Improved Sleep" and turn the entire thing into a simple sleep. Do you think that would resolve the issue?
Yes that will work around the issue but if you have a lot of jobs running it may put undue strain on the system as I understand it. Not best practice and why the put together the "Improved Sleep" workflow. This is community supported but the bug is a vRO issue nothing to do with the workflows. I put together a package that broke vRO using OOTB actions and workflows. I never got a really great explanation of what was broken but the engineering build they gave me fixed it. My support case number was 17569124809 for reference.
Thanks so much, I will try the work around tomorrow! I'll let you know!
Same issue here.
Can you give me the way you fix that please !
My provisioning VM is stuck because multiple scripts are executed on the guest VM and I have the same error message.
Thank you for your reactivity !
If you are on vRO 7.4, please, open a SR to VMware to getr the fix as suggested by @qc4vmware
I asked the same question here: How many simultaneous/concurrent programs/script will VMware Tools support?
In short, VMware Tools/hostd do not support more than one guest operation at a time. Someone from GSS sent me this update:
Thanks. I am rom GSS. I raised the feedback for the documentation team to clearly state "Only one vSphere API Guest OP is allowed at a time per VM. StartProgramInGuest does not wait until program finishes, so when it comes back, you can call it again to start another program"
I also asked about VMware changing the return message of "INVALID STATE" to "BUSY" in this situation and the response was:
But if you need the reporting state to be changed from "InvalidState" to "Busy", I think this should come from the product development team.
This is not a bug but simply a limitation that has not been explicitly documented. If it helps you can ask GSS to look at 18918702409.
What this means is that customers are responsible for ensuring that only one guest operation is in play per virtual machine at a time. The most logical solution is to use Orchestrator locks as a way to queue up guest operations. This helped me:
You are right. This problem was brought to my attention and got the same information (No multiple operations on the same VM allowed at the same time) and yes using the LockingSystem was my plan to fix this but unfortunately never got the time to work on the fix.