I have a situation that occurred most likely because the ExternalWFStubBuildingMachine stub vCO workflow was swapped while six VM service requests were in flight. Now I have these VMs stuck within the WFStubBuildingMachine workflow and looping several times a second. I have a truck load full of logs and generating several times a second - and says exactly:
Machine , state BuildingMachine: InvokeRepositoryWorkflow, error caught: Machine , state BuildingMachine: InvokeRepositoryWorkflow. Repository workflow WFStubBuildingMachine timed out.
I tried everything that I know of. These include the below:
Restarting vCAC Manager, DEMs and Agent services
Ran vCO workflow to remove the modification of stubs on all WFStub workflows
Rebooted the entire stack (Luckily - pre-production)
Reverted back the WFStubBuildingMachine to older revision in ASD
Tried destroying machines in compute resources (results in error)
Deleted VMs in vCenter
Attempted to move machines to different compute/reservation (resulting in error)
Attempted delete of blueprint (but associates these VMs as managed objects)
The property entity bucket on these VMs look to be dropped and VMs listed as "Unknown" in compute resources > manage machines. I don't know where these jobs live or whether the only solution is taking tweezers to the SQL database. I was hoping for a command-line utility to force kill these jobs but can't find anything relevant. This doesn't appear to affect any functionality but eventually the file system will explode from log overdose. Any help is appreciated.
