rickmagoon
Contributor
Contributor

Runaway WFStubBuildingMachine workflows timing out and looping

Jump to solution

I have a situation that occurred most likely because the ExternalWFStubBuildingMachine stub vCO workflow was swapped while six VM service requests were in flight.  Now I have these VMs stuck within the WFStubBuildingMachine workflow and looping several times a second.  I have a truck load full of logs and generating several times a second - and says exactly:

Machine , state BuildingMachine: InvokeRepositoryWorkflow, error caught: Machine , state BuildingMachine: InvokeRepositoryWorkflow. Repository workflow WFStubBuildingMachine timed out.

I tried everything that I know of.  These include the below:

Restarting vCAC Manager, DEMs and Agent services

Ran vCO workflow to remove the modification of stubs on all WFStub workflows

Rebooted the entire stack (Luckily - pre-production)

Reverted back the WFStubBuildingMachine to older revision in ASD

Tried destroying machines in compute resources (results in error)

Deleted VMs in vCenter

Attempted to move machines to different compute/reservation (resulting in error)

Attempted delete of blueprint (but associates these VMs as managed objects)

The property entity bucket on these VMs look to be dropped and VMs listed as "Unknown" in compute resources > manage machines.  I don't know where these jobs live or whether the only solution is taking tweezers to the SQL database.  I was hoping for a command-line utility to force kill these jobs but can't find anything relevant.  This doesn't appear to affect any functionality but eventually the file system will explode from log overdose.  Any help is appreciated.

0 Kudos
1 Solution

Accepted Solutions
vmmeup
Expert
Expert

Your best bet is to first stop the DEM services until you remove the items from the database then:

As you put it take tweezers to the database.  You can go see what is in the DynamicOps.RepositoryModel.WorkflowInstances table and remove any related entries and then go to the dbo.VirtualMachine table and remove the entries for the machines.  There could be dependencies you may need to follow and delete as well.

Sid Smith

https://www.dailyhypervisor.com

Sid Smith ----- VCP, VTSP, CCNA, CCA(Xen Server), MCTS Hyper-V & SCVMM08 [http://www.dailyhypervisor.com] - Don't forget to award points for correct and helpful answers. 😉

View solution in original post

0 Kudos
1 Reply
vmmeup
Expert
Expert

Your best bet is to first stop the DEM services until you remove the items from the database then:

As you put it take tweezers to the database.  You can go see what is in the DynamicOps.RepositoryModel.WorkflowInstances table and remove any related entries and then go to the dbo.VirtualMachine table and remove the entries for the machines.  There could be dependencies you may need to follow and delete as well.

Sid Smith

https://www.dailyhypervisor.com

Sid Smith ----- VCP, VTSP, CCNA, CCA(Xen Server), MCTS Hyper-V & SCVMM08 [http://www.dailyhypervisor.com] - Don't forget to award points for correct and helpful answers. 😉

View solution in original post

0 Kudos