vCAC 5.2: MessageQueueErrorCode QueueNotFound for queue 'ForceExpire'.

[christianpg]

Hello community

After a hasty cluster migration from old to new vCenter server, I was able to get our vCAC 5.2 setup stabilized enough to provide virtual machines again.

However, when the VMs (pre-existing as well as newly created) reach expiration date, the stacktrace below fills the logs.

Apparently, something deep down in vCACs integration logic with Windows Workflow Foundation broke.

? Is there a DB table mapping endpoints or resources to WWF queues?

? How do I debug vCAC issues related to WWF?

Message:

Error processing lease changes Event Queue operation failed with MessageQueueErrorCode QueueNotFound for queue 'ForceExpire'.

Stacktrace:

at System.Workflow.Runtime.WorkflowQueuingService.EnqueueEvent(IComparable queueName, Object item) at System.Workflow.Runtime.WorkflowExecutor.EnqueueItem(IComparable queueName, Object item, IPendingWork pendingWork, Object workItem) at System.Workflow.Runtime.WorkflowInstance.EnqueueItem(IComparable queueName, Object item, IPendingWork pendingWork, Object workItem) at DynamicOps.VMPS.Service.VirtualMachineService.FireVirtualMachineEvent(Guid virtualMachineID, String eventName, Object data, Boolean synchronous) at DynamicOps.VMPS.Service.VirtualMachineService.FireVirtualMachineEvent(Guid virtualMachineID, String eventName, Object data) at DynamicOps.VMPS.Service.VirtualMachineObserver.ProcessLeaseCallback(Object state)

[Virtual10001]

Hi Christian,

Did you ever find a resolution to this ? I am hitting same issue after recently upgrading vCenter.

Appreciate any feedback on what resolved this for you

Thanks

[christianpg]

Additional info:

• The doDeletes has not been modified (set to default: true)
• The forceExpire is triggered when the leasetime is out. (no documentation found about this)
• Any machine that fails with the mentioned stacktrace can still be deleted successfully using the "Destroy" option. For now, the workaround is to manually clean up old VMs.

We have a support request open at VMware. So far, no productive response. Will keep you posted.

[christianpg · Accepted solution]

Today we were finally able to solve the issue:

The thread below mentions the root cause: an old VM that vCAC failed to "destroy"

You could trigger the destruction from vCAC,  but nothing happened.

1. We deleted the VM from vCenter.
2. Triggered state and inventory workflows from vCAC against the vCenter server.
3. Deleted the stuck VM from the vCAC database using the stored procedure which can be downloaded from KB 2077281  VMware KB:     Removing a virtual machine from Management in VMware vRealize Automation or vCloud Au...
4. AND MAGICALLY (no explanations from vmware support) the failing ForceExpire workflows started to run with subsequent removals of old archived VMs.

Re: vCAC 5.2: How to manually delete (SQL) a VM from vCAC?

[PowerPaul201110]

Hi Christian,

I'm getting the same error. Unfortunately I'm not sure how to figure out the VM that is causing this issue.

Do you have any hint for me? Is there a more detailed log file somewhere to look at?

Regards Paul