vRA 8.4.2 - intermittant error during VM provisioning

[sharan_kallur]

we quite infrequently get the following error:

SecurityException: : : Failed to query unique virtual machine by external id: [5022e22b-4504-03b6-5025-7e399dbee2cd]

Sometimes using the exact same blueprint, with the exact same inputs we get a successfull provisioning.

Question is what can be causing vRA so to fail the provisioning process so ”randomly”. There were instances where I could run the blueprint 20 times!! without failing, just to have it fail again after trying 2 hours later.

In the screenshots below you can see that there are occasions where we hit 10 continuos deployements withoug error, but other times is completely random....

The error is always the same, the UUID of the VM seems missing.  

SecurityException: : : Failed to query unique virtual machine by external id: [5022e22b-4504-03b6-5025-7e399dbee2cd]

Are there some vRA logs we can check? We already checked and in vCenter there are no errors logged during the whole provisioning process so we suspect it may be a vRA thing...

Any help would be greatly appriciated. 

[xian_]

Check the logs on the appliance, especially /services-logs/prelude/provisioning-service-app/file-logs/provisioning-service-app.log

In our case the self signed certificate became untrusted on the cloud endpoint after an upgrade. The preceding error message, right before the SecurityException, shown the actual error:

Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderExcepti
on: unable to find valid certification path to requested target

[sharan_kallur]

Seems like this is know issue and fixed in 8.5+

https://docs.vmware.com/en/vRealize-Automation/8.4.2/rn/vRealize-Automation-842-releasenotes.html New Intermittent failure to deploy machine connected to an NSX-T network and contains tags The deployment fails with an error: "SecurityException: : : Failed to query unique virtual machine by external id: [UUID]". This occurs when vRA queries NSX for the machine in order to tag it on NSX and receives multiple records since the machine is migrated during vMotion. Workaround: Try to deploy again or disable vMotion.

[ronaldod]

Also a bug in the latest NSX-tT Seen double vm's in the NSX-T manager. Or even disappearing together and registered back in the manager without tags.

[sharan_kallur]

VMware provided the hotfix for this issue.. thanks a lot all the suggestions.

[kritika021]

@xian_  We have done patching to our vRA8.12.1 environment recently and after that the provisions started failing at "Add chef client" step with error "javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderExcepti
on: unable to find valid certification path to requested target" 
You said that after upgrade the certificate became untrusted in your environment. Could you please tell me how you resolved this issue?

[xian_]

As far as I remember we re-validated the cloud account (opened it and clicked on Validate) in Cloud Assembly. Then it popped up a dialog to accept the cert and we were done.

Do you use the chef plugin of Orchestrator? I cannot see direct Aria integration to Chef. With this plugin I can see only Add and Delete Chef Host workflows, you may try to delete and re-add your host to cache the new cert.

[kritika021]

@xian_ We have re-added the chef host but it didn't work