Sporadically failing VM Provisioned Workflow while running Customization Specification

[JensVM]

Hello,

since a few days I am exeriencing a strange issue with my Windows Customization workflow which is triggered by vRealize Automation Center after each provisioning of a Windows VM. This workflow has been running for months without any problems, but since a few days it sometimes fails with the following error message:

Attribute xsi:nil not allowed on element gateway, which is not nillable.

while parsing property "ip" of static type CustomizationIpGenerator

while parsing serialized DataObject of type vim.vm.customization.IPSettings

at line 39, column 5

while parsing property "adapter" of static type CustomizationIPSettings

while parsing serialized DataObject of type vim.vm.customization.AdapterMapping

at line 38, column 4

while parsing property "nicSettingMap" of static type ArrayOfCustomizationAdapterMapping

while parsing serialized DataObject of type vim.vm.customization.Specification

at line 5, column 3

while parsing call information for method CustomizeVM_Task

at line 3, column 2

while parsing SOAP body

at line 2, column 1

while parsing SOAP envelope

at line 1, column 38

while parsing HTTP request for method customize

on object of type vim.VirtualMachine

at line 1, column 0

If I interpret this error message right, it looks like that the element gateway value is null. After checking the variables of the failed workflow I can confirm that the gateway value and some other network related values are missing and therefore it is clear that the workflow cannot finish successfully. Actually I couldn't explain to myself why this sometimes happens now and only sporadically (sometimes it runs without any problem and the VM is customized correctly). All the information is passed on by vRealize Automation Center in the VirtualMachineProvisioned Workflow that is triggered automatically after doing a request in vRealize Automation. I know that there has been a bug a few months ago concerning VC:VirtualMachine objects not having passed correctly to the Orchestrator Workflow (if I remember right). Therefore I also applied the fix provided and afterwards I didn't have any problems anymore so far.

Some further notes:

Nothing changed in the configuration of the Blueprint and the Workflow which is triggered afterwards in vRealize Orchestrator. The machine is being prepared correctly by vRA in vCenter, I checked that. So the machine is ready for the additional customization workflow that runs afterwards - that shouldn't be the problem.

Some workflow details (the steps):

1. GetCustomizationGlobalIPSettings

2. Get Windows Customization (Sysprep)

3. Get NIC Setting Map

4. Update Nic Setting Map with changed Values

5. Get Customization Specification

6. Customize using customizeVM_Task

Does anybody know what could be the cause for this sporadically "loss" of the network settings and therefore the failing provisioning of VMs?

Best Regards

JensVM

[JensVM]

Hi there,

is there anybody who has an idea what this strange error could be related to? I've been checking all Application logs on the vRA and vRO Appliances and also all the logs of the IaaS Server - everything seems fine and also in the Appliance Management Console all services are registered. The only thing I remembered / found out is that I had to rejoin the IaaS Server to the Domain about 2 months ago because of some trust relationship trouble - now I saw that since then there are sometimes a few error events in the Windows Event Viewer produced by DEM Worker which haven't been produced before:

1. Access to the path 'C:\TEMP\RepositoryCache\Repository\DynamicOps.ManagementModel.Common.dll' is denied.

2. Error occurred while pinging repository

3. Error occurred writing to the repository tracking log

-> System.Net.WebException: The remote server returned an error: (500) Internal Server Error.

4. Failed to start repository service. Reason: System.UnauthorizedAccessException: Access to the path 'C:\TEMP\RepositoryCache\Repository\DynamicOps.ManagementModel.Common.dll' is denied.

I checked the repository status and it seems to run normally e.g. https://SERVERNAME/repository/data/ManagementModelEntities.svc/ is reachable via browser without problems.

Also Repository Update Messages are in the Windows Event Logs so it generally the repository connection seems to work.