Failure executing 10_customizeos.bat

[Dean3]

Hey all,

Quick question, has anybody come across this error? "Failure to execute 10_customizeos.bat?

In VRA, seems like the machine gets passed buildingMachine, MachineProvisioning but hangs on customizeOS state and then disposes after timeout.

Error occurs provisioning a windows 2012 template.

It is a vCAC Guest Agent script I believe but don't understand why it is not executing? The logs don't offer much support, but in the log it looks as if it gets to "waiting for completion customizeos.bat" but nothing is logged after this..?

Cheers

[Telmur]

Did you unblock try to "unblock" the file, or the whole directory?

The VMware Documentation says the following

Download the Software agent bootstrap file from https://vRealize_VA_Hostname_fqdn:5480/service/software/index.html. a Download the Software bootstrap agent file https://vRealize_VA_Hostname_fqdn:5480/service/software/download/vmware-vra-software-agent-bootstrap-windows_version.zip. b Right-click the file and select Properties. c Click General. d Click Unblock. Important If you do not disable this Windows security feature, you cannot use the Software agent bootstrap file.

Prepare a Windows Reference Machine to Support Software

I hope this help...

[Dean3]

Hi Telmur,

Thanks for your reply and assistance. Yes, the files/folder are unblocked.

Strange things is, the results are inconsistent. Sometimes provisioning runs and executes all scripts fine but other times it hangs during customization either with executing customizeos.bat or finalize_noop.bat error...

No other ideas why this would occur...

[darrenoid]

Hello Dean3,

We have experienced this as well and it is very disappointing when an automation framework is not consistent. Those scripts are cobbled together and as you noticed, the logging is pretty much non existent. You may also notice a doagentsvc.exe running in task manager as well as a wait.vbs script that is recreated every 30 seconds. That bat file should be running during the CustomizeOS state or the CustomizeGeustOS state, even though in the vcac guest agent config.xml file, those states are set to false...

So far we have not been able to figure this out, but we suspect that occasionally there is a communication breakdown between the vcac guest agent service and the IaaS manager service. Once the breakdown happens and the guest agent does not receive a workitem, it never recovers. You may want to try changing the vcac guest agent service to have a delayed startup instead of automatic startup. We suspect it may be the time it takes for the virtual machines network adapter to initialize after sysprep. We have not tested this yet.

Regards,

Darrenoid

[Dean3]

Hey Darrenoid,

Thanks for your reply and assistance. Yes, it is a frustrating issue, more logging would greatly help resolve the issue..

So yes, I think you are correct. The inconsistent results suggests to me a connection issue, and yes it is possibly between the agent and the iaas manager service and the time between executing workitems.

I attempted to put delays as custom properties on the VRA build profile the vm would receive during provisioning, set them to 10mins.

-   VirtualMachine.SoftwareN.PostInstallDelay: 00:10:00

-   VirtualMachine.Admin.CustomizeOsDelay: 00:10:00

Provisioning did seem to function better, (more successful runs) however it was prone to timeout during customizeMachine stage so don't think it fixed the issue rather that it replaced it with another issue. However I opened the issue with VMWare. If I figure out anything I'll be sure to update the thread. Can you also update if you get around to testing the fix you suggested darrenoid? Would be greatly appreciated. Thanks!

Kind Regards

[proz]

I'm experiencing the same issue here... any update from support?

[darrenoid]

For me, I was never able to test my delayed startup theory. Just to update, I have not seen this issue in quite a while since updating to vRA 7.2. I also updated all my templates with the newer version of the guest agent. That may have solved it for me. I also started to pre-position the IaaS manager certificate file inside the VRMGuestAgent folder as cert.pem. Previously the guest agent would fetch it, but after the upgrade it would time out during customize os. I stumbled on this article (for vra 6) http://www.vmtocloud.com/vcac-6-linux-guest-agent-certificate-gotcha/ and decided to try it out and it fixed my issue. Perhaps that was the culprit all along, the guest agent occasionally failing to auto fetch the certificate? I don't know.

Regards,

Darrenoid