the horrors of "Guest OS customization timeout"

Hey all,

i've come across an abundance of things that can cause guest OS customization to time out. Especially if your template VMs are prepared with Guest User Agent (btw: to install Guest User Agent on windows you're best off using this script: The missing prepare_vra_template » Extending Clouds )

This is all windows guests, but might help for linux troubleshooting as well ...

typically the procedure is

a) vsphere customization is invoked -- Log A in guest OS: C:\Windows\Temp\vmware-imc\guestcust.log

b) then it tries to connect to Manager/IaaS (SSL handshake), which can be found in Log B: C:\VRMGuestAgent\axis2\logs\gugent-axis.log

c) once that succeds the agent logs here: Log C: C:\opt\vmware-appdirector\agent\logs\agent_bootstrap.log

1. if using vSphere 5.5, upgrade your ESXi to latest (at least VMware ESXi 5.5 Update 3)

in our case sometimes a machine failed to customize as it tried to join the domain. But all network connectors were disconnected on the VM at this stage, so the domain join would never happen.

In Log A: you find typically:

Its trying to connect interface

[2016-07-21T02:01:47 DEBUG] Got VMX response 'disconnected'

[2016-07-21T02:01:49 DEBUG] Trying to connect network interfaces, attempt 5

[2016-07-21T02:01:49 DEBUG] Rpci: Sending request='deployPkg.update.state 4 103 C:\Windows\TEMP\vmware-imc\guestcust.log@4000'

But fails after 5 tries:

[2016-07-21T02:01:54 ERROR] Can't connect network interfaces after 5 attempts, giving up

[2016-07-21T02:01:54 INFO] GuestCustUtil exiting.

when you try to connect the adapters manually in vCenter, this also fails:

ESXi logs similar to this:

2016-07-29T14:53:37.987Z [69CC1B70 info 'Libs' opID=70468395-37 user=vpxuser] Vix: [311413 foundryHandles.c:1526]: Error VIX_E_FAIL in Vix_AddRefHandleImpl(): Add ref a destroyed/recycled handle=-1, file=(null), line=0.

2016-07-29T14:53:37.987Z [69CC1B70 info 'Libs' opID=70468395-37 user=vpxuser] Vix: [311413 foundryHandles.c:947]: Error VIX_E_FAIL in Vix_ReleaseHandleImpl(): Releasing a recycled or destroyed handle=-1, file=(null), line=0.

2016-07-29T14:53:38.136Z [687C1B70 verbose 'Vmsvc.vm:/vmfs/volumes/54c3a8e8-15ba7ec7-2889-c81f66ec975c/guestVmName/guestVmName.vmx'] NIC: connection control message: Failed to connect virtual device ethernet0.

totally different issue but exact same logging: Changing the boot delay on a powered on Virtual Machine on ESXi 5.5 fails (2138218) | VMware KB --> Update vSphere layer, resolved the issue for me.

If you are able to connect the Networking adapters, make sure your virtual machines HW version is upodated as well as VMware Tools

2. appd.properties not found in Log C

check first Log B if the agent can actually connect to the Manager/IaaS

Problem 1:

[Fri Aug 05 05:48:53 2016] [debug] ..\..\src\core\transport\http\sender\http_transport_sender.c(246) ctx_epr:https://yourManagerFQDN:443/VMPS2

[Fri Aug 05 05:48:53 2016] [debug] ..\..\src\core\transport\http\sender\http_transport_sender.c(805) using axis2 native http sender.

[Fri Aug 05 05:48:53 2016] [debug] ..\..\src\core\transport\http\sender\http_sender.c(416) msg_ctx_id:urn:uuid:d014410f-69f5-4c19-9705-b3751d963467

[Fri Aug 05 05:48:53 2016] [error] ..\..\src\core\transport\http\sender\http_client.c(238) Error creating a socket. Most probably error returned by OS

[Fri Aug 05 05:48:53 2016] [error] ..\..\src\core\transport\http\sender\http_client.c(239) Socket Creation failed.

If you're using load-balanced vRA, check that LB is forwarding to the correct (active) manager. Best is to simply try in browser what is reported as URL in first line (in the logs above)

Problem 2:

Fri Aug 05 05:54:31 2016] [debug] ..\..\src\core\transport\http\sender\http_transport_sender.c(246) ctx_epr:https://yourManagerFQDN:443/VMPS2

[Fri Aug 05 05:54:31 2016] [debug] ..\..\src\core\transport\http\sender\http_transport_sender.c(805) using axis2 native http sender.

[Fri Aug 05 05:54:31 2016] [debug] ..\..\src\core\transport\http\sender\http_sender.c(416) msg_ctx_id:urn:uuid:4a4aaa32-7e5d-4e76-aee4-976715fbf36d

[Fri Aug 05 05:54:32 2016] [info]  [ssl client] Client certificate chain filenot specified

This indicates certificate problem, but did not make sense as the last few days deployment was ok. So I checked and the load balancer was pointing to the wrong INF server (need to check my monitoring rules)

3. Misconfigured Guest user agent

make sure to use script The missing prepare_vra_template » Extending Clouds

when running the script it requires three parameters

1. inf server where to download components. Use your infra server FQDN (not loadbalancer). if you used load balancer, simply use 1st INF server

2. IaaS server for agent communication: In loadbalanced environ,ment I use FQDN of Manager on loadbalancer. In small environment FQDN of infra server

3. Darwin password: create a new password (strong!!)

I'll probably add to it whatever else comes to mind., Please feel free to comment I'll modify accordingly. Those problems have taken too much time of my precious life ...

0 Kudos
3 Replies

Meanwhile, vRealize Automation 7.2 keeps eating my Precious Life Time:

I'm deploying Windows 2012 Blueprint with an addidtional Powershell script in the form of "Software Component".

I have configured the windows Template exactly as mentioned in the Documentation.

The VM is ready..However the powershell command has not yet been executed.

This is the situation of Logs...

  • C:\VRMGuestAgent\axis2\   "Logs" folder has not been created   AND......

  • INFO 2017-04-04 16:23:54,067 - C:\opt\vmware-appdirector\agent\appd.properties not detected.

     INFO 2017-04-04 16:23:54,067 - Wait for 30 seconds before retry ...

     INFO 2017-04-04 16:24:24,042 - C:\opt\vmware-appdirector\agent\appd.properties not detected.

     INFO 2017-04-04 16:24:24,042 - Wait for 30 seconds before retry ...

What could i be missing???

I did not use the script to prepare the template, since it's not built for 7.2

0 Kudos

I would start with re-installing the agent on the Windows machine.  If I am not mistaken the powershell script is available on 7.2  https://cafe-appliance:5480/software/  Also did you create the definitions needed for the scripts to be called?

Steve Beaver
VMware Communities User Moderator
VMware vExpert 2009 - 2020
VMware NSX vExpert - 2019 - 2020
Co-Author of "VMware ESX Essentials in the Virtual Data Center"
(ISBN:1420070274) from Auerbach
Come check out my blog: [www.virtualizationpractice.com/blog|http://www.virtualizationpractice.com/blog/]
Come follow me on twitter http://www.twitter.com/sbeaver

**The Cloud is a journey, not a project.**
0 Kudos
VMware Employee
VMware Employee

Only prep  the templates with the prepare_vra_template.ps1 and .sh commands respectively for guest agent and SW agent support as identified here:

vRealize Automation 7.2 Information Center

Any manual installation will most likely lead to errors due to a missed step.  Reprep this template as per above pub.

0 Kudos