Strange Customize Machine Timeout for Windows 10 vRA 7.4

[darrenoid]

Hello vRA Community,

We are noticing a strange issue with Windows 10 VM provisioning in vRA 7.4. The deployment times out during the CustomizeMachine state (2 hour timeout). We have also noticed that if we logon during the time the VM is hung at the CustomizeMachine state, but has completed sysprep, the deployment continues to the next state. As far as I know this state is where vmware guest customization is running for sysprep and changing the hostname, etc.

How does vRA talk to the VM to tell if vmware OS guest customization has completed? Are there any logs that will show the communication? How would logging on to the VM trigger the deployment to continue?

What I have verified so far is:

1. Guest customization is finishing completely without issues. The vmware tools log shows this and the VM is joined to the domain and the hostname is changed

2. I am using the latest vmware tools and vra guest agent for this environment

3. I have checked the vra guest agent log, but it has nothing to show. I don't think the guest agent is involved in this part of the process. It just keeps polling the vRA manager service for a work item.

4. I don't see any info in the IaaS server logs.

5. Logging onto the machine seems to trigger the deployment to continue

6. Other OSs (Server 2016, Windows 8.1, Windows 7, Server 2012R2) deploy fine using the same template creation process and OS Guest Customization spec in vcenter.

My environment:

VCenter 6.7
vRA 7.4 distributed deployment with embedded vRO

Windows 10 Build 10.0.17134

VMWare Tools Build 10.2.1

vRA Guest Agent Build 7.4.0.13461

Regards,
Darren

[daphnissov]

Can you explain the blueprint on which this is based? What functions are you invoking which require the guest agent?

[darrenoid]

I spoke too soon. This may not be vRA related. A closer look at the guestcust.log in C:\Windows\Temp\vmware-imc\ and I can see a delay in the timestamps starting from right after the sysprep folder is removed to when I logged in manually 47 minutes later. Here is a snippet showing the 47 minute time gap:

[2018-07-22T19:04:05:                              :   INFO] GuestCustUtil invoked.

[2018-07-22T19:04:05:                              :   INFO] Params: deleteContainingFolder

[2018-07-22T19:04:05:                              :  DEBUG] Registry value 'CustomRescheduleFlags' is not set. Returning default value: 0

[2018-07-22T19:04:05:                              :   INFO] Checking for Windows Sysprep Respecialize operation.

[2018-07-22T19:04:05:                              :  DEBUG] Sysprep process is not running.

[2018-07-22T19:04:05:                              :   INFO] Windows Sysprep Respecialize operation was not detected.

[2018-07-22T19:04:05:                              :  DEBUG] Deleting "C:\sysprep" ...

[2018-07-22T19:04:05:                              :  DEBUG] Deleted folder "C:\sysprep" .

[2018-07-22T19:04:05:                              :   INFO] GuestCustUtil exiting.

[2018-07-22T19:51:53:                              :   INFO] GuestCustUtil invoked.

[2018-07-22T19:51:53:                              :   INFO] Params: post-run HandleCustomizationCompleted<\|/>

[2018-07-22T19:51:53:                              :  DEBUG] Registry value 'CustomRescheduleFlags' is not set. Returning default value: 0

[2018-07-22T19:51:53:                              :   INFO] Checking for Windows Sysprep Respecialize operation.

[2018-07-22T19:51:53:                              :  DEBUG] Sysprep process is not running.

[2018-07-22T19:51:53:                              :   INFO] Windows Sysprep Respecialize operation was not detected.

[2018-07-22T19:51:53:                              :   INFO] Post-customization run operation: script 'HandleCustomizationCompleted<\|/>'

[2018-07-22T19:51:53:                              :   INFO] post-run HandleCustomizationCompleted

[2018-07-22T19:51:53:                              :   INFO] Deleting post-customization task to avoid re-execution

[2018-07-22T19:51:53:     WindowsOsAbstractionLayer:   INFO] Executing command: 'C:\windows\system32\schtasks.exe /query /tn "VMware Guest OS Customization Task"'

[2018-07-22T19:51:53:     WindowsOsAbstractionLayer:   INFO] Command execution succeeded, output: '

Besides the time gap, the log looks the same as one from a successful deployment. It looks like sysprep is finishing properly, but some remaining guest customization tasks are not completing until someone logs in.

To answer your question daphnissov, my blueprint relies on the guest agent to run a few InstallSoftware commands.

Regards,

Darren

[daphnissov]

Is this a clone workflow (from a vSphere template)?

[darrenoid]

Yes, it's a clone workflow from a vsphere template.

[daphnissov]

If it's a clone (and you're using a vCenter customization spec), are you invoking sysprep programmatically? Or did you configure your template to generalize OOTB and shutdown?

[darrenoid]

Sysprep is running as part of the guest customization process as per the vcenter guest customization specification. We do not manually invoke sysprep.

[darrenoid]

I was able to get vra to complete the deployment as a workaround by disabling waiting for guest to complete with the following property:

VirtualMachine.Customize.WaitComplete

set to False

I also added VirtualMachine.Admin.CustomizeGuestOSDelay set to 00:10:00 for a 10 minute delay. This delay does seem to halt the guest agent between the CustomizeOS state and InstallSoftware state for 10 minutes, allowing the guest customization time to complete sysprep.

The issue remains that until someone logs in guest customization won't fully complete. I have a vsphere support ticket open for this to see what they say about it.

Thanks,

Darren

[KThorlund]

Hi Darren, 

did you ever find out what was causing the issue? I also have my Windows 10 machines hanging never completing..

Regards, K

[Dell_Technologi]

Im having the same iisue but with Ubuntu 16.04

Anyone who figured it out ?

[darrenoid]

After looking at it VMWare support suggested changing the guest customization specification to have an auto logon count of at least 1. Something in Windows was not allowing guest customization to complete unless someone was actively logged in. We never tried it out to see as the issue went away when we updated our Windows 10 templates to version 1809. It seems this issue was only happening on Windows 10 version 1803.

[LeroyJenkins608]

darrenoid - did you ever test this out with other versions of Win10?

I'm seeing this same behavior with Win10 1903. Guest Customization halts until someone logs in. I even have the Customization Spec with an autologin count of 1 like you suggested, but one still has to login for it to finish.

[darrenoid]

Hey LeroyJenkins608,

I have not seen this issue past version 1803. We have recently added version 1903 and I am not seeing the issue either. Specific patch version for our Windows 10 1903 template is 10.0.18362.267.

Regards,

Darren