Issues with PowerCLI and New-OSCustomizationSpec with Windows 10

[JBartholomew]

I am in the midst of trying to write a script to further our VM Creation Automation, and have so far managed to modify my original VM Creation script to use OSCustomizeSpec's to do more of the manual work (Domain Join, Set IP) for Windows 2012/2016/2019 VMs. Unfortunately I am running into a roadblock with Windows 10, and am curious if anyone else has run into this problem before.

So Currently the OSCustomizeSpec is as follows:

If($Global:VMOS -like "*Windows 10*") {
New-OSCustomizationSpec -Name "OSCustomSpec" -FullName Administrator -AdminPassword "*************" -Domain $Domain -DomainCredentials $Global:Credentials -OrgName "Company Name" -OSType "Windows" -Type NonPersistent -ChangeSid -NamingScheme Fixed -NamingPrefix $VMName -TimeZone $TimeZone -AutoLogonCount 1 | Out-Null
Get-OSCustomizationNicMapping -OSCustomizationSpec "OSCustomSpec" | Set-OSCustomizationNicMapping -IpMode UseStaticIP -IpAddress $IP -SubnetMask "255.255.255.0" -DefaultGateway $IPGateway -Dns $DNS | Out-Null
$Global:OSCustomization = Get-OSCustomizationSpec "OSCustomSpec"

}

When I use this OSCustomize to deploy a Windows Server based Template (2012/2016/2019) Everything is as expected. The VM gets renamed to $VMName, The Domain $Domain is joined, and the IP Address is set to static with the correct IP ($IP), gateway and DNS ($DNS)

Now when I use this to deploy a Windows 10 Template (I have tried 1909, 20H2,21H2) Something really odd happens. There is a computer object created in the domain with the correct $VMName, and when I log into the VM, the IP Address is set to static with the correct information, but for some reason the Computer itself is named a Default name 'Computer-Randomletters' and is in a workgroup named after the domain. If I try to manually join the domain once the VM is booted, it succeeds without issue

The only event relating to this in the VMWare Event log is: 'Started Customization for the VM .... ' No Errors indicated in the VMWare log.

Below is the C:\Windows\Temp\VMWare-Imc\guestcust file

[2022-01-12T05:32:27: : INFO] GuestCustUtil invoked.
[2022-01-12T05:32:27: : INFO] Params: customize -sealparam "/quiet /generalize /oobe /reboot" -nics 4000 -schedulenativeunobfusc
[2022-01-12T05:32:27: : DEBUG] Registry value 'CustomRescheduleFlags' is not set. Returning default value: 0
[2022-01-12T05:32:27: : INFO] Checking for Windows Sysprep Respecialize operation.
[2022-01-12T05:32:27: : DEBUG] Sysprep process is not running.
[2022-01-12T05:32:27: : INFO] Windows Sysprep Respecialize operation was not detected.
[2022-01-12T05:32:27: GuestCustUtil: DEBUG] Deleting status marker file C:/Windows/.post-gc-status
[2022-01-12T05:32:27: GuestCustUtil: DEBUG] Status marker file C:/Windows/.post-gc-status doesn't exist
[2022-01-12T05:32:27: : INFO] Extended customization is not enabled!
[2022-01-12T05:32:27: : DEBUG] HandleCustomizationStarted
[2022-01-12T05:32:27: : DEBUG] Customization in progress set to 1 at 2022-Jan-12 05:32:27
[2022-01-12T05:32:27: : DEBUG] GOScPostRebootStatus feature is enabled
[2022-01-12T05:32:27: GuestCustUtil: DEBUG] SetVmxProperty: key-'guestinfo.gc.notification', value-'Started'
[2022-01-12T05:32:27: WindowsOsAbstractionLayer: INFO] VMware tools installation path=C:\Program Files\VMware\VMware Tools\
[2022-01-12T05:32:27: WindowsOsAbstractionLayer: INFO] Executing command: '"C:\Program Files\VMware\VMware Tools\vmtoolsd.exe" --cmd "info-set guestinfo.gc.notification Started"'
[2022-01-12T05:32:27: WindowsOsAbstractionLayer: INFO] Command execution succeeded, output: '

'
[2022-01-12T05:32:27: : DEBUG] Rpci: Sending request='deployPkg.update.state 4 0 C:/Windows/TEMP/vmware-imc/guestcust.log'

[2022-01-12T05:32:27: : DEBUG] Rpci: Sent request='deployPkg.update.state 4 0 C:/Windows/TEMP/vmware-imc/guestcust.log', reply='', len=0, status=1

[2022-01-12T05:32:27: : DEBUG] Temporary sysprep dir path: C:\sysprep
[2022-01-12T05:32:27: : DEBUG] Copying file/directory from 'sysprep' to 'C:\sysprep'
[2022-01-12T05:32:27: : INFO] Sysprep File is C:\Windows\Panther\unattend.xml
[2022-01-12T05:32:27: : INFO] Password unobfuscation in progress
[2022-01-12T05:32:27: : DEBUG] Decrypting passwords
[2022-01-12T05:32:27: : DEBUG] Password found and decrypted for <AdministratorPassword><EncryptedValue>
[2022-01-12T05:32:27: : DEBUG] Password found and decrypted for <Credentials><EncryptedPassword>
[2022-01-12T05:32:27: : DEBUG] Password found and decrypted for <Password><Value>
[2022-01-12T05:32:27: : DEBUG] Tag -de not found after <RunSynchronousCommand
[2022-01-12T05:32:27: : INFO] Password unobfuscation successful
[2022-01-12T05:32:27: : DEBUG] Get current computer name: WIN-KJJ06GR317B
[2022-01-12T05:32:28: : DEBUG] select * from win32_networkadapter where Manufacturer != 'Microsoft' and ServiceName != 'VMnetAdapter' and manufacturer is not null and MACAddress is not null
[2022-01-12T05:32:28: : DEBUG] Found 1 objects. Pointer 666210. return code 0(0x0)
[2022-01-12T05:32:28: : DEBUG] Found 0 objects. Pointer 0. return code 1(0x1)
[2022-01-12T05:32:28: : DEBUG] Returning value 00:50:56:8A:F4:D4 for system property
[2022-01-12T05:32:28: : DEBUG] Setting dhcp for nic # 0
[2022-01-12T05:32:28: : DEBUG] Returning value \\WIN-KJJ06GR317B\ROOT\CIMV2:Win32_NetworkAdapter.DeviceID="1" for system property
[2022-01-12T05:32:28: : DEBUG] ASSOCIATORS OF {\\WIN-KJJ06GR317B\ROOT\CIMV2:Win32_NetworkAdapter.DeviceID="1"} where ResultClass = Win32_NetworkAdapterConfiguration
[2022-01-12T05:32:28: : DEBUG] Found 1 objects. Pointer 6643e0. return code 0(0x0)
[2022-01-12T05:32:28: : DEBUG] Found 0 objects. Pointer 0. return code 1(0x1)
[2022-01-12T05:32:28: : DEBUG] Clearing gateway ip addresses.
[2022-01-12T05:32:28: : DEBUG] Enabling DHCP on the computer
[2022-01-12T05:32:28: : DEBUG] Returning value \\WIN-KJJ06GR317B\ROOT\CIMV2:Win32_NetworkAdapterConfiguration.Index=1 for system property
[2022-01-12T05:32:28: : DEBUG] Setting DNS Server to ip
[2022-01-12T05:32:28: : DEBUG] Resetting NIC's static IPv6 settings...
[2022-01-12T05:32:28: Netsh: DEBUG] Fetching adapters information...
[2022-01-12T05:32:28: Netsh: DEBUG] Interface 6 is found
[2022-01-12T05:32:28: Netsh: DEBUG] Looking IPv6 Unicast addresses for interface 6
[2022-01-12T05:32:28: Netsh: DEBUG] Found address: fe80::bcee:1d5e:7dc4:ad34
[2022-01-12T05:32:28: Netsh: DEBUG] Interface 6 is found
[2022-01-12T05:32:28: Netsh: DEBUG] Looking IPv6 Gateways for interface 6
[2022-01-12T05:32:28: : DEBUG] Returning value Ethernet0 for system property
[2022-01-12T05:32:28: Netsh: INFO] Deleting all IPv6 DNS Servers
[2022-01-12T05:32:28: WindowsOsAbstractionLayer: INFO] Executing command: 'netsh int ipv6 delete dnsserver name="Ethernet0" all'
[2022-01-12T05:32:32: WindowsOsAbstractionLayer: INFO] Command execution succeeded, output: '

There are no Domain Name Servers (DNS) configured on this computer.

'
[2022-01-12T05:32:32: : WARN] Resetting NIC's static IPv6 settings has succeeded
[2022-01-12T05:32:32: SysprepDriveLettersWorkaround: DEBUG] Backing up HKLM MountedDevices before sysprep runs
[2022-01-12T05:32:32: : DEBUG] Enabling SE_BACKUP_NAME privilege
[2022-01-12T05:32:32: : DEBUG] Retrieving process access token for SeBackupPrivilege
[2022-01-12T05:32:32: : DEBUG] Getting LUID
[2022-01-12T05:32:32: : DEBUG] Adjusting privileges
[2022-01-12T05:32:32: : DEBUG] Backing up SYSTEM\MountedDevices to C:\sysprep\MountedDevicesBackup
[2022-01-12T05:32:32: : DEBUG] Executing command C:\windows\system32\sysprep\sysprep.exe /quiet /generalize /oobe /reboot /unattend:C:\Windows\Panther\unattend.xml

[2022-01-12T05:32:32: : DEBUG] Successfully executed command C:\windows\system32\sysprep\sysprep.exe /quiet /generalize /oobe /reboot /unattend:C:\Windows\Panther\unattend.xml

[2022-01-12T05:32:32: : DEBUG] Trying to connect network interfaces, attempt 1
[2022-01-12T05:32:32: : DEBUG] Rpci: Sending request='deployPkg.update.state 4 103 C:/Windows/TEMP/vmware-imc/guestcust.log@4000'

[2022-01-12T05:32:32: : DEBUG] Rpci: Sent request='deployPkg.update.state 4 103 C:/Windows/TEMP/vmware-imc/guestcust.log@4000', reply='queryNicsSupported', len=18, status=1

[2022-01-12T05:32:32: GuestCustUtilLib: DEBUG] Got VMX response 'queryNicsSupported'
[2022-01-12T05:32:32: : DEBUG] Rpci: Sending request='deployPkg.update.state 4 104 C:/Windows/TEMP/vmware-imc/guestcust.log@4000'

[2022-01-12T05:32:32: : DEBUG] Rpci: Sent request='deployPkg.update.state 4 104 C:/Windows/TEMP/vmware-imc/guestcust.log@4000', reply='connected', len=9, status=1

[2022-01-12T05:32:32: GuestCustUtilLib: DEBUG] Got VMX response 'connected'
[2022-01-12T05:32:32: : INFO] The network interfaces are connected on 0 second
[2022-01-12T05:32:32: : INFO] GuestCustUtil exiting.

[2022-01-12T05:34:01: : INFO] GuestCustUtil invoked.
[2022-01-12T05:34:01: : INFO] Params: restoreMountedDevices
[2022-01-12T05:34:01: : DEBUG] Registry value 'CustomRescheduleFlags' is not set. Returning default value: 0
[2022-01-12T05:34:01: : INFO] Checking for Windows Sysprep Respecialize operation.
[2022-01-12T05:34:01: : DEBUG] Sysprep process is currently running.
[2022-01-12T05:34:01: : INFO] Windows Sysprep Respecialize operation was not detected.
[2022-01-12T05:34:01: SysprepDriveLettersWorkaround: DEBUG] Restoring HKLM MountedDevices to fix sysprep issue
[2022-01-12T05:34:01: : DEBUG] Enabling SE_RESTORE_NAME privilege
[2022-01-12T05:34:01: : DEBUG] Retrieving process access token for SeRestorePrivilege
[2022-01-12T05:34:01: : DEBUG] Getting LUID
[2022-01-12T05:34:01: : DEBUG] Adjusting privileges
[2022-01-12T05:34:01: : DEBUG] Restoring SYSTEM\MountedDevices from C:\sysprep\MountedDevicesBackup
[2022-01-12T05:34:01: : INFO] GuestCustUtil exiting.

[2022-01-12T05:34:01: : INFO] GuestCustUtil invoked.
[2022-01-12T05:34:01: : INFO] Params: flagComplete
[2022-01-12T05:34:01: : DEBUG] Registry value 'CustomRescheduleFlags' is not set. Returning default value: 0
[2022-01-12T05:34:01: : INFO] Checking for Windows Sysprep Respecialize operation.
[2022-01-12T05:34:01: : DEBUG] Sysprep process is currently running.
[2022-01-12T05:34:01: : INFO] Windows Sysprep Respecialize operation was not detected.
[2022-01-12T05:34:01: GuestCustUtil: DEBUG] Status marker file C:/Windows/.post-gc-status doesn't exist
[2022-01-12T05:34:01: WindowsOsAbstractionLayer: INFO] Executing command: 'C:\Windows\system32\schtasks.exe /query /tn "VMware Guest OS Customization Task"'
[2022-01-12T05:34:01: WindowsOsAbstractionLayer: WARN] Command execution failed with exit code: 1, output: 'ERROR: The system cannot find the file specified.

'
[2022-01-12T05:34:01: GuestCustUtil: DEBUG] SchedulePostCustomizationActivity: Run Command-'HandleCustomizationCompleted<\|/>'
[2022-01-12T05:34:01: GuestCustUtil: INFO] Scheduling post-customization activity to run after reboot...
[2022-01-12T05:34:01: WindowsOsAbstractionLayer: INFO] Executing command: 'C:\Windows\system32\schtasks.exe /query /tn "VMware Guest OS Customization Task"'
[2022-01-12T05:34:01: WindowsOsAbstractionLayer: WARN] Command execution failed with exit code: 1, output: 'ERROR: The system cannot find the file specified.

'
[2022-01-12T05:34:01: GuestCustUtil: INFO] Task 'VMware Guest OS Customization Task' exists and will be deleted: false
[2022-01-12T05:34:01: GuestCustUtil: DEBUG] Creating sysprep copy in dir:'C:\sysprep1004'
[2022-01-12T05:34:01: GuestCustUtil: DEBUG] Artifact full path: C:\sysprep\libeay32.dll
[2022-01-12T05:34:01: GuestCustUtil: INFO] Copying C:\sysprep\libeay32.dll to C:\sysprep1004.
[2022-01-12T05:34:01: : DEBUG] Copying file/directory from 'C:\sysprep\libeay32.dll' to 'C:\sysprep1004'
[2022-01-12T05:34:01: GuestCustUtil: DEBUG] Artifact full path: C:\sysprep\ssleay32.dll
[2022-01-12T05:34:01: GuestCustUtil: INFO] Copying C:\sysprep\ssleay32.dll to C:\sysprep1004.
[2022-01-12T05:34:01: : DEBUG] Copying file/directory from 'C:\sysprep\ssleay32.dll' to 'C:\sysprep1004'
[2022-01-12T05:34:01: GuestCustUtil: DEBUG] Artifact full path: C:\sysprep\guestcustutil.exe
[2022-01-12T05:34:01: GuestCustUtil: INFO] Copying C:\sysprep\guestcustutil.exe to C:\sysprep1004.
[2022-01-12T05:34:01: : DEBUG] Copying file/directory from 'C:\sysprep\guestcustutil.exe' to 'C:\sysprep1004'
[2022-01-12T05:34:01: GuestCustUtil: DEBUG] Schedule command: 'C:\sysprep1004\guestcustutil.exe post-run HandleCustomizationCompleted<\|/>'
[2022-01-12T05:34:01: : INFO] Scheduling 'VMware Guest OS Customization Task'
[2022-01-12T05:34:01: : DEBUG] Executing command C:\Windows\system32\schtasks.exe /create /sc ONSTART /rl HIGHEST /np /z /v1 /ru SYSTEM /tr "C:\sysprep1004\guestcustutil.exe post-run HandleCustomizationCompleted<\|/>" /tn "VMware Guest OS Customization Task"
[2022-01-12T05:34:02: GuestCustUtil: INFO] Post-customization activity successfully scheduled.
[2022-01-12T05:34:02: : INFO] GuestCustUtil exiting.

[2022-01-12T05:34:02: : INFO] GuestCustUtil invoked.
[2022-01-12T05:34:02: : INFO] Params: deleteContainingFolder
[2022-01-12T05:34:02: : DEBUG] Registry value 'CustomRescheduleFlags' is not set. Returning default value: 0
[2022-01-12T05:34:02: : INFO] Checking for Windows Sysprep Respecialize operation.
[2022-01-12T05:34:02: : DEBUG] Sysprep process is currently running.
[2022-01-12T05:34:02: : INFO] Windows Sysprep Respecialize operation was not detected.
[2022-01-12T05:34:02: : DEBUG] Deleting "C:\sysprep" ...
[2022-01-12T05:34:02: : DEBUG] Succeeded to delay deletion of folder "C:\sysprep" upon reboot.
[2022-01-12T05:34:02: : INFO] GuestCustUtil exiting.

[2022-01-12T08:36:10: : INFO] GuestCustUtil invoked.
[2022-01-12T08:36:10: : INFO] Params: post-run HandleCustomizationCompleted<\|/>
[2022-01-12T08:36:10: : DEBUG] Registry value 'CustomRescheduleFlags' is not set. Returning default value: 0
[2022-01-12T08:36:10: : INFO] Checking for Windows Sysprep Respecialize operation.
[2022-01-12T08:36:10: : DEBUG] Sysprep process is not running.
[2022-01-12T08:36:10: : INFO] Windows Sysprep Respecialize operation was not detected.
[2022-01-12T08:36:10: : INFO] Post-customization run operation: script 'HandleCustomizationCompleted<\|/>'
[2022-01-12T08:36:10: : INFO] post-run HandleCustomizationCompleted
[2022-01-12T08:36:10: : INFO] Deleting post-customization task to avoid re-execution
[2022-01-12T08:36:10: WindowsOsAbstractionLayer: INFO] Executing command: 'C:\Windows\system32\schtasks.exe /query /tn "VMware Guest OS Customization Task"'
[2022-01-12T08:36:11: WindowsOsAbstractionLayer: INFO] Command execution succeeded, output: '

Folder: \

TaskName Next Run Time Status

======================================== ====================== ===============

VMware Guest OS Customization Task N/A Running

'
[2022-01-12T08:36:11: GuestCustUtil: INFO] Task 'VMware Guest OS Customization Task' exists and will be deleted: true
[2022-01-12T08:36:11: : INFO] Deleting 'VMware Guest OS Customization Task'
[2022-01-12T08:36:11: : DEBUG] Executing command C:\Windows\system32\schtasks.exe /delete /f /tn "VMware Guest OS Customization Task"
[2022-01-12T08:36:11: : DEBUG] HandleCustomizationCompleted
[2022-01-12T08:36:11: : DEBUG] Customization in progress set to 0 at 2022-Jan-12 08:36:11
[2022-01-12T08:36:11: : INFO] Cleaning up obfuscation details from registry
[2022-01-12T08:36:11: : DEBUG] GOScPostRebootStatus feature is enabled
[2022-01-12T08:36:11: GuestCustUtil: DEBUG] SetVmxProperty: key-'guestinfo.gc.notification', value-'Completed'
[2022-01-12T08:36:11: WindowsOsAbstractionLayer: INFO] VMware tools installation path=C:\Program Files\VMware\VMware Tools\
[2022-01-12T08:36:11: WindowsOsAbstractionLayer: INFO] Executing command: '"C:\Program Files\VMware\VMware Tools\vmtoolsd.exe" --cmd "info-set guestinfo.gc.notification Completed"'
[2022-01-12T08:36:11: WindowsOsAbstractionLayer: INFO] Command execution succeeded, output: '

'
[2022-01-12T08:36:11: : DEBUG] Rpci: Sending request='deployPkg.update.state 5 0 C:/Windows/TEMP/vmware-imc/guestcust.log'

[2022-01-12T08:36:11: : DEBUG] Rpci: Sent request='deployPkg.update.state 5 0 C:/Windows/TEMP/vmware-imc/guestcust.log', reply='', len=0, status=1

[2022-01-12T08:36:11: : DEBUG] Deleting "C:\sysprep1004" ...
[2022-01-12T08:36:11: : DEBUG] Succeeded to delay deletion of folder "C:\sysprep1004" upon reboot.
[2022-01-12T08:36:11: : INFO] GuestCustUtil exiting.

There is nothing in the Setup Event Log, and in the System event log I only get these cryptic messages:

Log Name: System
Source: Service Control Manager
Date: 1/12/2022 8:34:13 AM
Event ID: 7001

The Network Connectivity Assistant service depends on the IP Helper service which failed to start because of the following error:
The operation completed successfully.

and this really weird one referencing both the expected (LCOA-LBIF-WTE3) and the actual Computer names

Log Name: System
Source: User32
Date: 1/12/2022 8:34:14 AM
Event ID: 1074
Task Category: None
Level: Information
Keywords: Classic
User: SYSTEM
Computer: WIN-KJJ06GR317B
Description:
The process C:\Windows\system32\winlogon.exe (DESKTOP-U2ASQSA) has initiated the restart of computer LCOA-LBIF-WTE3 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Upgrade (Planned)
Reason Code: 0x80020003
Shutdown Type: restart

Has anyone come across these issues before when using OSCustomizationSpec's with Windows 10?

[LucD]

How did you specify the User in 

$Global:Credentials

That should be in the format user@domain, not domain/user

[JBartholomew]

The $Global:Credentials is a get-credential, also used in the actual login to the vCenter and is being passed as User@Domain

I just find it odd that the OSCustomization is working fine for any server based Template, but as soon as I throw it at a Client OS it starts behaving odd.

We are running vSphere 6.7 Update 3o with Power CLI  Version 12.4 (In case this is a weird API interoperability issue between vSphere and Active Directory and how Windows 10 does Domain Joins.

[LucD]

You could check on the DCs if you see the domain join attempt coming in.

[JBartholomew]

Looking for Event ID 4741 on theDCs, I do see an entry for the VM, and I can also see a computer object in the Computers OU of AD. So the script is correctly creating a computer object in AD, but the VM itself does not get renamed or join the Domain correctly.

My thought is it is something in the Computer rename that is causing issues because if the Computer is still named 'Desktop-randomcharacters' then it wont authenticate to the domain, as the computer name does not match the Computer object in AD

[LucD]

Is by any chance the name of the Template longer than 15 characters?
See Solved: OS Customization error deploying Server 2019 Core ... - VMware Technology Network VMTN

[JBartholomew]

Yes the template name is longer than 15 Characters, but the template names for the 2012/2016/2019 templates are also over 15 characters as well

[LucD]

I think you better open an SR

[kwhornlcs]

Was looking to test this and had issues with a clone of Windows 10 on our environment that may be related. If you manually clone the Windows 10 template to a new VM, then try to sysprep and generalize that new VM to OOBE, are you successful or does it error?

Apparently there can be some issues with application installs to one but not all profiles that can prevent sysprep from customizing the OS.

Some references that are similar to what I saw, and you're describing:

https://kb.vmware.com/s/article/82532

https://communities.vmware.com/t5/VMware-vSphere-Discussions/Guest-Customization-Windows-10-1803-sta...

[JBartholomew]

After some further investigation the issue most definitely appears to be a Windows 10 based issue and not anything to do with the script at all. Looking in the C:\Windows\System32\Sysprep\Panther\setuperr.log there are a number of entries that from a quick look all appear to be related to issues with Windows 10 and the Windows Store. 

2022-01-20 16:51:00, Error TOOL BCD: BiUpdateEfiEntry failed c000000d
2022-01-20 16:51:00, Error TOOL BCD: BiExportBcdObjects failed c000000d
2022-01-20 16:51:00, Error TOOL BCD: BiExportStoreAlterationsToEfi failed c000000d
2022-01-20 16:51:00, Error TOOL BCD: Failed to export alterations to firmware. Status: c000000d

I have not resolved these issues yet, but as the issues are not script related I will be closing this post

[LucD]

Apparently, this is a known Windows 10 issue.
Seems the Windows Store Update needs to be stopped.
See Windows 10 sysprep - Microsoft Q&A

[JBartholomew]

Yes. This is the same articles I have been seeing. In our case stopping the service via the script did not resolve the issue, but I think it has to do more with our networking as we have GPOs in place in our domain to route to our internal WSUS server, and it has caused issues in the past with Windows store things (Installing RSAT, etc).

I am trying a new VM on an isolated vmworkstation VM to see if being outside the network will make it happier