Servers joining domain through customization spec in one script, but not another

[JLogan2016]

I have a build script that has been working for some time at a company I contract to. The script takes input from a csv, and builds the server based on that info. I have just started working on another script; the only difference being that instead of a tech entering data on the CSV a requester enters it on a sharepoint form which is then saved to csv. Other than a couple of aesthetic changes, there is no seeming difference between the two. However, while both scripts build the server successfully, the new one will not join the domain during customization. What's worse, it doesn't appear to even try; I've encountered in the past where someone put in the wrong IP or Gateway and the customization just sits there spinning as it tries/fails/retries, upwards of 10 minutes. But on this new script it just blips by after the reboots.

Below is the function that builds the VM:

function _buildVM {
    [CmdLetBinding()]
    Param (
        [Parameter(Mandatory, Position=1)]
            [string] $newVM,
        [Parameter(Mandatory, Position=2)]
            [string] $vCPU,
        [Parameter(Mandatory, Position=3)]
            [string] $RamGB,
        [Parameter(Mandatory, Position=4)]
            [string] $IP,
        [Parameter(Mandatory, Position=5)]
            [string] $destFolder,
        [Parameter(Mandatory, Position=6)]
            [string] $Desc,
        [Parameter(Mandatory, Position=7)]
            [string] $Cluster,
        [Parameter(Mandatory, Position=8)]
            [string] $dStore,
        [Parameter(Mandatory, Position=9)]
            [string] $OS,
        [Parameter(Mandatory, Position=10)]
            [string] $Location,
        [Parameter(Mandatory, Position=11)]
            [string] $Domain,
        [Parameter(Mandatory, Position=12)]
            [AllowEmptyString()]
            [string] $drive1Size,
        [Parameter(Mandatory, Position=13)]
            [AllowEmptyString()]
            [string] $drive2Size,
        [Parameter(Mandatory, Position=14)]
            [AllowEmptyString()]
            [string] $drive3Size,
        [Parameter(Mandatory, Position=15)]
            [AllowEmptyString()]
            [string] $drive4Size,
        [Parameter(Mandatory, Position=16)]
            [AllowEmptyString()]
            [string] $drive5Size
    )
    $PortGroup = ""
    $netMask = "0.0.0.0"
    $Gateway = "0.0.0.0"
    $DNS1 = "0.0.0.0"
    $DNS2 = "0.0.0.0"
    switch ($Domain) {
        "Sub1" {
            $sDomain = "sub1.company.com"
        }
        "Parent" {
            $sDomain = "parent.company.com"
        }
        "Sub2" {
            $sDomain = "sub2.company.com"
        }
    }
    switch ($OS) {
        {$_ -match "2012"} {
            $template = "WS_2012_R2_DE"
            $customSpec = "W2k12R2"
        }
        {$_ -match "2016"} {
            $template = "WS_2016_DE"
            $customSpec = "W2k16"
        }
        {$_ -match "ubuntu"} {
            $template = "ubuntu-16-minimal"
            $customSpec = "Linux v1"
        }
        {$_ -match "centos"} {
            $template = "centos-7-minimal"
            $customSpec = "Linux v1"
        }
        {$_ -match "RHEL"} {
            $template = "rhel-7-minimal"
            $customSpec = "Linux v1"
        }
    }
#region Determine vCenter to point to=============================
    switch ($Location) {
        "SanFrancisco" {
            $vCenter = "SF0001"
            if ($Domain -match "Sub1") {
                $PortGroup = "dvPortGroup-CX-VLAN-22-172-24-16-0-21"
                $netMask = "255.255.248.0"
                $Gateway = "172.24.16.6"
                $DNS1 = "172.24.16.165"
                $DNS2 = "172.24.16.166"
            } else {
                $PortGroup = "dvPortGroup-CX-VLAN-55-172-24-56-0-23"
                $netMask = "255.255.254.0"
                $Gateway = "172.24.56.1"
                $DNS1 = "172.24.56.20"
                $DNS2 = "172.24.56.21"
            }
            Write-Log -Path $LogFile -Message "     Building $($newVM) in San Francisco on VLAN $($PortGroup.Split("-")[3])"  -Component "VM Build" -Type Info
        }
        "Detroit" {
            $vCenter = "DET0001"
            if ($Domain -match "Sub1") {
                $PortGroup = "dvPortGroup-CX-VLAN-22-172-24-160-0-21"
                $netMask = "255.255.248.0"
                $Gateway = "172.24.160.1"
                $DNS1 = "172.24.160.71"
                $DNS2 = "172.24.160.72"
            } else {
                $PortGroup = "dvPortGroup-CX-VLAN-155-172-24-156-0-23"
                $netMask = "255.255.254.0"
                $Gateway = "172.24.156.1"
                $DNS1 = "172.24.156.20"
                $DNS2 = "172.24.156.21"
            }
            Write-Log -Path $LogFile -Message "     Building $($newVM) in Detroit on VLAN $($PortGroup.Split("-")[3])"  -Component "VM Build" -Type Info
        }
    }
#endregion Determine vCenter to point to=============================
#region Check Datastore===================================
    try {
        $myDataStore = Get-Datastore -Name $dStore -Server $vCenter -ErrorAction Stop
    } catch [Exception] {
        $aLine = _getlowestProvisionedPercent -DatastoreCluster $dStore | Sort-Object Provisioned%
        $myDataStore = Get-Datastore -Name $aLine[0].Datastore -Server $vCenter
    }
    $iTotalDiskSpace = (70 + [int]$drive1Size + [int]$drive2Size + [int]$drive3Size + [int]$drive4Size + [int]$drive5Size)
        if ($myDataStore.FreeSpaceGB -lt ($iTotalDiskSpace + 20)) {
            return "DataStoreFail"
        } else {
            Write-Log -Path $LogFile -Message "     Datastore will be $($myDataStore.Name)"  -Component "VM Build" -Type Info
        }
#endregion Check Datastore==============================================================================================================
#region Copy Customization Spec================
    try {
        Get-OSCustomizationSpec -Name $customSpec -Server $vCenter -ErrorAction Stop | Out-Null
        Write-Log -Path $LogFile -Message "     Found template Customization Spec $($customSpec)"  -Component "VM Build" -Type Info
        try {
            Get-OSCustomizationSpec -Name $newVM -Server $vCenter -ErrorAction Stop | Remove-OSCustomizationSpec -Confirm: $false -ErrorAction Stop
            Start-Sleep 5
        } catch {
            Start-Sleep 1
        }
        try {
            New-OSCustomizationSpec -Name $newVM -Server $vCenter -OSCustomizationSpec $customSpec -Type NonPersistent -ErrorAction Stop
            Write-Log -Path $LogFile -Message "     Successfully created new OS Customization Spec for $($newVM)"  -Component "VM Build" -Type Info
            Start-Sleep 5
            try {
                Get-OSCustomizationSpec -Name $newVM -Server $vCenter | Set-OSCustomizationSpec -Domain $sDomain -DomainCredentials $creds -ErrorAction Stop
                Write-Log -Path $LogFile -Message "     Successfully reconfigured OS Customization Spec for $($newVM)"  -Component "VM Build" -Type Info
            } catch {
                #
            }
        } catch {
            return "CustCreateFail"
        }
    } catch {
        return "CustFindFail"
    }
#endregion Copy Customization Spec=============
    Start-Sleep 5
#region Set NICMapping on Customization Spec==================
    if (($template -match "WS_2012_R2_DE") -or ($template -match "WS_2016_DE")) {
        try {
            Get-OSCustomizationSpec -Name $newVM -Server $vCenter -ErrorAction Stop |
               Get-OSCustomizationNicMapping -ErrorAction Stop |
                  Set-OSCustomizationNicMapping -IpMode UseStaticIP `
                                                -IpAddress $IP `
                                                -SubnetMask $netMask `
                                                -DefaultGateway $Gateway `
                                                -DNS $DNS1,$DNS2 `
                                                -ErrorAction Stop |
                                                 Out-Null
            Write-Log -Path $LogFile -Message "     Successfully set NIC mapping on $($newVM) "  -Component "VM Build" -Type Info
        } catch {
            Write-Log -Path $LogFile -Message "     Unable to set NIC Mapping on $($newVM) Custom Spec, error is $($error[0]). Please configure manually."  -Component "VM Build" -Type Warning
            $iWarnings += 1
        }    
    } else {
        try {
            Get-OSCustomizationSpec -Name $newVM -Server $vCenter -ErrorAction Stop |
               Get-OSCustomizationNicMapping -ErrorAction Stop |
                  Set-OSCustomizationNicMapping -IpMode UseStaticIP `
                                                -IpAddress $IP `
                                                -SubnetMask $netMask `
                                                -DefaultGateway $Gateway `
                                                -ErrorAction Stop |
                                                 Out-Null
            Write-Log -Path $LogFile -Message "     Successfully set NIC mapping on $($newVM) "  -Component "VM Build" -Type Info
        } catch {
            Write-Log -Path $LogFile -Message "     Unable to set NIC Mapping on $($newVM) Custom Spec, error is $($error[0]). Please configure manually."  -Component "VM Build" -Type Warning
            $iWarnings += 1
        }
    }
 #endregion Set NICMapping on Customization Spec===============
   Start-Sleep 5
#region Build VM from Customization Spec=======================
    $sVM = @{
        Name = $newVM
        Template = (Get-Template $template -Server $vCenter)
        DataStore = $myDataStore
        OSCustomizationSpec = $newVM
        ResourcePool = (Get-Cluster $Cluster)
        RunAsync = $true
    }
    Write-Log -Path $LogFile -Message "     Beginning creation of $($newVM) on Cluster $($Cluster)"  -Component "VM Build" -Type Info
    try {
        $t = New-VM @sVM -Server $vCenter
        While('Running','Queued' -contains $t.State) {
            Write-log -Path $LogFile -Message "          ...$($t.PercentComplete)%"   -Component "VM Build" -Type Info
            Start-Sleep 5
            $t = Get-Task -Id $t.Id
        }
        Write-Log -Path $LogFile -Message "     Successfully created $($newVM) on Cluster $($Cluster)"  -Component "VM Build" -Type Info
    } catch [Exception] {
        return "VMBuildFail"
    }           
#endregion Build VM from Customization Spec====================
   Start-Sleep 5
#region Configure VM before Power On===========================
    try {
        Set-VM -VM $newVM `
               -Server $vCenter `
               -NumCpu $vCPU `
               -MemoryGB $RamGB `
               -Description $Desc `
               -Confirm:$false `
               -ErrorAction Stop |
               Out-Null
        Write-Log -Path $LogFile -Message "     Successfully configured $($newVM): $($vCPU) CPU, $($RamGB)GB RAM"  -Component "VM Build" -Type Info
    } catch {
        Write-Log -Path $LogFile -Message "     Failed configuring settings on $($newVM), please configure manually before powering on"  -Component "VM Build" -Type Warning
        $iWarnings += 1
    }
#endregion Configure VM before Power On========================
   Start-Sleep 5
#region Configure VLAN Settings=======================
        try {
            Get-VM $newVM -Server $vCenter -ErrorAction Stop | 
                Get-NetworkAdapter -Name "Network adapter 1" -ErrorAction Stop | 
                    Set-NetworkAdapter -Portgroup $PortGroup -Confirm: $false -ErrorAction Stop
            Write-Log -Path $LogFile -Message "     Successfully connected network adapter for $($newVM) to VLAN $($PortGroup)"  -Component "VM Build" -Type Info
        } catch {
            Write-Log -Path $LogFile -Message "     Failed to connect network adapter to VLAN $($PortGroup), returned error: $($error[0]). Please configure manually before powering on." -Component "VM Build" -Type Warning
            $iWarnings += 1
        }
#endregion Configure VLAN Settings====================
   Start-Sleep 5
#region Configure NIC to 'Connect at Startup'=========
    try {
        Get-VM $newVM -Server $vCenter | 
            Get-NetworkAdapter -Name "Network adapter 1" | 
                Set-NetworkAdapter -StartConnected:$true -Confirm:$false -ErrorAction Stop | Out-Null
        Write-Log -Path $LogFile -Message "     Successfully configured network adapter on startup for $($newVM)"  -Component "VM Build" -Type Info
    } catch {
        Write-Log -Path $LogFile -Message "     Failed to configure network adapter for $($newVM) to start Connected. Please configure manually before powering on."  -Component "VM Build" -Type Warning
        $iWarnings += 1
    }
#endregion Configure NIC to 'Connect at Startup'======
   return "Success"
}
_buildVM -newVM $VM.Name -vCPU $VM.NumCPU -RamGB $VM.MemoryGB -IP $address -destFolder $VM.Folder -Desc $VM.Description `
         -Cluster $VM.Compute_Cluster -dStore $VM.Datastore_Cluster -OS $VM.OS -Location $VM.Location -Domain $VM.Domain `
         -drive1Size $VM.Drive1Size -drive2Size $VM.Drive2Size -drive3Size $VM.Drive3Size -drive4Size $VM.Drive4Size -drive5Size $VM.Drive5Size

As I mentioned, these are literally the only two changes:

1. Instead of SF or DET in switch($Location), spell out "SanFrancisco and Detroit" to match sharepoint form.
2. In switch($Location) include the VLAN the server will be built on in log output

I just cannot see why the customization spec would be consistently joining the domain in one script but not another. Also wondering if there are any other logs I can look into; I seem to recall the guestcust.log used to show the domain join process, and whether it succeeded or not, but on both the server that does join and the one that doesn't the log makes no mention of it.

[LucD]

Log locations for OSCustomization are listed in KB1026317

I had a quick browse through your function, and I couldn't find how you pass or create the $cred variable (passed on DomainCredentials).

Since that one is used to join the domain, it probably needs to be different, depending on which domain is selected.
But I can't see that logic.

[JLogan2016]

Thanks for responding so quickly. After reading through your response I think I am getting closer to what is going on (although not why, per se):

In the working script, the user running the script (all domain admins and vCenter admins) is prompted for credentials and these are saved in a $creds variable that is used throughout the rest of the script, including the custom spec for joining to the domain. In the non-working script, we plan on using a service account (with the same permissions both on the domain and in vCenter). For testing, I have just been putting the credentials in like so, with the intent to pull them from an encrypted file once it goes to prod:

Connect-VIServer -Server $aHosts -User "company\serviceAccount" -Password "myPassword" -ErrorAction Stop

This has been working for the last couple of weeks as I test.

I just tried changing it however, to where it prompts for credentials, and used my own creds. In this instance the VM joined the domain just fine. I also had it prompt for credentials and used the service account's credentials, and it joined to domain. So something seemed to have changed with embedding the User and Password in the Connect-VIServer call, though I am not sure what. I tried the embedded approach, using my credentials, and it failed as well. Just for grins, I also tried it with the fully qualified usernames, both for me and the service account <username>@company.com, but that did not work either.

So I guess I will look into the log files more and see if I can figure out why it works when prompted but not when put in the script. I just do not understand what changed that it was working until today.

[LucD · Accepted solution]

Are you sure that a user from DomainA is allowed to join a machine to DomainB?

That might be an issue when the domain trusts are not set up correctly.

When you use the 'embedded credentials', do you also populate the $creds variable?

Because that is the one used in the Set-OScustomizationSpec.

---------------------------------------------------------------------------------------------------------

Was it helpful? Let us know by completing this short survey here.

[JLogan2016]

Sorry for the delay. Thanks, this was it. I have an intern working with me that modified the script, putting the $creds back in. I missed that line. I have since implemented some versioning control to keep this from happening again