Need some assistance with automating new server builds

[Fender84]

Hello,
I'm trying to get some assistance with my script using PowerCLI. I am working on a multithreaded way to build servers on the fly. My script works, but sometimes it gives me mixed results. 

To summarize what I'm doing; I have a spreadsheet that I populate a bunch of information in to, and then I read the data from the spreadsheet and build the server based on that data. The server creation component is pretty straight forward, but the part I'm running into trouble is with the configuration of a new osCustomizationSpec. Since each server is unique in the sense that it has a unique IP and could be in a different DC, I decided to create a new customizationspec for each server. The spec is just named the same as the server and I check to make sure it doesn't already exist so that there is no conflict with another server. Sometimes the server builds out correctly, but other times I get an error message stating that the OSCustomizationSpec can't be found even though I defined it. 

Another piece of information is that for the multithreading I am using poshrsjob. I am also using a module called ImportExcel to grab the information from my spreadsheet.

One other issue I ran into is assuming the new server gets built with the customspec sheet, sometimes I get an error stating the server can't be found when I try to configure the network adapter. I tried putting a sleep and also a loop to check the server exists before continuing, but this seems to get stuck. Any assistance is greatly appreciated.

Here is an example of an error I would get for a server.

09.21.2023 2:17:35 PM Get-OSCustomizationSpec Could not find Customization Specification with name 'TDVS01ECBO7858'.
09.21.2023 2:17:35 PM Get-OSCustomizationSpec Could not find Customization Specification with name 'TDVS01ECBO7858'.
09.21.2023 2:17:35 PM Get-OSCustomizationSpec Could not find Customization Specification with name 'TDVS01ECBO7858'.

Here is my script.

I am using the newest version of powercli - 13.1.0.21624340 and powershell 5.1

CLS

#=================== Install VMware PowerCLI if it doesn't already exist

if($NULL -eq (Get-Module -ListAvailable -Name "VMware.PowerCLI"))
{	
	Install-Module -Name VMware.PowerCLI -Scope CurrentUser -Force -AllowClobber -Confirm:$False
}

#=================== PoshRSJob (multitasking)

if($NULL -eq (Get-Module -ListAvailable -Name "PoshRSJob"))
{	
	Install-Module -Name PoshRSJob -Scope CurrentUser -Force -Confirm:$False
}

#=================== ImportExcel Module 

if((Get-Module -ListAvailable -Name "ImportExcel") -or (Get-Module -Name "ImportExcel"))
{
	Import-Module ImportExcel
}
else
{
    #Install NuGet (Prerequisite) first
	Install-PackageProvider -Name NuGet -Scope CurrentUser -Force -Confirm:$False
	
    Install-Module -Name ImportExcel -Scope CurrentUser -Force -Confirm:$False
	Import-Module ImportExcel
}

if(([Net.ServicePointManager]::SecurityProtocol) -ne "Tls, Tls11, Tls12")
{
    [Net.ServicePointManager]::SecurityProtocol = 'Tls', 'Tls11','Tls12'
}

#Clear screen again
CLS

#Start Timestamp
$Start = Get-Date

$Path = (Split-Path $script:MyInvocation.MyCommand.Path)

#===================  Setup Excel Variables

#The file we will be reading from
$ExcelFile = (Get-ChildItem -Path "$Path\*.xlsx").FullName

#Worksheet we are working on (by default this is the 1st tab)
$worksheet = (((New-Object -TypeName OfficeOpenXml.ExcelPackage -ArgumentList (New-Object -TypeName System.IO.FileStream -ArgumentList $ExcelFile,'Open','Read','ReadWrite')).Workbook).Worksheets[0]).Name

$ServerBuilds = Import-Excel -Path $ExcelFile -WorkSheetname $worksheet -StartRow 1

#===================  Connect to vCenter Servers

if($NULL -eq ($global:DefaultVIServers.Name))
{
    $cred = (Get-Credential (whoami))
    
    Connect-VIServer "pvsa01vcsa0001" -Protocol https -Credential $cred -AllLinked -WarningAction 0 | Out-NULL
    Connect-VIServer "pvsa02vcsa0001" -Protocol https -Credential $cred -AllLinked -WarningAction 0 | Out-NULL
}

#===================  Configure the Customizations to use for our machine(s)

$ServerBuilds | Start-RSJob -Throttle (($ServerBuilds | Measure-Object).count) -ScriptBlock {
    Param($Server)

    #=================== Configure our Variables

    $ServerName = $Server.ServerName
    $ServerIP = $Server.ServerIP
    $ServerClone = $Server.vmClone
    $ServerDC = $Server.DataCenter
    $ServerCluster = $Server.Cluster
    $ServerVmLocation = $Server.VmLocation
    $CPU = $Server.CPU
    $Mem = $Server.Memory
    $SQL = $Server.SQL
    $DMZ = $Server.DMZ
    $Test = $Server.Test
    $Tier = $Server.Tier
    $ASA = $Server.ASA
    $BS = $Server.BS

    #=================== Make sure the IP field is NOT blank/empty

    if($NULL -eq $ServerIP)
    {
        Write-Host "Make sure you proide an IP for your server: $ServerName"
    }

    #=================== Ping IP to make sure the IP is not taken
    if(($NULL -ne (Get-CimInstance -ClassName Win32_PingStatus -Filter "Address='$ServerIP' AND Timeout=100").ResponseTime) -AND ($NULL -ne $ServerIP))
    {
        Write-Host "It looks like the server IP $ServerIP is in use."
    }

    #Configure out Subnet
    $ServerSubnet = "255.255.255.0"

    #Grab Vlan from our Spreadsheet based on the Server IP
    $ServerVLAN = ($ServerIP.Split('.')[1..2] -join '').Remove(1,1)

    #Set the vcenter and DNS info based on which DC the server is on
    if($ServerDC -eq "DC01")
    {
        $vCenter = "pvsa01vcsa0001"
        $vmDNS = "1.2.3.4","2.3.4.5"
    }
    if($ServerDC -eq "DC02")
    {
        $vCenter = "pvsa02vcsa0001"
        $vmDNS = "2.3.4.5","1.2.3.4"
    }

    #Gateway depends on whether the server is in the DMZ
    if($DMZ -eq $True)
    {
        #Configure Gateway
        $ServerGateway = $ServerIP -replace '(?<=^(\d+\.){3})\d+', '1'
        
        #Get the Virtual network adapter we will be assigning based on vlan
		$vmNet = (Get-VirtualNetwork -Server $vCenter -Name "DMZ-$($ServerVLAN)*").Name
		
        #Configure CustomSpec based on whether we're in a DMZ
        $CustomizationSpec = "DMZ_Static_IP_Deployment"
        $IPMode = "UseStaticIP"
    }
    else
    {
        #Configure Gateway
        $ServerGateway = $ServerIP -replace '(?<=^(\d+\.){3})\d+', '254'
		
        #Get the Virtual network adapter we will be assigning based on vlan
		$vmNet = (Get-VirtualNetwork -Server $vCenter -Name "*$($ServerVLAN)*").Name
		
        #Configure CustomSpec based on whether we're in a DMZ
        $CustomizationSpec = "Static_IP_Deployment"
        $IPMode = "UseStaticIP"
    }

    #===================  Setup Our Configurations and Parameters

    #Set our DataStore volume we will use
    $vmDS = (Get-Cluster -Server $vCenter -Name $ServerCluster | Get-Datastore | where{$_.Name -like "PSP*"}).Name
	
    #If the datastore can't be found above, check another similar server and get their datastore
    if($NULL -eq $vmDS)
    {
		if($DMZ -eq $True)
		{
			$SearchServer = ((get-view -Server $vCenter -viewtype virtualmachine -Filter @{"Name" = "^$($ServerName -replace '\d+$','9')"}) | sort Name)[0]
		}
		else
		{
			$SearchServer = ((get-view -Server $vCenter -viewtype virtualmachine -Filter @{"Name" = "^$($ServerName -replace '\d+$','')"}) | sort Name)[0]
		}

        $vmDS = (Get-View -Server $vCenter -Id $SearchServer.Datastore -Property Name).Name -join '|'
    }
    
    ###=================================

    #Grab our Template
	$Template = (Get-View -ViewType VirtualMachine -Server $vCenter -Property Name, Config.Template | Where-Object {($_.Config.Template -eq "True") -AND ($_.Name -like "$ServerClone")}).Name

	if((($Template | Measure-Object).count) -gt 1)
	{
		$Template = $Template[0]
	}

    #Create a new Customization spec if it doesn't already exist
    if($ServerName -notin (Get-OScustomizationspec -Server $vCenter -Type NonPersistent).Name)
	{
		New-OSCustomizationSpec -OSCustomizationSpec $CustomizationSpec -Name $ServerName -Server $vCenter -Type NonPersistent -Confirm:$False -ErrorAction 0 -WarningAction 0
    }

    #Grab our customization spec and apply network settings
    if($NULL -eq ((Get-OScustomizationspec -Server $vCenter -Type NonPersistent -Name $ServerName) | Get-OSCustomizationNicMapping).IPAddress)
    {
        Get-OSCustomizationSpec -Server $vCenter -Name  $ServerName | Get-OSCustomizationNicMapping | Set-OSCustomizationNicMapping -Server $vCenter -IpMode $IPMode -IpAddress $ServerIP -SubnetMask $ServerSubnet -DefaultGateway $ServerGateway -Dns $vmDNS
    }

    if($ServerIP -eq ((Get-OScustomizationspec -Server $vCenter -Type NonPersistent -Name  $ServerName | Get-OSCustomizationNicMapping).IPAddress))
    {
        #Store our custom spec in a variable
        $OSCustSpec = Get-OSCustomizationSpec -Name $ServerName -Server $vCenter

        #=================== Create our new VM with all the configurations we defined
        Try
        {
            #Create Clone from Template        
            New-VM -Server $vCenter -Name $ServerName -Template $Template -ResourcePool $ServerCluster -Datastore $vmDS -Location $ServerVmLocation  -OSCustomizationSpec $osCustSpec -DiskStorageFormat Thin -RunAsync -Confirm:$false -ErrorAction 0 -WarningAction 0
        }
        Catch
        {
            #Create Clone from Machine
            New-VM -Server $vCenter -Name $ServerName -VM $Template -ResourcePool $ServerCluster -Datastore $vmDS -Location $ServerVmLocation –OSCustomizationSpec $osCustSpec -DiskStorageFormat Thin -RunAsync -Confirm:$false  -ErrorAction 0 -WarningAction 0
        }

        #Make sure the server exists before we configure the Network Adapter
        do
        {
            $testServer = ((get-view -Server $vCenter -viewtype virtualmachine -Filter @{"Name" = "$($ServerName)"}))
        }while($NULL -eq $testServer)
        
        #Set Network Configurations
        Get-NetworkAdapter -Server $vCenter -VM $ServerName | Set-NetworkAdapter -NetworkName $vmNet -StartConnected:$true -RunAsync -Confirm:$false
    }

    #If the server we build needs more CPU or memory than default, here is where we will do that.
	if($NULL -ne $CPU)
    {
        Set-VM -Server $vCenter -VM $ServerName -NumCpu $CPU -CoresPerSocket (($CPU)/2) -RunAsync -Confirm:$false
    }
	
	if($NULL -ne $Mem)
    {
        Set-VM -Server $vCenter -VM $ServerName -MemoryGB $Mem -RunAsync -Confirm:$false
    }

    #=================== Disable hot swap

    $vm = Get-VM -Name $ServerName -Server $vCenter

    $spec = New-Object VMware.Vim.VirtualMachineConfigSpec

    $spec.CpuHotAddEnabled = $False
    $spec.MemoryHotAddEnabled = $False

    $vm.ExtensionData.ReconfigVM($spec)

    #=================== Assign VMware Tags

    #At minimum assign the OS as a tag to the new machine
    $OS = (Get-Tag -Server $vCenter -ErrorAction 0 -WarningAction 0 | Where-Object { (($_.Name -like "*2019*") -AND ($_.Category -like "Operating System")) -AND (($_.UID -like "*$vCenter*") -AND ($_.UID -notlike "*.domain.com*")) })
    Try{if($NULL -ne $OS){Get-VM -Name $erverName -Server $vCenter -ErrorAction 0 -WarningAction 0 | New-TagAssignment -Server $vCenter -Tag $OS -Confirm:$false -ErrorAction 0 -WarningAction 0 }}Catch{}
    
    #If we define the tier the server is in, then assign it as a tag
    if($NULL -ne $Tier)
    {
        $Tier = (Get-Tag -Server $vCenter -ErrorAction 0 -WarningAction 0 | Where-Object { (($_.Name -like "*$(($Tier).Trim())*") -AND ($_.Category -like "Business Criticality")) -AND (($_.UID -like "*$vCenter*") -AND ($_.UID -notlike "*.domain.com*")) })
        Try{if($NULL -ne $Tier){Get-VM -Name $ServerName -Server $vCenter -ErrorAction 0 -WarningAction 0 | New-TagAssignment -Server $vCenter -Tag $Tier -Confirm:$false -ErrorAction 0 -WarningAction 0 }}Catch{}
    }

    #If we define the App Admn the server is in, then assign it as a tag
    if($NULL -ne $ASA)
    {
        $ASA = (Get-Tag -Server $vCenter -ErrorAction 0 -WarningAction 0 | Where-Object { (($_.Name -like "*$(($ASA).Trim())*") -AND ($_.Category -like "Primary ASA")) -AND (($_.UID -like "*$vCenter*") -AND ($_.UID -notlike "*.domain.com*")) })
        Try{if($NULL -ne $ASA){Get-VM -Name $ServerName -Server $vCenter -ErrorAction 0 -WarningAction 0 | New-TagAssignment -Server $vCenter -Tag $ASA -Confirm:$false -ErrorAction 0 -WarningAction 0 }}Catch{}
    }

    #If we define the Business Service the server is in, then assign it as a tag
    if($NULL -ne $BS)
    {
        $BS = (Get-Tag -Server $vCenter -ErrorAction 0 -WarningAction 0 | Where-Object { (($_.Name -like "*$(($BS).Trim())*") -AND ($_.Category -like "Business Service")) -AND (($_.UID -like "*$vCenter*") -AND ($_.UID -notlike "*.domain.com*")) })
        Try{if($NULL -ne $BS){Get-VM -Name $ServerName -Server $vCenter -ErrorAction 0 -WarningAction 0 | New-TagAssignment -Server $vCenter -Tag $BS -Confirm:$false -ErrorAction 0 -WarningAction 0 }}Catch{}
    }

    #=================== Power on the machine

    Start-VM -Server $vCenter -VM $Server.ServerName -confirm:$false -RunAsync

   ###=================================

   Write-Output ""
   Write-Output "=================================================================="
   Write-Output ""

} | Wait-RSJob -ShowProgress | Receive-RSJob

#=================== Remove our custom template(s) once we built all of our server(s)

Get-OSCustomizationSpec -Type NonPersistent | Remove-OSCustomizationSpec -Confirm:$false

$End = Get-Date

$End - $Start

[LucD]

Can you try using Start-Job instead of the PoshRSJob module?
The latter is based on runspaces and unfortunately PowerCLI is not thread-safe afaik.

[Fender84]

I will give it a go, thank you! Are there any examples of using start-job with powercli? I can do some research as well to see what I can find.

[LucD]

I did a short post on Start-job and PowerCLI, see Running a background job
That might help to get you started.

Also if you search in here for Start-Job you will find a number threads, several of them answered

[Fender84]

Hello, I had a quick question, I came across this post that might be useful: https://communities.vmware.com/t5/VMware-PowerCLI-Discussions/Multithreading-PowerCLI-with-RunspaceP...

I'm not sure I understand the usage of that script though. I know the original creator said he won't provide support on it, but I was wondering how would I use that (in the most basic form for example just running get-vm); so I have a general idea of how to pipe my script above through this.

[LucD]

In $script_block you store the code, in a code block, you want to execute.
In the array $some_list you store the arguments that you want to pass to the code in $scriptBlock.
Make sure to use a Param statement in the scriptblock, or else use $args[0].

But, again, PowerCLI is not threadsafe, that code uses runspaces, you are bound to encounter issues.