VMWare build automation - issues

drizuid · ‎10-22-2013

Here's some background. We were previously hand creating massive amounts of servers and just needed to increase our efficiency. I created this script to minimize our hands on (and increase quality by reducing typos) and maximize our time. I've tried to launch the subscripts as jobs (commented out in my master.ps1 code) and been unsuccessful with that. I'm also having an issue removing jobs in my worker.ps1 script which is causing the memory utilization to skyrocket as the number of builds goes up until i run out of ram and powershell crashed. I'm provisions hundreds of servers at a time (with a limit shown in the master.ps1 code) I'm trying to figure out 1) how to make this more efficient and 2) how to remove completed jobs to reduce memory utilization.

The inputs are the 2 attachments the first is all my VM info, the second is for additional harddrives (specific to windows)

the code WORKS and we've deployed over 500 VMs from this, so feel free to utilize it, im just trying to tweak. with 8GB ram, i run into memory issues after about 30 servers being provisioned, this caused me some heartache when i tried to deploy 112.

Master.ps1

<################################################################################

#

# Title: VM Mass Deploy Script

# Version: 2.0w1

# Author: Will Longo (will.longo@nospam.com)

# Modified by:

# Purpose: Mass provisioning of VMs for rapid builds. By populating

#    CSV files, user can specify the following specifications for VMs:

#

#    Name, # of CPUs, Memory, DataCenter, Cluster, ESX Host, DataStore, Folder,

#    Resource Pool, Folder, IP, Subnet, Gateway, DNS (non-linux), Template to base from,

#    Additional HDDs (with unique DS location and size)

#

# Usage: Requires one parameter to the Master.ps1 script, being the something.csv file

#    Master then reads the file, spawning a separate PS instance (Worker.PS1) for

#    each and providing the parameters for that VM. Master controls the number

#    of worker processes by querying VC for active clone tasks and governing it

#    to a defined value.

#

#    Worker builds a VM spec for the Sysprep and deploys from the template. After

#    a period, it then adjusts the # of CPUs and RAM, reads the HDD.csv file for

#    any additional HDDs required (specific DS & size specs accepted, one line

#    per additional HDD) and finally starts the VM. The VM then performs auto-login,

#    Syspreps the box with hostname and IP info, then reboots.

#

# Versioning:

# 1.0w3 - Added jobs, linux/windows support, post install

# 1.0w2 - Added vmware tools update (removed), modified gui(removed) and additional disks doc

#    1.0 - Added GUI and changed required format of the excel doc for clinical

#

#

################################################################################>

if ( (Get-PSSnapin | Where-Object { $_.Name -eq "VMware.VimAutomation.Core"}) -eq $null ){

Add-PSSnapIn VMware.VimAutomation.Core}

#function to browse to the csv

Function Browse(){

    [System.Reflection.Assembly]::LoadWithPartialName("System.windows.forms") |Out-Null

    [System.Windows.Forms.Application]::EnableVisualStyles()

    $OpenFileDialog = New-Object System.Windows.Forms.OpenFileDialog

    $OpenFileDialog.initialDirectory = (get-location).path

    $OpenFileDialog.filter = "CSV (*.csv)|*.csv|All files (*.*)|*.*"

    $loop = $true

    while($loop)

    {

        if ($OpenFileDialog.ShowDialog() -eq "OK")

        {

            $loop = $false

        }

        else

        {

            $res = [System.Windows.Forms.MessageBox]::Show("You clicked Cancel. Try again or return to main form?", "Choose a directory", [System.Windows.Forms.MessageBoxButtons]::RetryCancel)

            if($res -eq "Cancel")

            {

                #End script

Remove-VICredentialStoreItem $vcserver -confirm:$false -wa 0

return

            }

        }

    }

    $OpenFileDialog.filename

}

#the actual stuff

<#Function Limit-Jobs {

Param([int]$MaxConcurrent,[int]$PauseTime)

$jobs = (get-job -state running | Measure-Object).count

$RunningJobs = 0

if($jobs -ne $null) {$RunningJobs = $jobs}

while($RunningJobs -ge $MaxConcurrent)

{

$jobs = (get-job -state running | Measure-Object).count

$RunningJobs = 0

if($jobs -ne $null){

$RunningJobs = $jobs

}

Write-Warning "Current Running Jobs: $RunningJobs"

get-job

start-sleep -seconds $PauseTime

}

}#>

function Program{

# Set Static Variables

$DeploymentThreads = "4" #NUMBER OF DEPLOYMENTS TO RUN CONCURRENTLY

# set your job limit below at -MaxConcurrent on line 120

#Export Credentials

$credSuper = $Host.ui.PromptForCredential("VCenter Credentials","Enter your vcenter super account and password","","")

if(!$credSuper){write-host "Cancelled by User!" -ForegroundColor Yellow -BackgroundColor Red;return}

$cred = new-vicredentialstoreitem -host $vcserver -user $credSuper.UserName -Password $credSuper.GetNetworkCredential().password

#import and connect

$CSV = Browse

if (!$csv){$cancel=$true}

if ($cancel){write-host "Cancelled by User!" -ForegroundColor Yellow -BackgroundColor Red;return}

$INFILE = Import-Csv $CSV

Connect-VIServer -server $cred.Host -user $cred.User -Password $cred.Password -wa 0

$host.ui.RawUI.WindowTitle = "VM Mass Deploy Script - Master"

foreach ($VM in $INFILE)

{

#send the sheetpath/name, vm name, and current vcserver over to the worker

$loc=(get-location).path

$loc2=$loc+"\worker.ps1"

Write-Host "Starting job to deploy: " -NoNewline; Write-Host $VM.Name -ForegroundColor Yellow -BackgroundColor Red

########

#Hjob handling that works; only launches one at a time for me, but WORKS

########

$params = $loc2 +" "+ $CSV +" "+ $VM.Name +" "+ $vcserver+" "+ $loc

cmd /c start powershell -Command $params -NoExit

sleep 45 # Time for clone task to begin or we get slammed with jobs

while ( ( get-task | where { $_.Name -eq "CloneVM_Task" } | where { $_.State -eq "Running"} ).count -ge $DeploymentThreads )

{

Write-Host "Sleeping..."

sleep 60

}

########

# how im trying to do jobs, launches cleanly and limit-jobs is nice but doesnt complete jobs...

# Using jobs with concurrency limit

########

#Limit-Jobs -MaxConcurrent 3 -PauseTime 90

#start-job -FilePath $loc2 -ArgumentList @($CSV,$VM.Name,$vcserver,$loc)

}

Disconnect-VIServer -Server $cred.host -Confirm:$False

write-host "Completed..."

}

##Initial start

Write-host "Will Longo's Mass Provisioning tool"

write-host ""

write-host "1 for server1"

write-host "2 for server2"

write-host "3 for server3"

write-host "4 for server4"

write-host "5 for server5"

write-host "6 for server6"

write-host "7 for server7"

write-host "8 for server8"

$x = read-host "Select your vcenter by the number above"

if ($x -eq 1){$vcserver = "server1"}

elseif($x -eq 2){$vcserver = "server2"}

elseif($x -eq 3){$vcserver = "server3"}

elseif($x -eq 4){$vcserver = "server4"}

elseif($x -eq 5){$vcserver = "server5"}

elseif($x -eq 6){$vcserver = "server6"}

elseif($x -eq 7){$vcserver = "server7"}

elseif($x -eq 8){$vcserver = "server8"}

else{Write-Host "Invalid Selection or server not added to script"}

Program

Worker.ps1

<#

Support for Windows customization specs

Author: Will Longo

#>

#load Snapin

if ( (Get-PSSnapin | Where-Object { $_.Name -eq "VMware.VimAutomation.Core"}) -eq $null ){

Add-PSSnapIn VMware.VimAutomation.Core}

# Open VC connection, assign passed variables and notify user which machine will be deployed

$CSV = Import-Csv $args[0]

$CSV1 = $args[0]

$vmname = $args[1]

$VM = $CSV | Where-Object {$_.Name -eq $vmname}

$VM | fl

$vcserver = $args[2]

$loc = $args[3]

$cred=Get-VICredentialStoreItem -host $vcserver

#WL - if it's linux you can't pass dns or wins variables

$newSpec = "VMmd_"+ $VM.OSCustSpec +"_"+ $vmname

if ($VM.Operating -eq "Linux")

{

$loc2=$loc + "\linuxspec.ps1"

start-job -FilePath $loc2 -ArgumentList @($csv1,$vmname,$vcserver,$loc) -RunAs32

sleep 15

write-Host "Linux CustomizationSpec created"

}else{

$loc2=$loc + "\windowsspec.ps1"

write-host "Starting job"

start-job -FilePath $loc2 -ArgumentList @($csv1,$vmname,$vcserver,$loc) -RunAs32

write-host "job started"

get-job | wait-job | out-null

write-host "checking job"

Remove-Job -State Completed

write-host "removing job"

#sleep 15

write-Host "Windows CustomizationSpec created"

}

Connect-VIServer -server $cred.Host -user $cred.User -Password $cred.Password -wa 0

$host.ui.RawUI.WindowTitle = $vmname

Write-Host "Now Deploying: " -NoNewline; Write-Host $VM.Name -ForegroundColor Yellow -BackgroundColor Red

if(get-vm -name $vm.name -ea 0){Disconnect-VIServer -Confirm:$False;return;}

New-VM -Name $VM.Name -Template $VM.Template -VMHost (Get-VMHost $VM.ESXHost) -Datastore $VM.DataStore -OSCustomizationSpec (Get-OSCustomizationSpec $newSpec)

Write-Host "Completed deploying and preparing to customize hardware for: " -NoNewline; Write-Host $VM.Name -ForegroundColor Yellow -BackgroundColor Red

$VMobj = Get-VM $VM.Name

# Verify location of disks.csv

$fPath = (get-childitem $CSV1).DirectoryName

if(!(test-path $fPath\disks.csv))

{

if(!(test-path $fPath\..\disks.csv))

{

write-host "Disks.csv not present"

}

else

{

$infile = import-csv $fPath\..\Disks.csv

}

}

else

{

$infile = import-csv $fPath\Disks.csv

}

foreach ($line in $infile)

{

if ($line.VMName -eq $VMobj.Name)

{

if ($line.isRDM -eq "N")

{

$VMobj | New-HardDisk -Datastore $line.Datastore -CapacityGB $line.CapacityGB

}

elseif($line.isRDM -eq "Y")

{

$VMobj | New-HardDisk -DiskType $line.RDMDiskType -DeviceName $line.RDMDeviceName

}

else

{

Write-Host "Invalid RDM Selection"

}

}

}

# Set final VM specs & power-on

$VMobj | Set-VM -NumCpu $VM.NumCPU -MemoryMB $VM.MemoryMB -Confirm:$false

$VMobj | Get-NetworkAdapter | Set-NetworkAdapter -NetworkName $VM.PortGroup -startconnected:$true -confirm:$false #!!!

Write-Host "Preparing to power on: " -NoNewline; Write-Host $VM.Name -ForegroundColor Yellow -BackgroundColor Red

$VMobj | Start-VM -RunAsync -Confirm:$false

sleep 1000 #sleep to let it join the domain and post install has like 3 reboots

#$VMobj | Update-Tools #this will uninstal vmware tools if they're already current?!?! WTF

#sleep 480

Write-Host "Removing temporary OS Customization on " -NoNewline; Write-Host $VM.Name -ForegroundColor Yellow -BackgroundColor Red

Get-OSCustomizationSpec $newSpec | Remove-OSCustomizationSpec -Confirm:$false

Write-Host $vmname deployment complete -ForegroundColor Yellow -BackgroundColor Red

#Post Install Section

if ($VM.Operating -eq "Windows")

{

$loc3=$loc + "\" + $VM.Name + "defsvrbld.cmd"

$file = $VM.Name + "defsvrbld.cmd"

$filerun = "c:\installs\" + $file

#copy the post install scripts that was generated by windowsspec over

Copy-VMGuestFile -LocalToGuest -VM $VM.Name -Source "$loc3" -Destination "c:\installs\" -GuestUser "Administrator" -GuestPassword "s3cre3tP@ssw0rd"

Invoke-VMScript -ScriptType "bat" -ScriptText $filerun -VM $VM.Name -GuestUser "Administrator" -GuestPassword "s3cre3tP@ssw0rd" -ErrorAction SilentlyContinue

sleep 120

$cmd1 = "del " + $filerun

Invoke-VMScript -ScriptText $cmd1 -VM $VM.Name -GuestUser "secretadmin" -GuestPassword "secretpassword" -ErrorAction SilentlyContinue

Remove-Item $loc3

}

else

{

write-host "No post-install"

}

Disconnect-VIServer -Confirm:$False

write-host "Disconnected from VCenter"

windowsspec.ps1

<#

Support for Windows customization specs

Author: Will Longo

#>

#prep the environment

$CSV = Import-Csv $args[0]

$vmname = $args[1]

$vcserver = $args[2]

$loc = $args[3]

$VM = $CSV | Where-Object {$_.Name -eq $vmname}

#post build stuff

$loc2=$loc + "\" + $VM.Name + "defsvrbld.cmd"

write-host $loc2

Add-Content $loc2 "@echo off"

if ($VM.Environment -eq "QOL"){

$devenvironLAN = "ServerXQOL_LAN"

$devenviron = "ServerXQA"

$domain = "someserver2"

Add-Content $loc2 "reg add HKLM\System\currentcontrolset\services\tcpip\parameters /v SearchList /d ServerXqol.server2.net /f"}

elseif ($VM.Environment -eq "PROD"){

$devenvironLAN = "ServerX_LAN"

$devenviron = "ServerX"

$domain = "someserver1"

Add-Content $loc2 "reg add HKLM\System\currentcontrolset\services\tcpip\parameters /v SearchList /d ServerX.server1.net /f"

Add-Content $loc2 "net localgroup Administrators /ADD ServerX\dsdasdffdf"}

elseif ($VM.Environment -eq "QA"){

$devenvironLAN = "ServerXQA_LAN"

$devenviron = "ServerXQA"

$domain = "someserver4"

Add-Content $loc2 "reg add HKLM\System\currentcontrolset\services\tcpip\parameters /v SearchList /d ServerXqa.server4.net /f"

Add-Content $loc2 "net localgroup Administrators /ADD ServerXQA\blah"}

elseif ($VM.Environment -eq "PF"){

$devenvironLAN = "ServerXPF_LAN"

$devenviron = "ServerX"

$domain = "someserver3"

Add-Content $loc2 "reg add HKLM\System\currentcontrolset\services\tcpip\parameters /v SearchList /d serverxPF.someserver3 /f"

Add-Content $loc2 "net localgroup Administrators /ADD ServerXPF\dsadasdsadsadee"

Add-Content $loc2 "reg add HKLM\SOFTWARE\Microsoft\Rpc\Internet /v Ports /t REG_MULTI_SZ /d 5000-5100 /f"

Add-Content $loc2 "reg add HKLM\SOFTWARE\Microsoft\Rpc\Internet /v PortsInternetAvailable /t REG_SZ /d Y /f"

Add-Content $loc2 "reg add HKLM\SOFTWARE\Microsoft\Rpc\Internet /v UseInternetPorts /t REG_SZ /d Y /f"

Add-Content $loc2 "net localgroup Administrators /ADD ServerXPF\dsdasdas"}

else{write-host "You do not match any environments, check your csv"}

Add-Content $loc2 "netsh interface set interface name = ""Local Area Connection 5"" newname = ""$devenvironLAN"""

Add-Content $loc2 "net stop ""Windows Firewall"""

Add-Content $loc2 "sc config ""Windows Firewall"" start= disabled"

Add-Content $loc2 "reg add HKLM\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters /v DisabledComponents /t reg_DWORD /d 1107296255 /f"

Add-Content $loc2 "reg add HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v EnableLUA /t REG_DWORD /d 0 /f"

Add-Content $loc2 "net user console0 ""secretpassword111"""

Add-Content $loc2 "net localgroup Administrators /DEL ""$devenviron\domain admins"""

#Add-Content $loc2 "powershell -command ""&{$NIC = gwmi win32_networkadapterconfiguration -computer $vmname | where{$_.IPEnabled -eq $TRUE};$nic.SetDNSDomain($domain)}"

Add-Content $loc2 "net user Administrator ""secret password"" & wmic useraccount where name='Administrator' call rename name='secretadmin'"

if ($VM.Meditech -eq "Y"){

Add-Content $loc2 "reg add HKLM\SYSTEM\CurrentControlSet\Control\FileSystem /v NtfsDisable8dot3NameCreation /t reg_DWORD /d 00000001 /f"

Add-Content $loc2 "reg add HKLM\SYSTEM\CurrentControlSet\Control\FileSystem /v NtfsMftZoneReservation /t reg_DWORD /d 00000004 /f"

Add-Content $loc2 "reg add HKLM\SYSTEM\CurrentControlSet\Control\FileSystem /v NtfsDisableLastAccessUpdate /t reg_DWORD /d 00000001 /f"

Add-Content $loc2 "reg add HKLM\SYSTEM\CurrentControlSet\Control\Lsa\FIPSAlgorithmPolicy /v Enabled /t reg_DWORD /d 00000001 /f"

Add-Content $loc2 "reg add HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters /v MaxUserPort /t reg_DWORD /d 00008192 /f"

Add-Content $loc2 "reg add HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters /v TcpMaxDataRetransmissions /t reg_DWORD /d 00000004 /f"

Add-Content $loc2 "reg add HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters /v TcpTimedWaitDelay /t reg_DWORD /d 00000032 /f"

Add-Content $loc2 "reg add HKLM\System\currentcontrolset\services\tcpip\parameters /v SearchList /d ""anpserver.medcity.net,$domain"" /f"

Add-Content $loc2 "net localgroup Administrators /ADD $devenviron\Meditech"

Add-Content $loc2 "net localgroup Administrators /ADD $devenviron\Meditech-bkg"

Add-Content $loc2 "net localgroup Administrators /ADD $devenviron\Meditech-admin"

Add-Content $loc2 "net localgroup ""Remote Desktop Users"" /ADD $devenviron\Meditech"

Add-Content $loc2 "net localgroup ""Remote Desktop Users"" /ADD $devenviron\Meditech-bkg"

Add-Content $loc2 "net localgroup ""Remote Desktop Users"" /ADD $devenviron\Meditech-admin"

}else{write-host "Not Meditech"}

# Prep the VM environment

if ( (Get-PSSnapin | Where-Object { $_.Name -eq "VMware.VimAutomation.Core"}) -eq $null ){

Add-PSSnapIn VMware.VimAutomation.Core}

# Open VC connection, assign passed variables and notify user which machine will be deployed

$cred=Get-VICredentialStoreItem -host $vcserver

Connect-VIServer -server $cred.Host -user $cred.User -Password $cred.Password -wa 0

$newspecname = "VMmd_"+ $VM.OSCustSpec +"_"+ $vmname

$newspec = New-OSCustomizationSpec -name $newspecname -spec $VM.OSCustSpec

if ($VM.Environment -eq "QOL"){

$DNSDomain = "someserver2"}

elseif ($VM.Environment -eq "PROD"){

$DNSDomain = "someserver1"}

elseif ($VM.Environment -eq "QA"){

$DNSDomain = "someserver4"}

elseif ($VM.Environment -eq "PF"){

$DNSDomain = "someserver3"}

else{Write-host "No valid Environment Specified"}

write-host "setting customization"

Get-OSCustomizationSpec $newspecname | Get-OSCustomizationNicMapping | Set-OSCustomizationNicMapping -IpMode UseStaticIP -IpAddress $VM.IP -SubnetMask $VM.Subnet -DefaultGateway $VM.Gateway -Dns $VM.DNS1,$VM.DNS2,$VM.DNS3 -Wins $VM.WINS1,$VM.WINS2,$VM.WINS3

if($VM.Environment -eq "PF"){

$pfcredname = "ServerX\"+$cred.user

Get-OSCustomizationSpec $newspecname | Set-OSCustomizationSpec -DnsSuffix $DNSDomain -Domain $DNSDomain -DomainUsername $pfcredname -DomainPassword $cred.Password

}else{

Get-OSCustomizationSpec $newspecname | Set-OSCustomizationSpec -DnsSuffix $DNSDomain -Domain $DNSDomain -DomainUsername $cred.User -DomainPassword $cred.Password -confirm:$false

}

disconnect-viserver -confirm:$false;return;

Message was edited by: Will Longo removed some account names

LucD · ‎10-22-2013

I'm I misinterpreting this, but are you trying to do the Remove-Job from within the actual job ?

That doesn't work afaik, you will have to this in the same script that does the Start-Job.

Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference

drizuid · ‎10-23-2013

LucD, (i told a coworker "I hope LucD responds!" its like a famous guy hitting you up)

I actually have 2 jobs, 1 nested. If you look at master, it has a limited function and launches a job that starts worker.ps1 with some parameters. when it reaches the limit it waits for job completion. this works perfectly but the script stops around half way through and just hangs. so i commented it out and used a cmd /c start powershell -Command with get-task. I have some comments on this particular section in the code. regardless, in this particular portion i had removed the remove-job during testing and didnt put it back since it was commented.

The portion with the memory issue is in worker.ps1 where i use a job to launch windowsspec.ps1. now, to clarify, EVERYTHING works fine, the jobs just never close and i eventually run out of memory. The start-job and remove-job are both in worker.ps1.

write-host "Starting job"

start-job -FilePath $loc2 -ArgumentList @($csv1,$vmname,$vcserver,$loc) -RunAs32

write-host "job started"

get-job | wait-job | out-null

write-host "checking job"

Remove-Job -State Completed

write-host "removing job"

the problem in this case is that windowsspec.ps1 job will launch and run perfectly until it hits this point:

if($VM.Environment -eq "PF"){

$pfcredname = "ServerX\"+$cred.user

Get-OSCustomizationSpec $newspecname | Set-OSCustomizationSpec -DnsSuffix $DNSDomain -Domain $DNSDomain -DomainUsername $pfcredname -DomainPassword $cred.Password

}else{

Get-OSCustomizationSpec $newspecname | Set-OSCustomizationSpec -DnsSuffix $DNSDomain -Domain $DNSDomain -DomainUsername $cred.User -DomainPassword $cred.Password -confirm:$false

}

disconnect-viserver -confirm:$false;return;

once it completed the if statement, it doesnt continue, so no disconnect and no end of job which causes the worker.ps1 to hang at this point and never finish provisioning. if i dont remove the job, the sleep i put in works fine until i run out of mem.

hopefully this clears things up. thanks!

LucD · ‎10-23-2013

I see.

I have been trying to wrap my head around the code, it is quite a big and complex script :smileycool:

In the "master" you seem to start a PowerShell session for each VM in the CSV file.

In that PS session you then run the "worker" script.

Two questions here:

why do you start a new PowerShell session ? Couldn't you do this with the Start-Job ?
why do you start the PowerShell session with the NoExit parameter ? That way a PS session where the worker script has finished will stay in memory afaik

Now back to reading the code

Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference

drizuid · ‎10-23-2013

why do you start a new PowerShell session ? Couldn't you do this with the Start-Job ?

i want to do this with a job; i have it commented out in the code because it will execute about 3/4 of the worker.ps1 and then just hang indefinitely

why do you start the PowerShell session with the NoExit parameter ? That way a PS session where the worker script has finished will stay in memory afaik

ehhh i think i did this to debug and forgot to remove it :smileygrin: correcting! thanks