Hi,
I have a script Deploy-VM.ps1 which takes couple of parameters and deploys a VM into a large environment accordingly. The complete deployment procedure for a single VM takes around 10 minutes as I e.g. have to wait for sysprep to complete and the VM to be available on the network to do some post-deployment stuff. The goal is to deploy many thousand VM and doing so sequentially will take around half a year :smileygrin:
So what I need is the Deploy-VM script to called multiple time so the tasks can run in parallel - say 10 at a time.
$job = {
.\Deploy-VD.ps1 `
-name $args[0] `
-clusterName $args[1] `
-folder $args[2] `
-template $args[3] `
-custSpecName $args[4] `
-numCPU $args[5] `
-memoryGB $args[6] `
-diskGB $args[7] `
}
Write-Host "Beginning deployment of 5 VDs in parallel..."
Start-Job -ScriptBlock $job -ArgumentList 'vm001', ...
Start-Job -ScriptBlock $job -ArgumentList 'vm002', ...
Start-Job -ScriptBlock $job -ArgumentList 'vm003', ...
Start-Job -ScriptBlock $job -ArgumentList 'vm004', ...
Start-Job -ScriptBlock $job -ArgumentList 'vm005', ...
# wait for all jobs to complete
Get-Job | Wait-Job
Write-Host "Done!"
The problem I have with this this seems to be the vCenter connection. I can create it outside the job block but then it will not be there inside the block. I could add the Connect-VIServer inside the Deploy-VM.ps1 file but then a connection is created for EVERY deployment process and would require me to save the credentials in a variable and push them into the script via parameter.
Everything not every satisfying. Any ideas?
cheers
Mathias
Have a look at Clinton's post called Multithreading PowerCLI
He shows how to reuse the session token in all the jobs that are spawned.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Hi, that helped alot - thanks! Still I need some further assistance:
This is the script that starts the threads. In this case 5 at a time.
$job = {
Set-Location $args[0]
.\Deploy-VM.ps1 -session $($args[1]) -vcserver $($args[2]) -name $($args[3]) -clusterName $($args[4]) -folder $($args[5]) -template $($args[6]) -custSpecName $($args[7]) -numCPU $($args[8]) -memoryGB $($args[9]) -diskGB $($args[10])
}
$currentPath = (Split-Path -parent $MyInvocation.MyCommand.Definition)
$session = $global:DefaultVIServer | %{ $_.sessionsecret }
$vcserver = $global:DefaultVIServer.ServiceUri.Host
1..5 | foreach {
$id = "{0:D3}" -f $_
Start-Job -ScriptBlock $job -ArgumentList $currentPath, $session, $vcserver, "vd$id", 'cluster1', 'folder1', 'template1', 'custspec1', '2', '4', '50'
}
Get-Job | Wait-Job
Write-Host "Done!"
Used standalone, the Deploy-VM script works like a charm:
PowerCLI C:\> $session = $global:DefaultVIServer.sessionsecret
PowerCLI C:\> $vcserver = $global:DefaultVIServer.ServiceUri.host
PowerCLI C:\> $session
52143172-c2a1-2086-e52c-53a0bb772b85
PowerCLI C:\> $vcserver
vc1.lab.invalid
Now I clear the DefautlVIServer variable so that the script will try to reconnect using the provided session.
PowerCLI C:\> $global:DefaultVIServer = ""
Now I execute the script:
PowerCLI C:\> .\Deploy-VD.ps1 $session $vcserver vm1 cluster1 folder1 template1 custSpec1 2 4 50vc1.lab.invalid
Name Port User
---- ---- ----
vc1.lab.invalid 443 Administrator
Staring deployment process: 07/26/2013 16:42:30
Finding best Datastore ...
Cloning VM .......
The process runs through an everything is good. Now let me show you what happend when I do it the threaded way:
PowerCLI C:\> .\Deploy-5VDs.ps1
Id Name State HasMoreData Location Command
-- ---- ----- ----------- -------- -------
1 Job1 Running True localhost ...
PowerCLI C:\> Receive-Job Job1Name Port User
---- ---- ----
vc1.lab.invalid 443 Administrator
Staring deployment process: 07/26/2013 16:50:15
Finding best Datastore ...
Cloning VM ...
Looks like everything is fine ... unitl:
PowerCLI C:\> Get-Job
Id Name State HasMoreData Location Command
-- ---- ----- ----------- -------- -------
1 Job1 Failed False localhost ...PowerCLI C:\> Receive-Job Job1
Receive-Job : The background process reported an error with the following message: .
At line:1 char:12
+ Receive-Job <<<< Job1
+ CategoryInfo : OperationStopped: (System.Manageme...emotingChildJob:PSRemotingChildJob) [Receive-Job], PSRemotingTrans
portException
+ FullyQualifiedErrorId : JobFailure,Microsoft.PowerShell.Commands.ReceiveJobCommandPowerCLI C:\>
What's happening here??? I also get the message as shown in the attachement.
cheers
Mathias
The PSRemotingTransportException is strange, that normally indicates a problem with WinRM, but you are running the jobs locally.
It could be useful if you could catch that exception in the job, and then display the formatted error. Perhaps that gives some more clues about the cause of the problem.
$Error[0].Exception | Format-List *
I'll try to run something similar in my lab, and see if I get the same problem.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Hi, I am not quite sure where to put $Error[0].... in my Deploy-VM.ps1 script. Anywhere? Probably not. Exceptions in powercli are completely new to me
I would do that with a Try-Catch construct.
In the Try block you place your code, in the Catch the formatting of the $error variable.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Hi,
the try catch block did not turn up anything ... I am posting the (almost) complete code, maybe somebody sees something:
from Deploy-VM.ps1:
if(!$global:DefaultVIServer) {
if(!(Get-PSSnapin | Where {$_.name -eq "vmware.vimautomation.core"}))
{
try
{
Add-PSSnapin VMware.VimAutomation.Core -ea 0| out-null
}
catch
{
throw "Could not load PowerCLI snapin"
}
}
try
{
Connect-VIServer $vcserver -session $session
}
catch
{
throw "Failed to connect to VI server"
}
}$now = Get-Date
Write-Host "Staring deployment process: $now"
# find best datastore
Write-Host "Finding best Datastore ..."
$datastore = FindDatastoreForVM $clusterName $diskGB
if (!$datastore)
{
Write-Host "No datastore could be found. VD cannot be deployed! Aborting."
return
}
# fetch the OS customization spec
$custSpec = Get-OSCustomizationSpec $custSpecName -ErrorAction:Stop
if (!$custSpec)
{
Write-Host "Guest OS customization specification could not be found. Aborting."
return
}
Write-Host "Cloning VM ..."
try {
$vm = New-VM `
-Name $name `
-ResourcePool $clusterName `
-Location $folder `
-Datastore $datastore `
-Template $template `
-OSCustomizationSpec $custSpec `
-ErrorAction:Stop
}
catch
{
Write-Host "An error ocurred during template clone operation:"
# output all exception information
$_ | fl
Write-Host "Cleaning up ..."
# clean up and exit
$exists = Get-VM -Name $name -ErrorAction SilentlyContinue
If ($Exists){
Remove-VM -VM $exists -DeletePermanently
}
return
}
# make hardware adjustments
Write-Host "Changing virtual hardware configuration ..."
ChangeVMHardware $vm $numCPU $memoryGB $diskGB
# start VM and wait for it
Write-Host "Staring VM and waiting for Sysprep to complete ..."
$sysprep = StartVMAndWaitForSysprep $vm
if (!$sysprep)
{
Write-Host "Guest Customization failed.... Aborting."
$exists = Get-VM -Name $name -ErrorAction SilentlyContinue
If ($Exists){
Remove-VM -VM $exists -DeletePermanently
}
}
# wait until the vm shows on the network
Write-Host "Waiting for VMware Tools to report IP address ..."
$hasIp = $vm
$now = Get-Date
Write-Host "Deployment finished: $now"
return $vm
The functions used above (ChangeVMHardware, StartVMAndWaitForSysprep, WaitForToolsToReportIP) are not shown above, as they take a lot of space and I can't see why they should be causing the error I am seeing. But if you think they might, let me know and I'll post them, too.
Control Script:
$job = {
Set-Location $args[0]
.\Deploy-VD.ps1 -session $($args[1]) -vcserver $($args[2]) -name $($args[3]) -clusterName $($args[4]) -folder $($args[5]) -template $($args[6]) -custSpecName $($args[7]) -numCPU $($args[8]) -memoryGB $($args[9]) -diskGB $($args[10])
}$currentPath = (Split-Path -parent $MyInvocation.MyCommand.Definition)
$session = $global:DefaultVIServer | %{ $_.sessionsecret }
$vcserver = $global:DefaultVIServer.ServiceUri.HostWrite-Host "CURRENTPATH = $currentPath"
Write-Host "SESSION = $session"
Write-Host "VCSERVER = $vcserver"
1..1 | foreach {
$id = "{0:D3}" -f $_
Start-Job -ScriptBlock $job -ArgumentList $currentPath, $session, $vcserver, "vd$id", 'cluster01', 'folder1', 'VMTPL', 'default', '2', '4', '50'
}Get-Job | Wait-Job
Write-Host "Done!"
I put
try
{
}
catch
{
$Error[0].Exception | Format-List *
}
around all the code from Deploy-VM.ps1 I posted and did see anything I haven't already seen before.
cheers
Mathias
I just created a little test just to test the general functionality of threads:
thread-ctrl.ps1
$job = {
Set-Location $args[0]
.\thread.ps1 -name $($args[1])
}
$currentPath = (Split-Path -parent $MyInvocation.MyCommand.Definition)
1..10 | foreach {
$id = "{0:D3}" -f $_
Start-Job -ScriptBlock $job -ArgumentList $currentPath, "thread-$id"
}
Get-Job | Wait-Job
Write-Host "Done!"
thread.ps1
Param (
[Parameter(Mandatory=$True)]
[String]$name
)Write-Host "Hi, my name is thread $name"
sleep -seconds 5
Output:
PowerCLI C:\scripts> .\thread-ctrl.ps1
Id Name State HasMoreData Location Command
-- ---- ----- ----------- -------- -------
101 Job101 Running True localhost ...
103 Job103 Running True localhost ...
105 Job105 Running True localhost ...
107 Job107 Running True localhost ...
109 Job109 Running True localhost ...
111 Job111 Running True localhost ...
113 Job113 Running True localhost ...
115 Job115 Running True localhost ...
117 Job117 Running True localhost ...
119 Job119 Running True localhost ...
119 Job119 Completed True localhost ...
117 Job117 Completed True localhost ...
115 Job115 Completed True localhost ...
113 Job113 Completed True localhost ...
111 Job111 Completed True localhost ...
109 Job109 Completed True localhost ...
107 Job107 Completed True localhost ...
105 Job105 Completed True localhost ...
103 Job103 Completed True localhost ...
101 Job101 Completed True localhost ...
Done!
PowerCLI C:\scripts> Get-Job | Receive-Job
Hi, my name is thread thread-010
Hi, my name is thread thread-009
Hi, my name is thread thread-008
Hi, my name is thread thread-007
Hi, my name is thread thread-006
Hi, my name is thread thread-005
Hi, my name is thread thread-004
Hi, my name is thread thread-003
Hi, my name is thread thread-002
Hi, my name is thread thread-001
Works! So it has to be something with my other code ...
cheers
Mathias
Interesting, can you now just add the Add-PSSnapin and the Connect-VIServer ?
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Sure:
$job = {
Set-Location $args[0]
.\thread.ps1 -session $($args[1]) -vcserver $($args[2]) -name $($args[3])
}
$currentPath = (Split-Path -parent $MyInvocation.MyCommand.Definition)
$session = $global:DefaultVIServer | %{ $_.sessionsecret }
$vcserver = $global:DefaultVIServer.ServiceUri.Host
1..1 | foreach {
$id = "{0:D3}" -f $_
Start-Job -ScriptBlock $job -ArgumentList $currentPath, $session, $vcserver, "thread-$id"
}
Get-Job | Wait-Job
Write-Host "Done!"
Param (
[Parameter(Mandatory=$True)]
$session,[Parameter(Mandatory=$True)]
[String]$vcserver,[Parameter(Mandatory=$True)]
[String]$name
)
Write-Host "Hi, my name is thread $name"if(!$global:DefaultVIServer) {
if(!(Get-PSSnapin | Where {$_.name -eq "vmware.vimautomation.core"}))
{
try
{
Write-Host "Initializing powercli"
Add-PSSnapin VMware.VimAutomation.Core -ea 0 | out-null
}
catch
{
throw "Could not load PowerCLI snapin"
}
}
try
{
Write-Host "Resuming session"
$conn = Connect-VIServer $vcserver -session $sessionWrite-Host "Done!"
}
catch
{
throw "Failed to connect to VI server"
}
}sleep -seconds 2
Output:
PowerCLI C:\scripts> .\thread-ctrl.ps1
Id Name State HasMoreData Location Command
-- ---- ----- ----------- -------- -------
1 Job1 Running True localhost ...
PowerCLI C:\scripts> Get-Job
Id Name State HasMoreData Location Command
-- ---- ----- ----------- -------- -------
1 Job1 Running True localhost ...
PowerCLI C:\scripts> Get-Job
Id Name State HasMoreData Location Command
-- ---- ----- ----------- -------- -------
1 Job1 Completed False localhost ...
PowerCLI C:\scripts>
PowerCLI C:\scripts> Receive-Job Job1
Hi, my name is thread thread-001
Initializing powercli
Resuming session
WARNING: There were one or more problems with the server certificate:
* The X509 chain could not be built up to the root certificate.
* The certificate's CN name does not match the passed value.
Certificate: [Subject]
E=support@vmware.com, CN=VMware default certificate, OU=vCenterServer_2013.07.19_111323, O="VMware, Inc."
[Issuer]
E=support@vmware.com, CN=vc.lab.invalid o, OU=vCenterServer_2013.07.19_111323, O="VMware, Inc."
[Serial Number]
100002
[Not Before]
7/18/2013 5:48:23 PM
[Not After]
7/17/2023 5:48:25 PM
[Thumbprint]
8C3341076CCB4D257A01B4BD6B4FE55C0DFA1639Done!
As you can see: no error / exception... It takes a while but finally all jobs show completed and receive-job doesn't show any errors.
Perhaps it's not actually the code in the thread.ps1, but the number of these you have running.
Since the error seems to be a PowerShell error, it could be that the PowerShell engine, in your environment, can't cope with that number of background jobs.
Perhaps you should increase the number of background jobs gradually (first 2, then 3....) and see when the problem occurs.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Hi!
I'm having a similar problem using the PowerCLI 5.5
I've tried to execute my scripts on both Windows Server 2008 R2 and 2012.
My scripts are in the attachments to this message.
A little bit of details:
Scripts are used to deploy multiple machines at a time from their templates
All the information about the machines is stored in the environment.xml and user has to edit it first so the scripts could get all necessary information from it.
Execution starts from the vCenterAutodeployment.ps1 which gets the info from the environment.xml and creates jobs for every single machine deployment (you can use Run.bat or Run.ps1 to start).
vCenterFunctions.ps1 contain a logic for a single machine deployment.
Functions.ps1 also contains some logic for after-deployment procedures.
Logging.ps1 creates and fills log files
PROBLEM:
Everything works great until it's time to execute a commandlet which execution time is at least 5 seconds long (e.g. New-VM or Stop-VM, start the exectution and watch the logs). If I don't use Jobs these commandlets work great.
Could someone help me with that?
Solution found: Start-Job -RunAs32 -PSVersion 2.0
Do not forget to set the executionPolicy on the 32 bit powershell/powercli
Hi everybody / Luc / Mattias,
I also would like to use PowerCLI in Jobs to deserialize my tasks.
I also tried to implement it in my script an got stuck.
Luc, you said that there is a solution to reuse the session token which is pawend over all jobs.
Unfortunately the link does not work any more.
How did you manage to get it working?
I also don´t want to create a new session in every job. I would like to reuse the initial established connection session to my vcenter server...
Did you try like this?
Note that you need to connect to the vCenter before running this
$server = [uri]::new($global:DefaultVIServer.ServiceUri).Host
$id = $global:DefaultVIServer | select -ExpandProperty SessionId
$job = {
param($serverName,$sessionId)
Connect-VIServer -Server $serverName -Session $sessionId > $null
Get-VMHost | select Name
}
1..3 | %{Start-Job -Name "Test$($_)" -ScriptBlock $job -ArgumentList $server,$id}
Receive-Job -Name "Test*" -Wait
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Hi Luc,
thanks for your quick reply.
I just tried it right now as you adviced:
cls
connect-VIServer -server $vcenterserver -Credential $MyCredential -force
$server = [uri]::new($global:DefaultVIServer.ServiceUri).Host
$id = $global:DefaultVIServer | select -ExpandProperty SessionId
$job = {
param($serverName,$sessionId)
Add-PSSnapin VMware.VimAutomation.Core
Connect-VIServer -Server $serverName -Session $sessionId > $null
Get-VMHost | select Name
}
1..3 | %{Start-Job -Name "Test$($_)" -ScriptBlock $job -ArgumentList $server,$id}
Receive-Job -Name "Test*" -Wait
In General it works. - But I had to load the module in the job (marked in red) to get it to work. - I know that inside the job, it´s a different scope and therefore the module is not available unless I integrate it into my Powershell - Profile... But even then the Module would be loaded in every Job / Scope.
Do I understand it correctly that there is no way to load the module once and connect to the vcenter - Server once?
Even when I execute in the job
Connect-VIServer -Server $serverName -Session $sessionId > $null
I connect to the same session, but I connect "once again".
I would like to reduce the "overhead" - But I have to execute the tasks asynchronously
When you go for PowerCLI 6.5.1 or later, there are no more PSSnapin, and with the module auto-load feature in PowerShell, you don't have to load anything anymore.
Provided the PowerCLI modules are located in one of the folders listed in $env:PSModulePath.
Yes, the "job" runs in another thread, which doesn't inherit anything from the thread where you started the job.
With the sessionId you kind of piggy-back on the existing session in the caller's thread.
No, there is, afaik, no way to avoid this Connect-VIServer line in a job.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
OK, so I will take the reference by Session ID.
That would do the "job" for me. 🙂
Thanks for your support!