VMware Cloud Community
mathiase
Enthusiast
Enthusiast

Problem with PowerCLI and Threads / Jobs

Hi,

I have a script Deploy-VM.ps1 which takes couple of parameters and deploys a VM into a large environment accordingly. The complete deployment procedure for a single VM takes around 10 minutes as I e.g. have to wait for sysprep to complete and the VM to be available on the network to do some post-deployment stuff. The goal is to deploy many thousand VM and doing so sequentially will take around half a year :smileygrin:

So what I need is the Deploy-VM script to called multiple time so the tasks can run in parallel - say 10 at a time.

$job = {

        .\Deploy-VD.ps1 `

                -name $args[0] `

                -clusterName $args[1] `

                -folder $args[2] `

                -template $args[3] `

                -custSpecName $args[4] `

                -numCPU $args[5] `

                -memoryGB $args[6] `

                -diskGB $args[7] `

}

Write-Host "Beginning deployment of 5 VDs in parallel..."

Start-Job -ScriptBlock $job -ArgumentList 'vm001', ...

Start-Job -ScriptBlock $job -ArgumentList 'vm002', ...

Start-Job -ScriptBlock $job -ArgumentList 'vm003', ...

Start-Job -ScriptBlock $job -ArgumentList 'vm004', ...

Start-Job -ScriptBlock $job -ArgumentList 'vm005', ...

# wait for all jobs to complete

Get-Job | Wait-Job

Write-Host "Done!"

The problem I have with this this seems to be the vCenter connection. I can create it outside the job block but then it will not be there inside the block. I could add the Connect-VIServer inside the Deploy-VM.ps1 file but then a connection is created for EVERY deployment process and would require me to save the credentials in a variable and push them into the script via parameter.

Everything not every satisfying. Any ideas?

cheers

Mathias

-- I would like to change the world, but they won't give me the source code ...
Reply
0 Kudos
17 Replies
LucD
Leadership
Leadership

Have a look at Clinton's post called Multithreading PowerCLI

He shows how to reuse the session token in all the jobs that are spawned.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

mathiase
Enthusiast
Enthusiast

Hi, that helped alot - thanks! Still I need some further assistance:

This is the script that starts the threads. In this case 5 at a time.

$job = {

        Set-Location $args[0]

        .\Deploy-VM.ps1 -session $($args[1]) -vcserver $($args[2]) -name $($args[3]) -clusterName $($args[4]) -folder $($args[5]) -template $($args[6]) -custSpecName $($args[7]) -numCPU $($args[8]) -memoryGB $($args[9]) -diskGB $($args[10])

}

$currentPath = (Split-Path -parent $MyInvocation.MyCommand.Definition)

$session = $global:DefaultVIServer | %{ $_.sessionsecret }

$vcserver = $global:DefaultVIServer.ServiceUri.Host

1..5 | foreach {

        $id = "{0:D3}" -f $_

        Start-Job -ScriptBlock $job -ArgumentList $currentPath, $session, $vcserver, "vd$id", 'cluster1', 'folder1', 'template1', 'custspec1', '2', '4', '50'

}

Get-Job | Wait-Job

Write-Host "Done!"

Used standalone, the Deploy-VM script works like a charm:

PowerCLI C:\> $session = $global:DefaultVIServer.sessionsecret
PowerCLI C:\> $vcserver = $global:DefaultVIServer.ServiceUri.host
PowerCLI C:\> $session
52143172-c2a1-2086-e52c-53a0bb772b85
PowerCLI C:\> $vcserver
vc1.lab.invalid

Now I clear the DefautlVIServer variable so that the script will try to reconnect using the provided session.

PowerCLI C:\> $global:DefaultVIServer = ""

Now I execute the script:

PowerCLI C:\> .\Deploy-VD.ps1 $session $vcserver vm1 cluster1 folder1 template1 custSpec1 2 4 50vc1.lab.invalid

Name                      Port  User
----                           ----    ----
vc1.lab.invalid          443   Administrator
Staring deployment process: 07/26/2013 16:42:30
Finding best Datastore ...
Cloning VM ...

....

The process runs through an everything is good. Now let me show you what happend when I do it the threaded way:

PowerCLI C:\> .\Deploy-5VDs.ps1

Id              Name            State      HasMoreData     Location             Command
--              ----            -----      -----------     --------             -------
1               Job1            Running    True            localhost            ...


PowerCLI C:\> Receive-Job Job1

Name                           Port  User
----                           ----  ----
vc1.lab.invalid          443   Administrator
Staring deployment process: 07/26/2013 16:50:15
Finding best Datastore ...
Cloning VM ...

Looks like everything is fine ... unitl:

PowerCLI C:\> Get-Job

Id              Name            State      HasMoreData     Location             Command
--              ----            -----      -----------     --------             -------
1               Job1            Failed     False           localhost            ...

PowerCLI C:\> Receive-Job Job1
Receive-Job : The background process reported an error with the following message: .
At line:1 char:12
+ Receive-Job <<<<  Job1
    + CategoryInfo          : OperationStopped: (System.Manageme...emotingChildJob:PSRemotingChildJob) [Receive-Job], PSRemotingTrans
   portException
    + FullyQualifiedErrorId : JobFailure,Microsoft.PowerShell.Commands.ReceiveJobCommand

PowerCLI C:\>

What's happening here??? I also get the message as shown in the attachement.

cheers

Mathias

-- I would like to change the world, but they won't give me the source code ...
Reply
0 Kudos
LucD
Leadership
Leadership

The PSRemotingTransportException is strange, that normally indicates a problem with WinRM, but you are running the jobs locally.

It could be useful if you could catch that exception in the job, and then display the formatted error. Perhaps that gives some more clues about the cause of the problem.

$Error[0].Exception | Format-List *

I'll try to run something similar in my lab, and see if I get the same problem.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
mathiase
Enthusiast
Enthusiast

Hi, I am not quite sure where to put $Error[0].... in my Deploy-VM.ps1 script. Anywhere? Probably not. Exceptions in powercli are completely new to me Smiley Sad

-- I would like to change the world, but they won't give me the source code ...
Reply
0 Kudos
LucD
Leadership
Leadership

I would do that with a Try-Catch construct.

In the Try block you place your code, in the Catch the formatting of the $error variable.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
mathiase
Enthusiast
Enthusiast

Hi,

the try catch block did not turn up anything ... I am posting the (almost) complete code, maybe somebody sees something:

from Deploy-VM.ps1:

if(!$global:DefaultVIServer) {
        if(!(Get-PSSnapin | Where {$_.name -eq "vmware.vimautomation.core"}))
        {
                try
                {
                        Add-PSSnapin VMware.VimAutomation.Core -ea 0| out-null
                }
                catch
                {
                        throw "Could not load PowerCLI snapin"
                }
        }
        try
        {
                Connect-VIServer $vcserver -session $session
        }
        catch
        {
                throw "Failed to connect to VI server"
        }
}

$now = Get-Date

Write-Host "Staring deployment process: $now"

# find best datastore

Write-Host "Finding best Datastore ..."

$datastore = FindDatastoreForVM $clusterName $diskGB

if (!$datastore)

{

        Write-Host "No datastore could be found. VD cannot be deployed! Aborting."

        return

}

# fetch the OS customization spec

$custSpec = Get-OSCustomizationSpec $custSpecName -ErrorAction:Stop

if (!$custSpec)

{

        Write-Host "Guest OS customization specification could not be found. Aborting."

        return

}

Write-Host "Cloning VM ..."

try {

        $vm = New-VM `

                -Name $name `

                -ResourcePool $clusterName `

                -Location $folder `

                -Datastore $datastore `

                -Template $template `

                -OSCustomizationSpec $custSpec `

                -ErrorAction:Stop

}

catch

{

        Write-Host "An error ocurred during template clone operation:"

        # output all exception information

        $_ | fl

        Write-Host "Cleaning up ..."

        # clean up and exit

        $exists = Get-VM -Name $name -ErrorAction SilentlyContinue

        If ($Exists){

                Remove-VM -VM $exists -DeletePermanently

        }

        return

}

# make hardware adjustments

Write-Host "Changing virtual hardware configuration ..."

ChangeVMHardware $vm $numCPU $memoryGB $diskGB

# start VM and wait for it

Write-Host "Staring VM and waiting for Sysprep to complete ..."

$sysprep = StartVMAndWaitForSysprep $vm

if (!$sysprep)

{

        Write-Host "Guest Customization failed.... Aborting."

        $exists = Get-VM -Name $name -ErrorAction SilentlyContinue

        If ($Exists){

                Remove-VM -VM $exists -DeletePermanently

        }

}

# wait until the vm shows on the network

Write-Host "Waiting for VMware Tools to report IP address ..."

$hasIp =   $vm

$now = Get-Date

Write-Host "Deployment finished: $now"

return $vm

The functions used above (ChangeVMHardware, StartVMAndWaitForSysprep, WaitForToolsToReportIP) are not shown above, as they take a lot of space and I can't see why they should be causing the error I am seeing. But if you think they might, let me know and I'll post them, too.

Control Script:

$job = {
        Set-Location $args[0]
        .\Deploy-VD.ps1 -session $($args[1]) -vcserver $($args[2]) -name $($args[3]) -clusterName $($args[4]) -folder $($args[5]) -template $($args[6]) -custSpecName $($args[7]) -numCPU $($args[8]) -memoryGB $($args[9]) -diskGB $($args[10])
}

$currentPath = (Split-Path -parent $MyInvocation.MyCommand.Definition)
$session = $global:DefaultVIServer | %{ $_.sessionsecret }
$vcserver = $global:DefaultVIServer.ServiceUri.Host

Write-Host "CURRENTPATH = $currentPath"
Write-Host "SESSION = $session"
Write-Host "VCSERVER = $vcserver"


1..1 | foreach {
        $id = "{0:D3}" -f $_
        Start-Job -ScriptBlock $job -ArgumentList $currentPath, $session, $vcserver, "vd$id", 'cluster01', 'folder1', 'VMTPL', 'default', '2', '4', '50'
}

Get-Job | Wait-Job
Write-Host "Done!"

I put

try

{

}

catch

{

     $Error[0].Exception | Format-List *

}

around all the code from Deploy-VM.ps1 I posted and did see anything I haven't already seen before.

cheers
Mathias

-- I would like to change the world, but they won't give me the source code ...
Reply
0 Kudos
mathiase
Enthusiast
Enthusiast

I just created a little test just to test the general functionality of threads:

thread-ctrl.ps1

$job = {

    Set-Location $args[0]

    .\thread.ps1 -name $($args[1])

}

$currentPath = (Split-Path -parent $MyInvocation.MyCommand.Definition)

1..10 | foreach {

    $id = "{0:D3}" -f $_

    Start-Job -ScriptBlock $job -ArgumentList $currentPath, "thread-$id"

}

Get-Job | Wait-Job

Write-Host "Done!"

thread.ps1

Param (

    [Parameter(Mandatory=$True)]
    [String]$name
)

Write-Host "Hi, my name is thread $name"

sleep -seconds 5

Output:

PowerCLI C:\scripts> .\thread-ctrl.ps1

Id              Name            State      HasMoreData     Location             Command
--              ----            -----      -----------     --------             -------
101             Job101          Running    True            localhost            ...
103             Job103          Running    True            localhost            ...
105             Job105          Running    True            localhost            ...
107             Job107          Running    True            localhost            ...
109             Job109          Running    True            localhost            ...
111             Job111          Running    True            localhost            ...
113             Job113          Running    True            localhost            ...
115             Job115          Running    True            localhost            ...
117             Job117          Running    True            localhost            ...
119             Job119          Running    True            localhost            ...
119             Job119          Completed  True            localhost            ...
117             Job117          Completed  True            localhost            ...
115             Job115          Completed  True            localhost            ...
113             Job113          Completed  True            localhost            ...
111             Job111          Completed  True            localhost            ...
109             Job109          Completed  True            localhost            ...
107             Job107          Completed  True            localhost            ...
105             Job105          Completed  True            localhost            ...
103             Job103          Completed  True            localhost            ...
101             Job101          Completed  True            localhost            ...
Done!


PowerCLI C:\scripts> Get-Job | Receive-Job
Hi, my name is thread thread-010
Hi, my name is thread thread-009
Hi, my name is thread thread-008
Hi, my name is thread thread-007
Hi, my name is thread thread-006
Hi, my name is thread thread-005
Hi, my name is thread thread-004
Hi, my name is thread thread-003
Hi, my name is thread thread-002
Hi, my name is thread thread-001

Works! So it has to be something with my other code ...

cheers

Mathias

-- I would like to change the world, but they won't give me the source code ...
Reply
0 Kudos
LucD
Leadership
Leadership

Interesting, can you now just add the Add-PSSnapin and the Connect-VIServer ?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
mathiase
Enthusiast
Enthusiast

Sure:

$job = {
    Set-Location $args[0]
    .\thread.ps1 -session $($args[1]) -vcserver $($args[2]) -name $($args[3])
}

$currentPath = (Split-Path -parent $MyInvocation.MyCommand.Definition)
$session = $global:DefaultVIServer | %{ $_.sessionsecret }
$vcserver = $global:DefaultVIServer.ServiceUri.Host

1..1 | foreach {
    $id = "{0:D3}" -f $_
    Start-Job -ScriptBlock $job -ArgumentList $currentPath, $session, $vcserver, "thread-$id"
}

Get-Job | Wait-Job
Write-Host "Done!"

Param (
    [Parameter(Mandatory=$True)]
    $session,

    [Parameter(Mandatory=$True)]
    [String]$vcserver,

    [Parameter(Mandatory=$True)]
    [String]$name
)


Write-Host "Hi, my name is thread $name"

if(!$global:DefaultVIServer) {
    if(!(Get-PSSnapin | Where {$_.name -eq "vmware.vimautomation.core"}))
    {
        try
        {
            Write-Host "Initializing powercli"
            Add-PSSnapin VMware.VimAutomation.Core -ea 0 | out-null
        }
        catch
        {
            throw "Could not load PowerCLI snapin"
        }
        }
       try
    {
        Write-Host "Resuming session"
        $conn = Connect-VIServer $vcserver -session $session

        Write-Host "Done!"
    }
    catch
    {
            throw "Failed to connect to VI server"
        }
}

sleep -seconds 2

Output:

PowerCLI C:\scripts> .\thread-ctrl.ps1

Id              Name            State      HasMoreData     Location             Command
--              ----            -----      -----------     --------             -------
1               Job1            Running    True            localhost            ...
PowerCLI C:\scripts> Get-Job

Id              Name            State      HasMoreData     Location             Command
--              ----            -----      -----------     --------             -------
1               Job1            Running    True            localhost            ...


PowerCLI C:\scripts> Get-Job

Id              Name            State      HasMoreData     Location             Command
--              ----            -----      -----------     --------             -------
1               Job1            Completed  False           localhost            ...

PowerCLI C:\scripts>

PowerCLI C:\scripts> Receive-Job Job1
Hi, my name is thread thread-001
Initializing powercli
Resuming session
WARNING: There were one or more problems with the server certificate:

* The X509 chain could not be built up to the root certificate.

* The certificate's CN name does not match the passed value.

Certificate: [Subject]
  E=support@vmware.com, CN=VMware default certificate, OU=vCenterServer_2013.07.19_111323, O="VMware, Inc."

[Issuer]
  E=support@vmware.com, CN=vc.lab.invalid o, OU=vCenterServer_2013.07.19_111323, O="VMware, Inc."

[Serial Number]
  100002

[Not Before]
  7/18/2013 5:48:23 PM

[Not After]
  7/17/2023 5:48:25 PM

[Thumbprint]
  8C3341076CCB4D257A01B4BD6B4FE55C0DFA1639

Done!

As you can see: no error / exception... It takes a while but finally all jobs show completed and receive-job doesn't show any errors.

-- I would like to change the world, but they won't give me the source code ...
Reply
0 Kudos
LucD
Leadership
Leadership

Perhaps it's not actually the code in the thread.ps1, but the number of these you have running.

Since the error seems to be a PowerShell error, it could be that the PowerShell engine, in your environment, can't cope with that number of background jobs.

Perhaps you should increase the number of background jobs gradually (first 2, then 3....) and see when the problem occurs.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
roxton_
Contributor
Contributor

Hi!

I'm having a similar problem using the PowerCLI 5.5

I've tried to execute my scripts on both Windows Server 2008 R2 and 2012.

My scripts are in the attachments to this message.

A little bit of details:

Scripts are used to deploy multiple machines at a time from their templates

All the information about the machines is stored in the environment.xml and user has to edit it first so the scripts could get all necessary information from it.

Execution starts from the vCenterAutodeployment.ps1 which gets the info from the environment.xml and creates jobs for every single machine deployment (you can use Run.bat or Run.ps1 to start).

vCenterFunctions.ps1 contain a logic for a single machine deployment.

Functions.ps1 also contains some logic for after-deployment procedures.

Logging.ps1 creates and fills log files

PROBLEM:

Everything works great until it's time to execute a commandlet which execution time is at least 5 seconds long (e.g. New-VM or Stop-VM, start the exectution and watch the logs). If I don't use Jobs these commandlets work great.

Could someone help me with that?

Reply
0 Kudos
roxton_
Contributor
Contributor

Solution found: Start-Job -RunAs32 -PSVersion 2.0

Do not forget to set the executionPolicy on the 32 bit powershell/powercli

Reply
0 Kudos
jimbeam128
Contributor
Contributor

Hi everybody / Luc / Mattias,

I also would like to use PowerCLI in Jobs to deserialize my tasks.

I also tried to implement it in my script an got stuck.

Luc, you said that there is a solution to reuse the session token which is pawend over all jobs.

Unfortunately the link does not work any more.

How did you manage to get it working?

I also don´t want to create a new session in every job. I would like to reuse the initial established connection session to my vcenter server...

Reply
0 Kudos
LucD
Leadership
Leadership

Did you try like this?

Note that you need to connect to the vCenter before running this

$server = [uri]::new($global:DefaultVIServer.ServiceUri).Host

$id = $global:DefaultVIServer | select -ExpandProperty SessionId

$job = {

    param($serverName,$sessionId)

    Connect-VIServer -Server $serverName -Session $sessionId > $null

    Get-VMHost | select Name

}

1..3 | %{Start-Job -Name "Test$($_)" -ScriptBlock $job -ArgumentList $server,$id}

Receive-Job -Name "Test*" -Wait


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
jimbeam128
Contributor
Contributor

Hi Luc,

thanks for your quick reply.

I just tried it right now as you adviced:

cls

connect-VIServer -server $vcenterserver -Credential $MyCredential -force

$server = [uri]::new($global:DefaultVIServer.ServiceUri).Host

$id = $global:DefaultVIServer | select -ExpandProperty SessionId

$job = {

    param($serverName,$sessionId)

   Add-PSSnapin VMware.VimAutomation.Core

    Connect-VIServer -Server $serverName -Session $sessionId > $null

    Get-VMHost | select Name

}

1..3 | %{Start-Job -Name "Test$($_)" -ScriptBlock $job -ArgumentList $server,$id}

Receive-Job -Name "Test*" -Wait

In General it works. - But I had to load the module in the job (marked in red) to get it to work. - I know that inside the job, it´s a different scope and therefore the module is not available unless I integrate it into my Powershell - Profile... But even then the Module would be loaded in every Job / Scope.

Do I understand it correctly that there is no way to load the module once and connect to the vcenter - Server once?

Even when I execute in the job

Connect-VIServer -Server $serverName -Session $sessionId > $null

I connect to the same session, but I connect "once again".

I would like to reduce the "overhead" - But I have to execute the tasks asynchronously

Reply
0 Kudos
LucD
Leadership
Leadership

When you go for PowerCLI 6.5.1 or later, there are no more PSSnapin, and with the module auto-load feature in PowerShell, you don't have to load anything anymore.

Provided the PowerCLI modules are located in one of the folders listed in $env:PSModulePath.

Yes, the "job" runs in another thread, which doesn't inherit anything from the thread where you started the job.

With the sessionId you kind of piggy-back on the existing session in the caller's thread.

No, there is, afaik, no way to avoid this Connect-VIServer line in a job.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
jimbeam128
Contributor
Contributor

OK, so I will take the reference by Session ID.

That would do the "job" for me. 🙂

Thanks for your support!

Reply
0 Kudos