11 Replies Latest reply on Jul 19, 2019 4:06 AM by LucD

    Execute multiple Powershell tasks in parallel

    NikhilPathak Novice

      Currently scenario:

      Read an xml file and Deploy VM's, Customize them and then invoke Installation.

      The above process takes almost 25 minutes for each VM. So in a foreach loop if i have 4 VM's, it takes around 1.5 ~ 2 hours

       

      Is there any way to perform the task in parallel and reduce the execution time.

        • 1. Re: Execute multiple Powershell tasks in parallel
          LucD Guru
          vExpertCommunity WarriorsUser Moderators

          Yes, you could use background jobs (with Start-Job).

          • 2. Re: Execute multiple Powershell tasks in parallel
            NikhilPathak Novice

            I tried using workflow as below,

             

            $VM = "R511-SRVA-WyU3", "R511-SRVB-0DV1"

             

            workflow pbatch{

            param ([string[]]$VM)

            foreach -parallel($a in $VM){

            $installscript = "hostname"

            $invokeinstalltask = Invoke-VMScript -ScriptText $installscript -VM $a -GuestUser "abc" -GuestPassword "Pass" -RunAsync

            }

            }

            pbatch -VM $VM

             

            but it gives me error as " Invoke-VMScript You are not currently connected to any servers. Please connect first using a Connect cmdlet. ".

            The error is strange because i dont get this error if i run it in a simple foreach loop.

             

            I Tried using Jobs as well with below example but the JOBs didn't stop even after 2 hrs. Ideally, the Get VM should not take 2 hrs when the VM name is provided.

             

            $abc = "R511-SRVA-WyU3", "R511-SRVB-0DV1"

            $block = {

            param ([string[]]$a)

            Get-VM -Name $a

            }

            $MaxThreads = 2

            #Remove all jobs

            Get-Job | Remove-Job

            foreach($a in $abc){

                While ($(Get-Job -state running).count -ge $MaxThreads){

                    Start-Sleep -Milliseconds 3

                }

                Start-Job -Scriptblock $Block -ArgumentList $a

            }

             

             

            #Wait for all jobs to finish.

            While ($(Get-Job -State Running).count -gt 0){

                start-sleep 1

            }

             

             

            #Get information from each job.

            foreach($job in Get-Job){

                $info= Receive-Job -Id ($job.Id)

            }

            #Remove all jobs created.

            Get-Job | Remove-Job

            • 3. Re: Execute multiple Powershell tasks in parallel
              LucD Guru
              Community WarriorsUser ModeratorsvExpert

              To use background jobs, you have to reconnect inside the job.

              See my Running a background job.

              • 4. Re: Execute multiple Powershell tasks in parallel
                NikhilPathak Novice

                Hi LucD, i tried the way you did parallel jobs it works fine with powercli functions, but i am not able to use my custom function.

                 

                 

                it fails with error as

                 

                The term 'Get-vspherevmname' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify

                that the path is correct and try again.

                    + CategoryInfo          : ObjectNotFound: (Get-vspherevmname:String) [], CommandNotFoundException

                    + FullyQualifiedErrorId : CommandNotFoundException

                    + PSComputerName        : localhost

                 

                 

                Can you please help?

                • 5. Re: Execute multiple Powershell tasks in parallel
                  LucD Guru
                  User ModeratorsvExpertCommunity Warriors

                  Where and how do you define your function?
                  The job environment is a new PS environment, you might have to dot-source the file that contains your function inside the job.

                  • 6. Re: Execute multiple Powershell tasks in parallel
                    NikhilPathak Novice

                    currently i have the Function created as a separate ps1.

                    I run that ps1 file before executing my main script.

                     

                    Sorry i am not an expert on Powershell, but your guidance will help..

                     

                    Do you have any example where you have executed custom Function in parallel..

                    • 7. Re: Execute multiple Powershell tasks in parallel
                      LucD Guru
                      User ModeratorsvExpertCommunity Warriors

                      You have to make your function known to the PS engine.
                      The PS engine in a background job can be considered as new.

                      So all functions that the calling script knows, are not known in the background job.

                      That is why you need to dot-source the .ps1 file inside the background job.
                      In the below example I assume the file is called myfunction.ps1 which contains the function Do-MyFunction.
                      Note the space between the 2 dots when dot-sourcing a .ps1 file.

                       

                      $block = {

                         param ([string[]]$a)


                         . .\myfunction.ps1

                         Do-MyFunction

                      }

                      • 8. Re: Execute multiple Powershell tasks in parallel
                        NikhilPathak Novice

                        Hi LucD, I am still getting the error as, "Get-CIVM You are not currently connected to any servers. Please connect first using a Connect cmdlet."

                         

                        Below is the code where i am trying the Job..  The get-civm is used in my function. Can you please check...

                        #-----------------------------------------------------------------------------------------------------------------------------------------------

                        $code = {

                            param(

                                [string]$Server,

                                [string]$SessionId,

                                [string]$VMName,

                                [string]$org,

                                [string]$vapp

                            )

                         

                            Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false | Out-Null

                            Connect-VIServer -Server $Server -Session $SessionId

                            . C:\Migration_Powershell_Scripts\New_functions\Get-vspherevmname.ps1

                            get-vspherevmname -vmname $VMName -org $org -vapp $vapp -Server $Server -SessionId $SessionId

                        }

                         

                        $vmName = 'R511-SRVA'

                        $org = "ID14434-1-1231231"

                        $vapp = "12345"

                        $sJOb = @{

                            ScriptBlock = $code

                            ArgumentList = $global:DefaultVIServer.Name, $global:DefaultVIServer.SessionId, $vmName, $org, $vapp

                        }

                         

                        $ja = Start-Job @sJob

                        #-----------------------------------------------------------------------------------------------------------------------------------------------------------

                         

                        My Custom Function

                         

                        Function Get-vspherevmname{

                            [CmdletBinding()]

                            [OutputType([psobject])] # Select the output type of this function as "string/PSObject/Int"

                         

                         

                        # Enter the list of Input Parameters for the function

                        #The Parameter should have properties as "Mandatory", HelpMessage

                        Param (

                                [Parameter(Mandatory=$true, HelpMessage='VM name on VCD')] 

                                [ValidateNotNull()]

                                [ValidateNotNullOrEmpty()]

                                [string]$vmname,

                         

                         

                                [Parameter(Mandatory=$true, HelpMessage='Org name on VCD')] 

                                [ValidateNotNull()]

                                [ValidateNotNullOrEmpty()]

                                [string]$org,

                         

                         

                                [Parameter(Mandatory=$true, HelpMessage='Vapp name on VCD')] 

                                [ValidateNotNull()]

                                [ValidateNotNullOrEmpty()]

                                [string]$vapp,

                         

                         

                                [Parameter(Mandatory=$true, HelpMessage='VI Server Name')] 

                                [ValidateNotNull()]

                                [ValidateNotNullOrEmpty()]

                                [string]$Server,

                         

                         

                                [Parameter(Mandatory=$true, HelpMessage='VI connect - Session ID')] 

                                [ValidateNotNull()]

                                [ValidateNotNullOrEmpty()]

                                [string]$SessionId

                               )

                        Begin{

                        }

                         

                         

                            Process {

                                Try{

                                #Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false | Out-Null

                                

                                    Connect-VIServer -Server $Server -Session $SessionId

                                    $vm = Get-CIVM -Org $org -VApp $vapp -Name $vmname

                                    $vmhref = $vm.ExtensionData.Href

                                    $vmstate = Invoke-vCloud -URI $vmhref -Method GET

                                    $vmnvramname = $vmstate.Vm.VirtualHardwareSection.ExtraConfig | Where-Object {$_.key -eq "nvram"}

                                    $vspherevmname = $vmnvramname.value.Split(".",6)[0]

                         

                         

                                    return $vspherevmname

                                }

                                Catch{Write-host "Catch Block"}

                                    }

                            end     {

                                Write-Verbose 'VM Name Read'

                                    }

                        }

                         

                         

                        • 9. Re: Execute multiple Powershell tasks in parallel
                          LucD Guru
                          Community WarriorsUser ModeratorsvExpert

                          You don't do a Connect-CIServer cmdlet in the background job, then it is normal I guess that you get that error on the Get-CIVm cmdlet.

                          The Connect-CIServer also supports a SessionId parameter, by which you can reuse an existing connection in the calling script.

                          • 10. Re: Execute multiple Powershell tasks in parallel
                            NikhilPathak Novice

                            Thanks a lot LucD.. I am able to get the output if i run it for one job, But have a problem while executing multiple jobs.. The output returned for is not correct. It always returns the output of the last VM queried.. Could you please find the mistake in my code...

                             

                            Below is the Job

                             

                            $code = {

                                param(

                                    [string]$VIServer,

                                    [string]$VISessionId,

                                    [string]$VMName,

                                    [string]$org,

                                    [string]$vapp,

                                    [string]$CIServer,

                                    [string]$CISessionId

                                )

                                 $rem1 = Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false | Out-Null

                                $rem2 = Connect-VIServer -Server $VIServer -Session $VISessionId

                                $rem3 = Connect-CIServer -Server $CIServer -SessionId $CISessionId

                                . C:\Migration_Powershell_Scripts\New_functions\Get-vspherevmname.ps1

                                get-vspherevmname -vmname $VMName -org $org -vapp $vapp -CIServer $CIServer -CISessionId $CISessionId -VIserver $VIServer -VIsessionid $VISessionId

                            }

                            $VM = @("R511-SRVA", "R511-SRVB", "VEPMASTER")

                            $results = @()

                            $org = "ID14434-1-1231231"

                            $vapp = "12345"

                            $sJOb = @{

                                ScriptBlock = $code

                                ArgumentList = $global:DefaultVIServer.Name, $global:DefaultVIServer.SessionId, $vmName, $org, $vapp, $global:DefaultCIServers.Name, $global:DefaultCIServers.SessionId

                            }

                            $VM = @("R511-SRVA", "R511-SRVB", "VEPMASTER")

                            $results = @()

                             

                            foreach($vmName in $VM){

                            $j = Start-Job @sJob

                            Wait-Job $j

                             

                            $row= new-object -TypeName PSObject -Property @{

                                'VM_name' = $VMname

                                'Job_ID' = $j.Id

                                'vspherename' = Receive-Job -Id $j.Id -Keep

                                }

                            $results += $row

                            }

                             

                            Output is

                             

                             

                            Id     Name            PSJobTypeName   State         HasMoreData     Location             Command                 

                            --     ----            -------------   -----         -----------     --------             -------                 

                            107    Job107          BackgroundJob   Completed     True            localhost            ...                     

                            VEPMASTER-ryQF

                            109    Job109          BackgroundJob   Completed     True            localhost            ...                     

                            VEPMASTER-ryQF

                            111    Job111          BackgroundJob   Completed     True            localhost            ...                     

                            VEPMASTER-ryQF

                             

                            Here i am expecting the return value to be different..

                             

                            My Custom Function is

                             

                             

                            Function Get-vspherevmname{

                                [CmdletBinding()]

                                [OutputType([psobject])] # Select the output type of this function as "string/PSObject/Int"

                             

                             

                            # Enter the list of Input Parameters for the function

                            #The Parameter should have properties as "Mandatory", HelpMessage

                            Param (

                                    [Parameter(Mandatory=$true, HelpMessage='VM name on VCD')] 

                                    [ValidateNotNull()]

                                    [ValidateNotNullOrEmpty()]

                                    [string]$vmname,

                             

                             

                                    [Parameter(Mandatory=$true, HelpMessage='VM name on VCD')] 

                                    [ValidateNotNull()]

                                    [ValidateNotNullOrEmpty()]

                                    [string]$org,

                             

                             

                                    [Parameter(Mandatory=$true, HelpMessage='VM name on VCD')] 

                                    [ValidateNotNull()]

                                    [ValidateNotNullOrEmpty()]

                                    [string]$vapp,

                             

                             

                                    [Parameter(Mandatory=$false, HelpMessage='Server Name')] 

                                    [ValidateNotNull()]

                                    [ValidateNotNullOrEmpty()]

                                    [string]$CIServer,

                             

                             

                                    [Parameter(Mandatory=$false, HelpMessage='VM name on VCD')] 

                                    [ValidateNotNull()]

                                    [ValidateNotNullOrEmpty()]

                                    [string]$CISessionId,

                             

                             

                                    [Parameter(Mandatory=$false, HelpMessage='VM name on VCD')] 

                                    [ValidateNotNull()]

                                    [ValidateNotNullOrEmpty()]

                                    [string]$VIserver,

                             

                             

                                    [Parameter(Mandatory=$false, HelpMessage='VM name on VCD')] 

                                    [ValidateNotNull()]

                                    [ValidateNotNullOrEmpty()]

                                    [string]$VIsessionid

                                   )

                            Begin{

                            }

                             

                             

                                Process {

                                    Try{

                                    #Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false | Out-Null

                                        $ignore = Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false | Out-Null

                                        $ciconnect = Connect-CIServer -Server $ciserver -Session $cisessionid

                                        $viconnect =  Connect-VIServer -Server $viServer -Session $viSessionId

                                        $vm = Get-CIVM -Org $org -VApp $vapp -Name $vmname

                                        $vmhref = $vm.ExtensionData.Href

                                        #Write-Host $vmhref

                                        ##[string]$ApiVersion = '31.0'

                                        ##$Headers = @{ "x-vcloud-authorization" = $vcloudtoken; "Accept" = 'application/*+xml;version=' + $ApiVersion }

                                        ##$body = $null

                                        ##Invoke-RestMethod -Method GET -Uri $vmhref -Headers $Headers -Body $body

                                        . 'C:\Migration_Powershell_Scripts\New_functions\Invoke-vCloud.ps1'

                                        $vmstate = Invoke-vCloud -URI $vmhref -Method GET -vCloudToken $CISessionId

                                        $vmnvramname = $vmstate.Vm.VirtualHardwareSection.ExtraConfig | Where-Object {$_.key -eq "nvram"}

                                        $vspherevmname = $vmnvramname.value.Split(".",6)[0]

                                        #$Headers = @{ "x-vcloud-authorization" = $mySessionID; "Accept" = 'application/*+xml;version=' + $ApiVersion }

                                        return $vspherevmname

                                    }

                                    Catch{Write-host "Catch Block"}

                                        }

                                end     {

                                    Write-Verbose 'VM Name Read'

                                        }

                            }

                            • 11. Re: Execute multiple Powershell tasks in parallel
                              LucD Guru
                              Community WarriorsUser ModeratorsvExpert

                              You should be filling the hash table ($sJob) inside the ForEach loop.

                              Now you are starting each job with the same parameters,