VMware Cloud Community
NikhilPathak
Enthusiast
Enthusiast
Jump to solution

Execute multiple Powershell tasks in parallel

Currently scenario:

Read an xml file and Deploy VM's, Customize them and then invoke Installation.

The above process takes almost 25 minutes for each VM. So in a foreach loop if i have 4 VM's, it takes around 1.5 ~ 2 hours

Is there any way to perform the task in parallel and reduce the execution time.

1 Solution

Accepted Solutions
LucD
Leadership
Leadership
Jump to solution

Why are you using the RunAsync switch on the Invoke-VMScript cmdlet?
Since you are running this job in parallel for multiple VMs, there is no real need to use the RunAsync switch.

That would also allow you to get rid of the loops after each Invoke-VMScript.

---------------------------------------------------------------------------------------------------------

Was it helpful? Let us know by completing this short survey here.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

View solution in original post

Reply
0 Kudos
17 Replies
LucD
Leadership
Leadership
Jump to solution

Yes, you could use background jobs (with Start-Job).


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
NikhilPathak
Enthusiast
Enthusiast
Jump to solution

I tried using workflow as below,

$VM = "R511-SRVA-WyU3", "R511-SRVB-0DV1"

workflow pbatch{

param ([string[]]$VM)

foreach -parallel($a in $VM){

$installscript = "hostname"

$invokeinstalltask = Invoke-VMScript -ScriptText $installscript -VM $a -GuestUser "abc" -GuestPassword "Pass" -RunAsync

}

}

pbatch -VM $VM

but it gives me error as " Invoke-VMScript You are not currently connected to any servers. Please connect first using a Connect cmdlet. ".

The error is strange because i dont get this error if i run it in a simple foreach loop.

I Tried using Jobs as well with below example but the JOBs didn't stop even after 2 hrs. Ideally, the Get VM should not take 2 hrs when the VM name is provided.

$abc = "R511-SRVA-WyU3", "R511-SRVB-0DV1"

$block = {

param ([string[]]$a)

Get-VM -Name $a

}

$MaxThreads = 2

#Remove all jobs

Get-Job | Remove-Job

foreach($a in $abc){

    While ($(Get-Job -state running).count -ge $MaxThreads){

        Start-Sleep -Milliseconds 3

    }

    Start-Job -Scriptblock $Block -ArgumentList $a

}

#Wait for all jobs to finish.

While ($(Get-Job -State Running).count -gt 0){

    start-sleep 1

}

#Get information from each job.

foreach($job in Get-Job){

    $info= Receive-Job -Id ($job.Id)

}

#Remove all jobs created.

Get-Job | Remove-Job

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

To use background jobs, you have to reconnect inside the job.

See my Running a background job.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
NikhilPathak
Enthusiast
Enthusiast
Jump to solution

Hi LucD, i tried the way you did parallel jobs it works fine with powercli functions, but i am not able to use my custom function.

it fails with error as

The term 'Get-vspherevmname' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify

that the path is correct and try again.

    + CategoryInfo          : ObjectNotFound: (Get-vspherevmname:String) [], CommandNotFoundException

    + FullyQualifiedErrorId : CommandNotFoundException

    + PSComputerName        : localhost

Can you please help?

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

Where and how do you define your function?
The job environment is a new PS environment, you might have to dot-source the file that contains your function inside the job.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
NikhilPathak
Enthusiast
Enthusiast
Jump to solution

currently i have the Function created as a separate ps1.

I run that ps1 file before executing my main script.

Sorry i am not an expert on Powershell, but your guidance will help..

Do you have any example where you have executed custom Function in parallel..

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

You have to make your function known to the PS engine.
The PS engine in a background job can be considered as new.

So all functions that the calling script knows, are not known in the background job.

That is why you need to dot-source the .ps1 file inside the background job.
In the below example I assume the file is called myfunction.ps1 which contains the function Do-MyFunction.
Note the space between the 2 dots when dot-sourcing a .ps1 file.

$block = {

   param ([string[]]$a)


   . .\myfunction.ps1

   Do-MyFunction

}


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
NikhilPathak
Enthusiast
Enthusiast
Jump to solution

Hi LucD, I am still getting the error as, "Get-CIVM You are not currently connected to any servers. Please connect first using a Connect cmdlet."

Below is the code where i am trying the Job..  The get-civm is used in my function. Can you please check...

#-----------------------------------------------------------------------------------------------------------------------------------------------

$code = {

    param(

        [string]$Server,

        [string]$SessionId,

        [string]$VMName,

        [string]$org,

        [string]$vapp

    )

    Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false | Out-Null

    Connect-VIServer -Server $Server -Session $SessionId

    . C:\Migration_Powershell_Scripts\New_functions\Get-vspherevmname.ps1

    get-vspherevmname -vmname $VMName -org $org -vapp $vapp -Server $Server -SessionId $SessionId

}

$vmName = 'R511-SRVA'

$org = "ID14434-1-1231231"

$vapp = "12345"

$sJOb = @{

    ScriptBlock = $code

    ArgumentList = $global:DefaultVIServer.Name, $global:DefaultVIServer.SessionId, $vmName, $org, $vapp

}

$ja = Start-Job @sJob

#-----------------------------------------------------------------------------------------------------------------------------------------------------------

My Custom Function

Function Get-vspherevmname{

    [CmdletBinding()]

    [OutputType([psobject])] # Select the output type of this function as "string/PSObject/Int"

# Enter the list of Input Parameters for the function

#The Parameter should have properties as "Mandatory", HelpMessage

Param (

        [Parameter(Mandatory=$true, HelpMessage='VM name on VCD')] 

        [ValidateNotNull()]

        [ValidateNotNullOrEmpty()]

        [string]$vmname,

        [Parameter(Mandatory=$true, HelpMessage='Org name on VCD')] 

        [ValidateNotNull()]

        [ValidateNotNullOrEmpty()]

        [string]$org,

        [Parameter(Mandatory=$true, HelpMessage='Vapp name on VCD')] 

        [ValidateNotNull()]

        [ValidateNotNullOrEmpty()]

        [string]$vapp,

        [Parameter(Mandatory=$true, HelpMessage='VI Server Name')] 

        [ValidateNotNull()]

        [ValidateNotNullOrEmpty()]

        [string]$Server,

        [Parameter(Mandatory=$true, HelpMessage='VI connect - Session ID')] 

        [ValidateNotNull()]

        [ValidateNotNullOrEmpty()]

        [string]$SessionId

       )

Begin{

}

    Process {

        Try{

        #Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false | Out-Null

        

            Connect-VIServer -Server $Server -Session $SessionId

            $vm = Get-CIVM -Org $org -VApp $vapp -Name $vmname

            $vmhref = $vm.ExtensionData.Href

            $vmstate = Invoke-vCloud -URI $vmhref -Method GET

            $vmnvramname = $vmstate.Vm.VirtualHardwareSection.ExtraConfig | Where-Object {$_.key -eq "nvram"}

            $vspherevmname = $vmnvramname.value.Split(".",6)[0]

            return $vspherevmname

        }

        Catch{Write-host "Catch Block"}

            }

    end     {

        Write-Verbose 'VM Name Read'

            }

}

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

You don't do a Connect-CIServer cmdlet in the background job, then it is normal I guess that you get that error on the Get-CIVm cmdlet.

The Connect-CIServer also supports a SessionId parameter, by which you can reuse an existing connection in the calling script.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
NikhilPathak
Enthusiast
Enthusiast
Jump to solution

Thanks a lot LucD.. I am able to get the output if i run it for one job, But have a problem while executing multiple jobs.. The output returned for is not correct. It always returns the output of the last VM queried.. Could you please find the mistake in my code...

Below is the Job

$code = {

    param(

        [string]$VIServer,

        [string]$VISessionId,

        [string]$VMName,

        [string]$org,

        [string]$vapp,

        [string]$CIServer,

        [string]$CISessionId

    )

     $rem1 = Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false | Out-Null

    $rem2 = Connect-VIServer -Server $VIServer -Session $VISessionId

    $rem3 = Connect-CIServer -Server $CIServer -SessionId $CISessionId

    . C:\Migration_Powershell_Scripts\New_functions\Get-vspherevmname.ps1

    get-vspherevmname -vmname $VMName -org $org -vapp $vapp -CIServer $CIServer -CISessionId $CISessionId -VIserver $VIServer -VIsessionid $VISessionId

}

$VM = @("R511-SRVA", "R511-SRVB", "VEPMASTER")

$results = @()

$org = "ID14434-1-1231231"

$vapp = "12345"

$sJOb = @{

    ScriptBlock = $code

    ArgumentList = $global:DefaultVIServer.Name, $global:DefaultVIServer.SessionId, $vmName, $org, $vapp, $global:DefaultCIServers.Name, $global:DefaultCIServers.SessionId

}

$VM = @("R511-SRVA", "R511-SRVB", "VEPMASTER")

$results = @()

foreach($vmName in $VM){

$j = Start-Job @sJob

Wait-Job $j

$row= new-object -TypeName PSObject -Property @{

    'VM_name' = $VMname

    'Job_ID' = $j.Id

    'vspherename' = Receive-Job -Id $j.Id -Keep

    }

$results += $row

}

Output is

Id     Name            PSJobTypeName   State         HasMoreData     Location             Command                 

--     ----            -------------   -----         -----------     --------             -------                 

107    Job107          BackgroundJob   Completed     True            localhost            ...                     

VEPMASTER-ryQF

109    Job109          BackgroundJob   Completed     True            localhost            ...                     

VEPMASTER-ryQF

111    Job111          BackgroundJob   Completed     True            localhost            ...                     

VEPMASTER-ryQF

Here i am expecting the return value to be different..

My Custom Function is

Function Get-vspherevmname{

    [CmdletBinding()]

    [OutputType([psobject])] # Select the output type of this function as "string/PSObject/Int"

# Enter the list of Input Parameters for the function

#The Parameter should have properties as "Mandatory", HelpMessage

Param (

        [Parameter(Mandatory=$true, HelpMessage='VM name on VCD')] 

        [ValidateNotNull()]

        [ValidateNotNullOrEmpty()]

        [string]$vmname,

        [Parameter(Mandatory=$true, HelpMessage='VM name on VCD')] 

        [ValidateNotNull()]

        [ValidateNotNullOrEmpty()]

        [string]$org,

        [Parameter(Mandatory=$true, HelpMessage='VM name on VCD')] 

        [ValidateNotNull()]

        [ValidateNotNullOrEmpty()]

        [string]$vapp,

        [Parameter(Mandatory=$false, HelpMessage='Server Name')] 

        [ValidateNotNull()]

        [ValidateNotNullOrEmpty()]

        [string]$CIServer,

        [Parameter(Mandatory=$false, HelpMessage='VM name on VCD')] 

        [ValidateNotNull()]

        [ValidateNotNullOrEmpty()]

        [string]$CISessionId,

        [Parameter(Mandatory=$false, HelpMessage='VM name on VCD')] 

        [ValidateNotNull()]

        [ValidateNotNullOrEmpty()]

        [string]$VIserver,

        [Parameter(Mandatory=$false, HelpMessage='VM name on VCD')] 

        [ValidateNotNull()]

        [ValidateNotNullOrEmpty()]

        [string]$VIsessionid

       )

Begin{

}

    Process {

        Try{

        #Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false | Out-Null

            $ignore = Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false | Out-Null

            $ciconnect = Connect-CIServer -Server $ciserver -Session $cisessionid

            $viconnect =  Connect-VIServer -Server $viServer -Session $viSessionId

            $vm = Get-CIVM -Org $org -VApp $vapp -Name $vmname

            $vmhref = $vm.ExtensionData.Href

            #Write-Host $vmhref

            ##[string]$ApiVersion = '31.0'

            ##$Headers = @{ "x-vcloud-authorization" = $vcloudtoken; "Accept" = 'application/*+xml;version=' + $ApiVersion }

            ##$body = $null

            ##Invoke-RestMethod -Method GET -Uri $vmhref -Headers $Headers -Body $body

            . 'C:\Migration_Powershell_Scripts\New_functions\Invoke-vCloud.ps1'

            $vmstate = Invoke-vCloud -URI $vmhref -Method GET -vCloudToken $CISessionId

            $vmnvramname = $vmstate.Vm.VirtualHardwareSection.ExtraConfig | Where-Object {$_.key -eq "nvram"}

            $vspherevmname = $vmnvramname.value.Split(".",6)[0]

            #$Headers = @{ "x-vcloud-authorization" = $mySessionID; "Accept" = 'application/*+xml;version=' + $ApiVersion }

            return $vspherevmname

        }

        Catch{Write-host "Catch Block"}

            }

    end     {

        Write-Verbose 'VM Name Read'

            }

}

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

You should be filling the hash table ($sJob) inside the ForEach loop.

Now you are starting each job with the same parameters,


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

NikhilPathak
Enthusiast
Enthusiast
Jump to solution

Thanks a lot LucD, you have been my saviour...

i was able to make many of the tasks to run in parallel. But, i am facing issue in executing "Invoke-vmscript".

I have a custom function to execute multiple tasks using invoke-vmscript. the problem here is,

when i execute the custom script as a Job, only the 1st call of "invoke-vmscript" gets executed. Rest are not..

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

This depends on what you are doing in the code submitted through Invoke-VMScript.

I would need to see your code to understand what is happening.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
NikhilPathak
Enthusiast
Enthusiast
Jump to solution

below are the scripts which are invoked..

            #Create a Scheduled task

            $Scheduletask = '

            $ST = New-ScheduledTaskAction -Execute "\\VEPMASTER\ESIS\setup.exe"

            $register = Register-ScheduledTask -Action $ST -TaskName "Experion-Installation" -Description "Task for invoking the experion Installation on this node"

            '

            $createscheduledtask = Invoke-VMScript -ScriptText $Scheduletask -VM $VSpherevm -GuestUser $username -GuestPassword $password -RunAsync

            Start-Sleep -Seconds 30

            do{

            Start-Sleep -Seconds 2

            }until($createscheduledtask.State -eq "Success")

            #Set-up Autologon to the VM

            $autologonscript = '

            $RegPath = "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon"

            $s1 = Set-ItemProperty $RegPath "AutoAdminLogon" -Value "1" -type String 

            $s2 = Set-ItemProperty $RegPath "DefaultUsername" -Value "administrator" -type String 

            $s3 = Set-ItemProperty $RegPath "DefaultPassword" -Value "Password1" -type String

            $s4 = Set-ItemProperty $RegPath "AutoLogonCount" -Value "2" -type DWord

            '

            $autologontask = Invoke-VMScript -ScriptText $autologonscript -VM $VSpherevm -GuestUser $username -GuestPassword $password -RunAsync

            do{

            Start-Sleep -Seconds 2

            }until($autologontask.State -eq "Success")

            #Invoke the Scheduled task on VM

            $installscript = '

            Start-ScheduledTask -TaskName "Experion-Installation"

            '

            $invokeinstalltask = Invoke-VMScript -ScriptText $installscript -VM $VSpherevm -GuestUser $username -GuestPassword $password -RunAsync

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

Why are you using the RunAsync switch on the Invoke-VMScript cmdlet?
Since you are running this job in parallel for multiple VMs, there is no real need to use the RunAsync switch.

That would also allow you to get rid of the loops after each Invoke-VMScript.

---------------------------------------------------------------------------------------------------------

Was it helpful? Let us know by completing this short survey here.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
NikhilPathak
Enthusiast
Enthusiast
Jump to solution

Thanks a lot LuCD.

Reply
0 Kudos
kompanets
Contributor
Contributor
Jump to solution

Hello NikhilPathak,

​I'm looking for means to automate VMs deployment process and as I can see you've already done it. Would you be so kind to share with your PowerShell scripts, functions, modules. Maybe you have your own repository on GitHub or website or something else...

​Thank you in advance.

P.S. I'm writing you here because private messaging isn't working at the moment.

Reply
0 Kudos