VMware Cloud Community
CNI0
Enthusiast
Enthusiast

Multithreading PowerCLI with RunspacePool

I am trying to create a script that collect a lot of info from VM's and also set info if needed. That is not my problem. The problem is that there is alot of VM's and it takes about 5-8 sec pr VM. I have create a small sample script to run a process on several VM's at the same time, but it fails in mysterious ways :smileycry: If I only allow 1 thread everything is behaving as expected. More than one thread, it fails with messages that states the PowerCLI cmdlet are not known "The term 'Set-PowerCLIConfiguration' is not recognized as the name of a cmdlet"

I am using PowerCLI 6

Any ideas on whats wrong?

Below is my sample code (some of it is from Google Smiley Wink)

# The per VM codeblock sample

$scriptBlock = {

  param($VMName, $vcenter, $session)

  Write-Host "VMThread - Start for: $($VMName)"

  Import-Module VMware.VimAutomation.Core -Global

  Write-Host "VMThread - Modules loaded: $(Get-Module)"

  $Temp = Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Scope Session -Confirm:$false

  $myVI = Connect-VIServer -Server $vcenter -Session $session

  $Myvm = Get-VM -Name $VMName

  if($Myvm)

  {

  Write-Host "VMThread - Getting VM info: $($Myvm.Name)"

  $newInfo = New-Object PSObject

  $newInfo | Add-Member -Type noteproperty -Name VMName -Value $Myvm.Name

  $newInfo | Add-Member -Type noteproperty -Name MemMB -Value $Myvm.MemoryMB

  $newInfo

  Start-Sleep -Seconds 3

  }

  else

  {

  Write-Host "VMThread - No info for VM: $($VMName) - Modules: $(Get-Module)" -ForegroundColor Red

  }

  Write-Host "VMThread - Done: $($VMName) - Error Count: $($Error.Count) - $Error"

}

#Start the threads and wait for result

function RunTest{

  param

  (

  $VMs,

  $Throttle = 1 #threads

  )

  process

  {

  $sessionstate = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()

  $sessionstate.ImportPSModule("VMware.VimAutomation.Core")

  $RunspacePool = [runspacefactory]::CreateRunspacePool(1, $Throttle, $sessionstate, $Host)

  $RunspacePool.Open()

  $Jobs = @()

  Write-Host "Starting threads.."

  foreach ($oneVM in $VMs)

  {

    $Job = [powershell]::Create().AddScript($ScriptBlock).AddArgument($oneVM.Name).AddArgument($global:DefaultVIServer.Name).AddArgument($global:DefaultVIServer.SessionSecret)

    $Job.RunspacePool = $RunspacePool

    $Jobs += New-Object PSObject -Property @{RunVM = $_.Name; Pipe = $Job; Result = $Job.BeginInvoke()}

  }

 

  Write-Host "Start waiting.."

  Do

  {

    Start-Sleep -Seconds 1

  }

  While ( $Jobs.Result.IsCompleted -contains $false)

  Write-Host "All jobs completed!"

 

  $Results = @()

  ForEach ($Job in $Jobs)

  {

  $Results += $Job.Pipe.EndInvoke($Job.Result)

  }

  $RunspacePool.Close()

  Write-Host "`n`nReturning $($Results.Count) objects..." -ForegroundColor Cyan

  $Results

  }

}

# Logon to vcenter

$user = Get-Credential domain\user

$vi = connect-viserver myvCenter -Credential $user

$AllVM = Get-VM                     # Return all my VM's, 2 in my test vcenter

RunTest -VMs $AllVM -Throttle 1     # Runs OK, returns one object for each VM

RunTest -VMs $AllVM -Throttle 2     # Fails

10 Replies
LucD
Leadership
Leadership

Can you display $PSModulePath in each thread ?

Is the content correct ?

Or try doing a 'Get-Module -ListAvailable' in each thread.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
CNI0
Enthusiast
Enthusiast

Hi

Sorry for the slow response, but i did not get a notification mail :smileyplain:

Outputs from the 2 commands are similar on all threads. It looks like this:

VMThread - PSModulePath: C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules\;C:\Program Files\Microsoft Monitoring Agent

\Agent\PowerShell\;C:\Program Files (x86)\Microsoft SQL Server\110\Tools\PowerShell\Modules\

VMThread - Get-Module: VMware.VimAutomation.Cis.Core VMware.VimAutomation.Cloud VMware.VimAutomation.Core VMware.VimAuto

mation.HA VMware.VimAutomation.PCloud VMware.VimAutomation.SDK VMware.VimAutomation.Storage VMware.VimAutomation.Vds App

Locker Appx BestPractices BitsTransfer BranchCache CimCmdlets DirectAccessClientComponents Dism DnsClient International

iSCSI IscsiTarget ISE Kds Microsoft.PowerShell.Diagnostics Microsoft.PowerShell.Host Microsoft.PowerShell.Management Mic

rosoft.PowerShell.Security Microsoft.PowerShell.Utility Microsoft.WSMan.Management MMAgent MsDtc NetAdapter NetConnectio

n NetEventPacketCapture NetLbfo NetNat NetQos NetSecurity NetSwitchTeam NetTCPIP NetworkConnectivityStatus NetworkTransi

tion NFS PcsvDevice PKI PrintManagement PSDesiredStateConfiguration PSDiagnostics PSLog PSScheduledJob PSWorkflow PSWork

flowUtility RemoteDesktop ScheduledTasks SecureBoot ServerCore ServerManager ServerManagerTasks SmbShare SmbWitness Soft

wareInventoryLogging StartScreen Storage TLS TroubleshootingPack TrustedPlatformModule UserAccessLogging VpnClient Wdac

Whea WindowsDeveloperLicense WindowsErrorReporting WindowsSearch Microsoft.MonitoringAgent.PowerShell SQLASCMDLETS SQLPS

0 Kudos
aka_MAC
Contributor
Contributor

Seems like module exports cmdlets in one runspace only:

ModuleType Version    Name                                ExportedCommands                                                                                                                       

---------- -------    ----                                ----------------                                                                                                                       

Manifest   3.1.0.0    Microsoft.PowerShell.Management     {Add-Computer, Add-Content, Checkpoint-Computer, Clear-Content...}                                                                     

Manifest   3.1.0.0    Microsoft.PowerShell.Utility        {Add-Member, Add-Type, Clear-Variable, Compare-Object...}                                                                              

Manifest   6.0.0.0    VMware.VimAutomation.Core                                                                                                                                                  

Manifest   6.0.0.0    VMware.VimAutomation.Sdk       

                                                                                                                                          

Manifest   3.1.0.0    Microsoft.PowerShell.Management     {Add-Computer, Add-Content, Checkpoint-Computer, Clear-Content...}                                                                     

Manifest   3.1.0.0    Microsoft.PowerShell.Utility        {Add-Member, Add-Type, Clear-Variable, Compare-Object...}                                                                              

Manifest   6.0.0.0    VMware.VimAutomation.Core           {Add-PassthroughDevice, Add-VirtualSwitchPhysicalNetworkAdapter, Add-VMHost, Add-VMHostNtpServer...}                                   

Manifest   6.0.0.0    VMware.VimAutomation.Sdk                                                                                                                    

0 Kudos
aka_MAC
Contributor
Contributor

I guess this is a bug and should be passed over to VMware.

Previous PowerCLI version worked well with RunSpaces

0 Kudos
LucD
Leadership
Leadership

I don't think this is really a bug, but I suspect it is due to the fact the current PowerCLI release (6R1) uses a mix of modules and pssnapins.

In fact this causes some other minor issues as well.

Can't wait to see a full module release.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
monosoul
Contributor
Contributor

Hi CNI0 !

Basically the fastest way to do what you wanted is this one:

$allVms = Get-View -ViewType VirtualMachine -Property Name,"Config.Hardware","Config.Template" -Filter @{"Config.Template"="False"}

$allVms | Select @{N="VMName";E={$_.Name}},@{N="MemMB";E={$_.Config.Hardware.MemoryMB}} | Out-GridView

You'll be impressed with how fast it is, even without runspaces!

Or, if you really want to use runspaces, then you could use my PS module, which contains functions to work with RunSpaces like with Jobs. You could grab it here: monosoul/basic_functions.psm1 · GitHub

(readme may be little outdated, byt every function has built-in documentation)

Here is an example code for what you wanted to do:

$user = Get-Credential domain\user 

$vi = connect-viserver myvCenter -Credential $user

$snapins = @()

$modules = @()

$snapins += "VMware.VimAutomation.Core"

if ((Get-PowerCLIVersion).Major -le 5) {

  $snapins += "VMware.VimAutomation.Vds"

  $snapins += "VMware.VimAutomation.Storage"

}

else {

  $modules += "VMware.VimAutomation.Vds"

  $modules += "VMware.VimAutomation.Storage"

}

$AllVM = Get-VM

#basically it's still better to use Get-View as it'd be a lot faster, but i'll stick to your example

#$AllVM = Get-View -ViewType VirtualMachine -Property Name,"Config.Hardware","Config.Template" -Filter @{"Config.Template"="False"}

#making synchronized queue out of array

[System.Collections.Queue]$AllVM = [System.Collections.Queue]::Synchronized( ([System.Collections.Queue]$AllVM) )

#creatin synchronized array list for results

$newInfo = [System.Collections.ArrayList]::Synchronized( (New-Object System.Collections.ArrayList) )

Start-RSJobs -ScriptBlock {

  param(

    $AllVM,

    $newInfo

  )

  while ($AllVM.Count -gt 0) {

    $Myvm = $AllVM.Dequeue()

    $newInfo.Add((New-Object PSObject -Property @{

      VMName = $Myvm.Name

      MemMB = $Myvm.MemoryMB

    }))

  }

} -snapinsToImport $snapins `

  -modulesToImport $modules `

  -variablesToImport @("DefaultVIServer", "DefaultVIServers") `

  -ArgumentList $AllVM, $newInfo `

| Wait-RSJob | Remove-RSJob

$newInfo

As of modules importing only in first runspace - yeah, I've faced that issue too even with PCLI6R3, and I think it's some kind of bug. But for now I solved it by using snap-in version of "VMware.VimAutomation.Core" (other modules are importing fine in every runspace).

eeldivady
Contributor
Contributor

I found a solution that allows me to still run multithread using runspaces. Since runspaces aren't threadsafe, you have to run each runspace outofprocess. You don't need to use snapins with this solution.  This thread is unanswered for 2 years.  I guess it's not priority for Vmware but I figured I'll share it as I haven't found a good solution by googling for past 2 days.  Here's what I had to do in my c# code. You just need to make adjustments in your powershell code to do the same by calling the RunspaceFactory.CreateOutOfProcessRunspace() Can you send me your code that you converted using the SDK? That would be helpful in my project too.  Thanks

        public async Task TestPowercli(string name, string vcenterHost) {

            if (string.IsNullOrWhiteSpace(name)) { return; }

            if (string.IsNullOrWhiteSpace(vcenterHost)) { return; }

            instanceName = name.Trim();

            int timeoutMins = 5;

            string script = @"c:\temp\testpowercli\testpowercli.ps1 " + instanceName + " " + vcenterHost + " -verbose";

            PowerShellProcessInstance instance = new PowerShellProcessInstance(new Version(5, 0), null, null, false);

            using (Runspace runspace = RunspaceFactory.CreateOutOfProcessRunspace(new TypeTable(new string[0]), instance)) {

                PowerShell ps = PowerShell.Create();

                runspace.Open();

                ps.Runspace = runspace;

                ps.AddScript(script);

                outputCollection.DataAdded += outputCollection_DataAddedPowercli;

                // the streams (Error, Debug, Progress, etc) are available on the PowerShell instance.

                // we can review them during or after execution.

                // we can also be notified when a new item is written to the stream (like this):

                ps.Streams.Error.DataAdded += Error_DataAddedPowercli;

                ps.Streams.Verbose.DataAdded += Verbose_DataAddedPowercli;

                IAsyncResult result = ps.BeginInvoke<PSObject, PSObject>(null, outputCollection);

                DateTime start = DateTime.Now;

                while (result.IsCompleted == false) {

                    if ((DateTime.Now - start).TotalMinutes > timeoutMins) {

                        Clients.All.getPowercliMessage(instanceName, "ERROR: Time out exceeded after " + timeoutMins + " minutes");

                        break;

                    }

                    await Task.Delay(1000);

                }

            }

            Clients.All.getPowercliMessage(instanceName, "Done");

        }

0 Kudos
grasshopper
Virtuoso
Virtuoso

Hi monosoul,

Thanks so much for sharing your code.  I have changed it slightly for my purposes and turned it into a module on github.  The objective was to provide a simple framework for people to customize and add the things they are interested in returning from ESX (using Runspaces of course).

It seems to be working fine with the latest PowerCLI (tested on 6.5.2).   I am open to ways to make it better.

GitHub - vmkdaily/EsxRunspace: Connects to one or more VMware ESX hosts using PowerShell Runspace jo...

-Mike

0 Kudos
DS0VMWARE
Contributor
Contributor

This post saved my soul. If you're like me, you maybe needed some deeper separation between the powershell runspaces and wanted to do multithreaded copies of files to remote datastores. The problem with the current powercli implementation is that it maps psdrives in a local scope, but makes the defaultviservers variable in a global scope. My recommendation to VMware is that the next version of powercli includes an option on the "Connect-VIServer" cmdlet to specify a scope of that connection, or at least patch the psdrives for "vmstores" and "vis" to a global scope that can be shared among runspaces.

I took the liberty of writing it out in powershell. This basically is an alternative to a runspace pool and should work in any powershell version past 3.0:

$Active_Jobs = @()

$index = 0

while ($true) {


   if (($Active_Jobs.count -lt $num_threads) -and $index -lt $some_list.Length) {

   $job = ($some_list[$index])

   Write-Host "Creating job for $job"


   $psinstance = [System.Management.Automation.PowerShell]::create()

   Add-Member -InputObject $psinstance -NotePropertyName "request_stop" -NotePropertyValue "N"

   $psinstance.AddScript($script_block).AddArgument($some_list[$index]) | Out-Null

   $index++

   $runspace = [runspacefactory]::CreateOutOfProcessRunspace([System.Management.Automation.Runspaces.TypeTable]::GetDefaultTypeFiles())

   if([Console]::InputEncoding -is [Text.UTF8Encoding] -and [Console]::InputEncoding.GetPreamble().Length -ne 0) {

   [Console]::InputEncoding = New-Object Text.UTF8Encoding $false

  }


   $runspace.Open()

   $psinstance.Runspace = $runspace

   $InputStream = New-Object -Typename System.Management.Automation.PSDataCollection[PSObject]

   $OutputStream = New-Object -Typename System.Management.Automation.PSDataCollection[PSObject]

   $Active_Jobs += New-Object PSObject -Property @{

  Thread = $psinstance

  Handle = $psinstance.BeginInvoke($InputStream,$OutputStream)

  runspace = $runspace

  output = $OutputStream

  }

   Start-Sleep -Seconds 1

  }


   Foreach ($Job in $($Active_Jobs | Where-Object {$_.Thread.request_stop -eq "Y"})) {

   $Job.Thread.Stop()

   Write-Host $Job.output

   $Job.Thread.Dispose()

   $Job.Runspace.Close()

   $Job.Runspace.Dispose()

   $Active_Jobs = $Active_Jobs -ne $Job

  }


   Foreach ($Job in $($Active_Jobs | Where-Object {$_.Handle.IsCompleted -eq $true})) {

   Write-Host $Job.output

   $Job.Thread.EndInvoke($Job.Handle) 2>$null

   $Job.Thread.Dispose()

   $Job.Runspace.Close()

   $Job.Runspace.Dispose()

   $Active_Jobs = $Active_Jobs -ne $Job

  }


   if ($Active_Jobs.count -eq 0 -and $index -eq $some_list.Length){

   break

  }


   Start-Sleep -Seconds 2

}

Write-Host "Jobs completed!"

The above code is provided as-is! Do not expect any kind of support from me on it.

0 Kudos
LucD
Leadership
Leadership

Nice one.
And yes, I support your suggestion on the scope option.
Why don't you raise that as an idea in PowerCLI Ideas?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos