I am trying to create a script that collect a lot of info from VM's and also set info if needed. That is not my problem. The problem is that there is alot of VM's and it takes about 5-8 sec pr VM. I have create a small sample script to run a process on several VM's at the same time, but it fails in mysterious ways :smileycry: If I only allow 1 thread everything is behaving as expected. More than one thread, it fails with messages that states the PowerCLI cmdlet are not known "The term 'Set-PowerCLIConfiguration' is not recognized as the name of a cmdlet"
I am using PowerCLI 6
Any ideas on whats wrong?
Below is my sample code (some of it is from Google )
# The per VM codeblock sample
$scriptBlock = {
param($VMName, $vcenter, $session)
Write-Host "VMThread - Start for: $($VMName)"
Import-Module VMware.VimAutomation.Core -Global
Write-Host "VMThread - Modules loaded: $(Get-Module)"
$Temp = Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Scope Session -Confirm:$false
$myVI = Connect-VIServer -Server $vcenter -Session $session
$Myvm = Get-VM -Name $VMName
if($Myvm)
{
Write-Host "VMThread - Getting VM info: $($Myvm.Name)"
$newInfo = New-Object PSObject
$newInfo | Add-Member -Type noteproperty -Name VMName -Value $Myvm.Name
$newInfo | Add-Member -Type noteproperty -Name MemMB -Value $Myvm.MemoryMB
$newInfo
Start-Sleep -Seconds 3
}
else
{
Write-Host "VMThread - No info for VM: $($VMName) - Modules: $(Get-Module)" -ForegroundColor Red
}
Write-Host "VMThread - Done: $($VMName) - Error Count: $($Error.Count) - $Error"
}
#Start the threads and wait for result
function RunTest{
param
(
$VMs,
$Throttle = 1 #threads
)
process
{
$sessionstate = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()
$sessionstate.ImportPSModule("VMware.VimAutomation.Core")
$RunspacePool = [runspacefactory]::CreateRunspacePool(1, $Throttle, $sessionstate, $Host)
$RunspacePool.Open()
$Jobs = @()
Write-Host "Starting threads.."
foreach ($oneVM in $VMs)
{
$Job = [powershell]::Create().AddScript($ScriptBlock).AddArgument($oneVM.Name).AddArgument($global:DefaultVIServer.Name).AddArgument($global:DefaultVIServer.SessionSecret)
$Job.RunspacePool = $RunspacePool
$Jobs += New-Object PSObject -Property @{RunVM = $_.Name; Pipe = $Job; Result = $Job.BeginInvoke()}
}
Write-Host "Start waiting.."
Do
{
Start-Sleep -Seconds 1
}
While ( $Jobs.Result.IsCompleted -contains $false)
Write-Host "All jobs completed!"
$Results = @()
ForEach ($Job in $Jobs)
{
$Results += $Job.Pipe.EndInvoke($Job.Result)
}
$RunspacePool.Close()
Write-Host "`n`nReturning $($Results.Count) objects..." -ForegroundColor Cyan
$Results
}
}
# Logon to vcenter
$user = Get-Credential domain\user
$vi = connect-viserver myvCenter -Credential $user
$AllVM = Get-VM # Return all my VM's, 2 in my test vcenter
RunTest -VMs $AllVM -Throttle 1 # Runs OK, returns one object for each VM
RunTest -VMs $AllVM -Throttle 2 # Fails
Can you display $PSModulePath in each thread ?
Is the content correct ?
Or try doing a 'Get-Module -ListAvailable' in each thread.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Hi
Sorry for the slow response, but i did not get a notification mail :smileyplain:
Outputs from the 2 commands are similar on all threads. It looks like this:
VMThread - PSModulePath: C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules\;C:\Program Files\Microsoft Monitoring Agent
\Agent\PowerShell\;C:\Program Files (x86)\Microsoft SQL Server\110\Tools\PowerShell\Modules\
VMThread - Get-Module: VMware.VimAutomation.Cis.Core VMware.VimAutomation.Cloud VMware.VimAutomation.Core VMware.VimAuto
mation.HA VMware.VimAutomation.PCloud VMware.VimAutomation.SDK VMware.VimAutomation.Storage VMware.VimAutomation.Vds App
Locker Appx BestPractices BitsTransfer BranchCache CimCmdlets DirectAccessClientComponents Dism DnsClient International
iSCSI IscsiTarget ISE Kds Microsoft.PowerShell.Diagnostics Microsoft.PowerShell.Host Microsoft.PowerShell.Management Mic
rosoft.PowerShell.Security Microsoft.PowerShell.Utility Microsoft.WSMan.Management MMAgent MsDtc NetAdapter NetConnectio
n NetEventPacketCapture NetLbfo NetNat NetQos NetSecurity NetSwitchTeam NetTCPIP NetworkConnectivityStatus NetworkTransi
tion NFS PcsvDevice PKI PrintManagement PSDesiredStateConfiguration PSDiagnostics PSLog PSScheduledJob PSWorkflow PSWork
flowUtility RemoteDesktop ScheduledTasks SecureBoot ServerCore ServerManager ServerManagerTasks SmbShare SmbWitness Soft
wareInventoryLogging StartScreen Storage TLS TroubleshootingPack TrustedPlatformModule UserAccessLogging VpnClient Wdac
Whea WindowsDeveloperLicense WindowsErrorReporting WindowsSearch Microsoft.MonitoringAgent.PowerShell SQLASCMDLETS SQLPS
Seems like module exports cmdlets in one runspace only:
ModuleType Version Name ExportedCommands
---------- ------- ---- ----------------
Manifest 3.1.0.0 Microsoft.PowerShell.Management {Add-Computer, Add-Content, Checkpoint-Computer, Clear-Content...}
Manifest 3.1.0.0 Microsoft.PowerShell.Utility {Add-Member, Add-Type, Clear-Variable, Compare-Object...}
Manifest 6.0.0.0 VMware.VimAutomation.Core
Manifest 6.0.0.0 VMware.VimAutomation.Sdk
Manifest 3.1.0.0 Microsoft.PowerShell.Management {Add-Computer, Add-Content, Checkpoint-Computer, Clear-Content...}
Manifest 3.1.0.0 Microsoft.PowerShell.Utility {Add-Member, Add-Type, Clear-Variable, Compare-Object...}
Manifest 6.0.0.0 VMware.VimAutomation.Core {Add-PassthroughDevice, Add-VirtualSwitchPhysicalNetworkAdapter, Add-VMHost, Add-VMHostNtpServer...}
Manifest 6.0.0.0 VMware.VimAutomation.Sdk
I guess this is a bug and should be passed over to VMware.
Previous PowerCLI version worked well with RunSpaces
I don't think this is really a bug, but I suspect it is due to the fact the current PowerCLI release (6R1) uses a mix of modules and pssnapins.
In fact this causes some other minor issues as well.
Can't wait to see a full module release.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Hi CNI0 !
Basically the fastest way to do what you wanted is this one:
$allVms = Get-View -ViewType VirtualMachine -Property Name,"Config.Hardware","Config.Template" -Filter @{"Config.Template"="False"}
$allVms | Select @{N="VMName";E={$_.Name}},@{N="MemMB";E={$_.Config.Hardware.MemoryMB}} | Out-GridView
You'll be impressed with how fast it is, even without runspaces!
Or, if you really want to use runspaces, then you could use my PS module, which contains functions to work with RunSpaces like with Jobs. You could grab it here: monosoul/basic_functions.psm1 · GitHub
(readme may be little outdated, byt every function has built-in documentation)
Here is an example code for what you wanted to do:
$user = Get-Credential domain\user
$vi = connect-viserver myvCenter -Credential $user
$snapins = @()
$modules = @()
$snapins += "VMware.VimAutomation.Core"
if ((Get-PowerCLIVersion).Major -le 5) {
$snapins += "VMware.VimAutomation.Vds"
$snapins += "VMware.VimAutomation.Storage"
}
else {
$modules += "VMware.VimAutomation.Vds"
$modules += "VMware.VimAutomation.Storage"
}
$AllVM = Get-VM
#basically it's still better to use Get-View as it'd be a lot faster, but i'll stick to your example
#$AllVM = Get-View -ViewType VirtualMachine -Property Name,"Config.Hardware","Config.Template" -Filter @{"Config.Template"="False"}
#making synchronized queue out of array
[System.Collections.Queue]$AllVM = [System.Collections.Queue]::Synchronized( ([System.Collections.Queue]$AllVM) )
#creatin synchronized array list for results
$newInfo = [System.Collections.ArrayList]::Synchronized( (New-Object System.Collections.ArrayList) )
Start-RSJobs -ScriptBlock {
param(
$AllVM,
$newInfo
)
while ($AllVM.Count -gt 0) {
$Myvm = $AllVM.Dequeue()
$newInfo.Add((New-Object PSObject -Property @{
VMName = $Myvm.Name
MemMB = $Myvm.MemoryMB
}))
}
} -snapinsToImport $snapins `
-modulesToImport $modules `
-variablesToImport @("DefaultVIServer", "DefaultVIServers") `
-ArgumentList $AllVM, $newInfo `
| Wait-RSJob | Remove-RSJob
$newInfo
As of modules importing only in first runspace - yeah, I've faced that issue too even with PCLI6R3, and I think it's some kind of bug. But for now I solved it by using snap-in version of "VMware.VimAutomation.Core" (other modules are importing fine in every runspace).
I found a solution that allows me to still run multithread using runspaces. Since runspaces aren't threadsafe, you have to run each runspace outofprocess. You don't need to use snapins with this solution. This thread is unanswered for 2 years. I guess it's not priority for Vmware but I figured I'll share it as I haven't found a good solution by googling for past 2 days. Here's what I had to do in my c# code. You just need to make adjustments in your powershell code to do the same by calling the RunspaceFactory.CreateOutOfProcessRunspace() Can you send me your code that you converted using the SDK? That would be helpful in my project too. Thanks
public async Task TestPowercli(string name, string vcenterHost) {
if (string.IsNullOrWhiteSpace(name)) { return; }
if (string.IsNullOrWhiteSpace(vcenterHost)) { return; }
instanceName = name.Trim();
int timeoutMins = 5;
string script = @"c:\temp\testpowercli\testpowercli.ps1 " + instanceName + " " + vcenterHost + " -verbose";
PowerShellProcessInstance instance = new PowerShellProcessInstance(new Version(5, 0), null, null, false);
using (Runspace runspace = RunspaceFactory.CreateOutOfProcessRunspace(new TypeTable(new string[0]), instance)) {
PowerShell ps = PowerShell.Create();
runspace.Open();
ps.Runspace = runspace;
ps.AddScript(script);
outputCollection.DataAdded += outputCollection_DataAddedPowercli;
// the streams (Error, Debug, Progress, etc) are available on the PowerShell instance.
// we can review them during or after execution.
// we can also be notified when a new item is written to the stream (like this):
ps.Streams.Error.DataAdded += Error_DataAddedPowercli;
ps.Streams.Verbose.DataAdded += Verbose_DataAddedPowercli;
IAsyncResult result = ps.BeginInvoke<PSObject, PSObject>(null, outputCollection);
DateTime start = DateTime.Now;
while (result.IsCompleted == false) {
if ((DateTime.Now - start).TotalMinutes > timeoutMins) {
Clients.All.getPowercliMessage(instanceName, "ERROR: Time out exceeded after " + timeoutMins + " minutes");
break;
}
await Task.Delay(1000);
}
}
Clients.All.getPowercliMessage(instanceName, "Done");
}
Hi monosoul,
Thanks so much for sharing your code. I have changed it slightly for my purposes and turned it into a module on github. The objective was to provide a simple framework for people to customize and add the things they are interested in returning from ESX (using Runspaces of course).
It seems to be working fine with the latest PowerCLI (tested on 6.5.2). I am open to ways to make it better.
-Mike
This post saved my soul. If you're like me, you maybe needed some deeper separation between the powershell runspaces and wanted to do multithreaded copies of files to remote datastores. The problem with the current powercli implementation is that it maps psdrives in a local scope, but makes the defaultviservers variable in a global scope. My recommendation to VMware is that the next version of powercli includes an option on the "Connect-VIServer" cmdlet to specify a scope of that connection, or at least patch the psdrives for "vmstores" and "vis" to a global scope that can be shared among runspaces.
I took the liberty of writing it out in powershell. This basically is an alternative to a runspace pool and should work in any powershell version past 3.0:
$index = 0
while ($true) {
if (($Active_Jobs.count -lt $num_threads) -and $index -lt $some_list.Length) {
$job = ($some_list[$index])
Write-Host "Creating job for $job"
$psinstance = [System.Management.Automation.PowerShell]::create()
Add-Member -InputObject $psinstance -NotePropertyName "request_stop" -NotePropertyValue "N"
$psinstance.AddScript($script_block).AddArgument($some_list[$index]) | Out-Null
$index++
$runspace = [runspacefactory]::CreateOutOfProcessRunspace([System.Management.Automation.Runspaces.TypeTable]::GetDefaultTypeFiles())
if([Console]::InputEncoding -is [Text.UTF8Encoding] -and [Console]::InputEncoding.GetPreamble().Length -ne 0) {
[Console]::InputEncoding = New-Object Text.UTF8Encoding $false
}
$runspace.Open()
$psinstance.Runspace = $runspace
$InputStream = New-Object -Typename System.Management.Automation.PSDataCollection[PSObject]
$OutputStream = New-Object -Typename System.Management.Automation.PSDataCollection[PSObject]
$Active_Jobs += New-Object PSObject -Property @{
Thread = $psinstance
Handle = $psinstance.BeginInvoke($InputStream,$OutputStream)
runspace = $runspace
output = $OutputStream
}
Start-Sleep -Seconds 1
}
Foreach ($Job in $($Active_Jobs | Where-Object {$_.Thread.request_stop -eq "Y"})) {
$Job.Thread.Stop()
Write-Host $Job.output
$Job.Thread.Dispose()
$Job.Runspace.Close()
$Job.Runspace.Dispose()
$Active_Jobs = $Active_Jobs -ne $Job
}
Foreach ($Job in $($Active_Jobs | Where-Object {$_.Handle.IsCompleted -eq $true})) {
Write-Host $Job.output
$Job.Thread.EndInvoke($Job.Handle) 2>$null
$Job.Thread.Dispose()
$Job.Runspace.Close()
$Job.Runspace.Dispose()
$Active_Jobs = $Active_Jobs -ne $Job
}
if ($Active_Jobs.count -eq 0 -and $index -eq $some_list.Length){
break
}
Start-Sleep -Seconds 2
}
Write-Host "Jobs completed!"
The above code is provided as-is! Do not expect any kind of support from me on it.
Nice one.
And yes, I support your suggestion on the scope option.
Why don't you raise that as an idea in PowerCLI Ideas?
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference