Poweroff and Poweron 1000 VMs in Parallel

[shahb]

Hi

I've a maintenance is coming up for which I need to shutdown 1000+ VMs and once the Maintenance is complete, Need to start those VMs back on.

I've a list of 1000 VMs and I can run in for loop but that will take hours to shutdown these VMs and I only have 2 hours of Maintenance window, Can someone please help ?

Here is the script which I got from the community and it works as expected but it runs serially.

foreach ($vmlist in (Get-Content -Path C:\vmlist1.txt)){

$vm = Get-VM -Name $vmlist

Start-VM -VM $vm -Confirm:$false -RunAsync

}

Thanks & Regards

[LucD]

You are running a rather old PowerCLI version, and you should definitely considering upgrading to a newer version.

To fix the issue for now with the missing cmdlet, we can import the required module in the script used in the jobs.

Try like this

$script = {

   param(

   [string]$VMName,

   [string]$vcId

   )

   Import-Module VMware.VimAutomation.Core

   Connect-VIServer -Session $vcId

   Get-VM -Name $VMName | Shutdown-VMGuest -Confirm:$false -Verbose

}

$fileName = '.\input.csv'

$MaxJobs = 10

$jobs = @()

Import-Csv -Path $fileName | ForEach-Object -Process {

   if ($jobs.Count -le $MaxJobs) {

   $job = (Start-Job -Verbose -ScriptBlock $script -ArgumentList $_.VMName, $global:DefaultVIServer.SessionId)

   $jobs += $job

   }

   else {

   Wait-Job -Job $jobs

   $jobs = @()

   }

}

[shahb]

Ok Thanks Sir.

I will upgrade and see if that makes any difference.

After adding the Import-Module , both Job 19 and 17 shows Blocked

[LucD]

That is probably because you have prompt in the background script.

I suspect it might be due to one or more Warning messages.

We can disable those before starting.

$script = {

   param(

   [string]$VMName,

   [string]$vcId

   )

   Set-PowerCLIConfiguration -DisplayDeprecationWarnings:$false -Scope Session -Confirm:$false

   Connect-VIServer -Session $vcId

   Get-VM -Name $VMName | Shutdown-VMGuest -Confirm:$false

}

$fileName = '.\input.csv'

$MaxJobs = 10

$jobs = @()

Import-Csv -Path $fileName | ForEach-Object -Process {

   if ($jobs.Count -le $MaxJob) {

   Write-Host "Job for $($_.VMName)"

   $job = (Start-Job -Verbose -ScriptBlock $script -ArgumentList $_.VMName, $global:DefaultVIServer.SessionId)

   Write-Host "`tStarted job $($job.Id)"

   Write-Host "`tRunning jobs $($jobs.Count)"

   $jobs += $job

   }

   else {

   Wait-Job -Job $jobs

   $jobs = @()

   }

}

[shahb]

OK Sir,

This time it went to Running and then Blocked state

[LucD]

Damn, we forgot the vCenter name.

Try now.

$script = {

   param(

   [string]$VMName,

   [string]$vcServer,

   [string]$vcId

   )

   Import-Module VMware.VimAutomation.Core

   Connect-VIServer -Server $vcServer -Session $vcId

   Get-VM -Name $VMName | Shutdown-VMGuest -Confirm:$false -Verbose

}

$fileName = '.\input.csv'

$MaxJobs = 10

$jobs = @()

Import-Csv -Path $fileName | ForEach-Object -Process {

   if ($jobs.Count -le $MaxJobs) {

   $job = (Start-Job -Verbose -ScriptBlock $script -ArgumentList $_.VMName, $global:DefaultVIServer.Name, $global:DefaultVIServer.SessionId)

   $jobs += $job

   }

   else {

   Wait-Job -Job $jobs

   $jobs = @()

   }

}

[shahb]

I guess already added that before running.

but will try again

[shahb]

Ok it shows running and then completed state but didn't poweroff the VMs

[LucD]

Did you try to receive the output of one of those that completed?

[sk84]

Get-VM -Name $VMName | Shutdown-VMGuest -Confirm:$false -Verbose

shahb​, just a side note, because you write "Power Off" all the time. The script tries to perform a graceful shutdown of the operating system. And this only works if VMware Tools are installed in all VMs. Is that the case?

[LucD]

I think he already confirmed that in this answer Re: Poweroff and Poweron 1000 VMs in Parallel

[shahb]

May be change the shutdown command to this block  as this Block is working fine.

foreach($vmlist in (Get-Content -Path C:\Scripts\MDC\vmlist1.txt)){

$vm = Get-VM -Name $vmlist

if($vm.Guest.State -eq "Running"){

Shutdown-VMGuest -VM $vm -Confirm:$false -RunAsync

}

else{

       Stop-VM -VM $vm -Confirm:$false -RunAsync

   }

}

[LucD]

So you don't have VMware Tools installed on all the VMs?

[LucD · Accepted solution]

To cover that we can indeed adapt the background job script.

$script = {

   param(

   [string]$VMName,

   [string]$vcServer,

   [string]$vcId

   )

   Import-Module VMware.VimAutomation.Core

   Connect-VIServer -Server $vcServer -Session $vcId

   $vm = Get-VM -Name $VMName

   if($vm.Guest.State -eq "Running"){

  Shutdown-VMGuest -VM $vm -Confirm:$false -RunAsync

   }

   else{

   Stop-VM -VM $vm -Confirm:$false -RunAsync

   }

}

$fileName = '.\input.csv'

$MaxJobs = 10

$jobs = @()

Import-Csv -Path $fileName | ForEach-Object -Process {

   if ($jobs.Count -le $MaxJobs) {

   $job = (Start-Job -Verbose -ScriptBlock $script -ArgumentList $_.VMName, $global:DefaultVIServer.Name, $global:DefaultVIServer.SessionId)

   $jobs += $job

   }

   else {

   Wait-Job -Job $jobs

   $jobs = @()

   }

}

[shahb]

Great and Thank you for your time and support. You're awesome !!

Please add this to your book in future edition. This is a great script for someone who wants to perform mass action on vCenter/VMs.

Salute to you and you deserve to be given "Sir" title in PowerCli

Thank you LuCD Sir

[StephenMoll]

As a matter of curiosity...

If the number of jobs reaches 10, what happens to to the object being processed in that loop?

My mind is telling me that if Jobs.Count reaches 10 then in that iteration of the loop that object will not get processed, but all jobs will be allowed to compete. The next iteration of the loop for the next object will get processed. Or am I missing something in the way PowerShell works here?

[LucD]

It jobs into the else-block, where the Wait-Job cmdlet will wait till the jobs are finished.
This could be fine-tuned by just waiting for 1 job to finish, and then going back to the main loop.
But that would require something else than the Wait-Job cmdlet.

[StephenMoll]

Say the script has worked its way down a long list of VMs...

VM001

VM002

:

:

VM200

VM201

:

and so on.

Lets assume the rate of Start-Job instructions is slightly higher that the rate at which the jobs finish and we get to VM079. Job.Count is 10, and the script drops into the else block and it waits until all jobs are completed and then goes around the loop again. Won't VM079 now be left running? If not, what causes PowerShell to run the process block for VM079 a second time so that it can get into the main block of the if statement?

[LucD]

Yes, you are right.
It will skip 1 job when it has to wait.
The correct way would be

$script = {

   param(

   [string]$VMName,

   [string]$vcServer,

   [string]$vcId

   )

   Import-Module VMware.VimAutomation.Core

   Connect-VIServer -Server $vcServer -Session $vcId

   $vm = Get-VM -Name $VMName

   if ($vm.Guest.State -eq "Running")

   {

  Shutdown-VMGuest -VM $vm -Confirm:$false -RunAsync

   }

   else

   {

   Stop-VM -VM $vm -Confirm:$false -RunAsync

   }

}

$fileName = '.\input.csv'

$MaxJobs = 10

$jobs = @()

Import-Csv -Path $fileName | ForEach-Object -Process {

   if ($jobs.Count -le $MaxJobs)

   {

   $job = (Start-Job -Verbose -ScriptBlock $script -ArgumentList $_.VMName, $global:DefaultVIServer.Name, $global:DefaultVIServer.SessionId)

   $jobs += $job

   }

   else

   {

   Wait-Job -Job $jobs

   $jobs = @()

   $job = (Start-Job -Verbose -ScriptBlock $script -ArgumentList $_.VMName, $global:DefaultVIServer.Name, $global:DefaultVIServer.SessionId)

   $jobs += $job

   }

}

[StephenMoll]

How about this:

$script = {

   param(

      [string]$VMName,

      [string]$vcServer,

      [string]$vcId

   )

   Import-Module VMware.VimAutomation.Core

   Connect-VIServer -Server $vcServer -Session $vcId

   $vm = Get-VM -Name $VMName

   if ($vm.Guest.State -eq "Running")

   {

        Shutdown-VMGuest -VM $vm -Confirm:$false -RunAsync

   }

   else

   {

       Stop-VM -VM $vm -Confirm:$false -RunAsync

   }

}

$fileName = '.\input.csv'

$MaxJobs = 10

$jobs = @()

Import-Csv -Path $fileName | ForEach-Object -Process {

   while ($jobs.Count -ge $MaxJobs)

   {

       Start-Sleep -Seconds 5

       $jobs = $jobs | where State -ne Completed

   }

   $job = (Start-Job -Verbose -ScriptBlock $script -ArgumentList $_.VMName, $global:DefaultVIServer.Name, $global:DefaultVIServer.SessionId)

   $jobs += $job

}

[LucD]

That should work as well.