VMware Cloud Community
adamjg
Hot Shot
Hot Shot

Fastest way to script powering on/off thousands of VMs

Hi all, I'm looking for the fastest way to power off/on say 2000 VMs.  We're doing a SAN vendor bake-off for a new VDI project and want to find the fastest way possible to script this.  I ran this to power down VMs this morning:

Get-VM | Where {$_.PowerState -eq "PoweredOn"} | Stop-VM -RunAsync -Confirm:$False

This started to run pretty fast, but still took almost 30 minutes to finish.  This cluster only contains VDI VMs, so I don't have to worry about shutting down vCenter or anything.  So what's the fastest way to get this accomplished?  Disabling HA and setting DRS to manual then sending reboot commands to the hosts themselves would be much faster, but based on future testing scenarios I don't really want to do this and would rather find a way to do this with PowerCLI.  Any ideas?

9 Replies
DZ1
Hot Shot
Hot Shot

I'm not sure about a faster way, but you are shutting down the VMs hard,  Shutdown-VMGuest would be better, although, it would take much longer, and a program or user logged on could stop the shutdown process, but it is safer. 

adamjg
Hot Shot
Hot Shot

DZ1 wrote:

I'm not sure about a faster way, but you are shutting down the VMs hard,  Shutdown-VMGuest would be better, although, it would take much longer, and a program or user logged on could stop the shutdown process, but it is safer.

Yeah, but at the point where we were shutting down/powering off, the VMs might have been in a state where they weren't fully powered up yet.  Basically we're just testing boot storms and the VMs are streamed off of Citrix PVS.  There's no worry about data loss here since it's only to test the storage.  Still, all that said, Shutdown-VMGuest will the same parameters would still take 30 minutes to run, since it would go at the same speed as Stop-VM.  Thanks for the reply.

Reply
0 Kudos
LucD
Leadership
Leadership

You could split up your 2000 VMs in a separate slices, and then use a Start-Job, with the Stop-VM in the code block, for each slice.

That way you would be running multiple stops in parallel.

But at one point you will most probably hit the limits of your vSphere server


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

adamjg
Hot Shot
Hot Shot

LucD wrote:

You could split up your 2000 VMs in a separate slices, and then use a Start-Job, with the Stop-VM in the code block, for each slice.

That way you would be running multiple stops in parallel.

But at one point you will most probably hit the limits of your vSphere server

Thanks for the reply Luc.  I was thinking more about this and just thought of something and like to run it past you.  In our VDI environment the hosts will be relatively static, especially for the purposes of the SAN test. What if I did a Start-Job for each host and connected directly to the host instead of going through vCenter?  Wouldn't that go much faster since the job is running on a host by host basis?  What do you think?

Reply
0 Kudos
LucD
Leadership
Leadership

That would be a good idea imho.

That way you avoid having to run everything through the vCenter.

And you can still split up the VMs in several batches on a specific ESXi server.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

adamjg
Hot Shot
Hot Shot

LucD wrote:

That would be a good idea imho.

That way you avoid having to run everything through the vCenter.

And you can still split up the VMs in several batches on a specific ESXi server.

Ok so here's basically what I have so far, without the Start-Job:

Connect-VIServer $vcenter | Out-Null

$VMHosts = Get-VMHost

Disconnect-VIServer $vcenter -Confirm:$false

Write-Host

Write-Host "Enter the root password to connect directly to each ESXi host in the cluster."

$cred = Get-Credential -Credential $null

ForEach ($VMHost in $VMHosts) {

     Connect-VIServer $VMHost.Name -Credential $cred

     Get-VM -Server $VMHost.Name | Where {$_.PowerState -eq "PoweredOff"} | Start-VM -RunAsync -Confirm:$false | Out-Null

}

So I connect to the vCenter, grab a list of hosts, then disconnect from the vCenter, prompt for the root credentials on the hosts, then loop through each host and power the VMs on (or off, that's the other script block).

I tried putting Start-Job -ScriptBlock in front of the 2nd line of the for loop, but it doesn't do anything.  I have a feeling it's with the way I'm passing the host variable in.  Can you help fix this?  This is basically what I'd want to do:

ForEach ($VMHost in $VMHosts) {

     Connect-VIServer $VMHost.Name -Credential $cred

     Start-Job -ScriptBlock {Get-VM -Server $VMHost.Name | Where {$_.PowerState -eq "PoweredOff"} | Start-VM -RunAsync -Confirm:$false | Out-Null}  #these two lines should be on one line but the editor split them

}

Reply
0 Kudos
LucD
Leadership
Leadership

There's a good write-up from Clinton in Multithreading PowerCLI


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
adamjg
Hot Shot
Hot Shot

Hey Luc, I did check that out, and have what I think is an acceptable context, but the jobs just never run on the hosts.  Here's the code I have now:

Connect-VIServer $vcenter | Out-Null

$VMHosts = Get-VMHost

Disconnect-VIServer $vcenter -Confirm:$false

Write-Host

Write-Host "Enter the root password to connect directly to each ESXi host in the cluster."

$cred = Get-Credential -Credential $null

ForEach ($VMHost in $VMHosts) {

     $ActiveHost = $VMHost.Name

     Connect-VIServer $ActiveHost -Credential $cred

     $poweroffjob = {Get-VM -Server $args[0] | Where {$_.PowerState -eq "PoweredOn"} | Stop-VM -RunAsync -Confirm:$false | Out-Null}

     Start-Job $poweroffjob -ArgumentList $ActiveHost | Out-Null

}

Sorry for the formatting but the code block button is gone in my editor.  This runs through and the jobs seem to be created and run (if I do a get-job after the script runs), but nothing is ever done on the hosts.  I've tried a number of different formats including the $using:ActiveHost variable, but nothing ever runs on the host.  If I do a Get-Job it shows the job completed and the block of code, but again, nothing on the hosts.  Any ideas?

Reply
0 Kudos
adamjg
Hot Shot
Hot Shot

Ahh, I figured it out.  I had to pass the Add-PSSnapin VMware.VimAutomation.Core and my Connect-VIServer lines into the job.  Once I did that it ran on each host.  Thanks for pushing me in the right direction!

Reply
0 Kudos