9 Replies Latest reply on Jun 20, 2014 2:03 PM by adamjg

    Fastest way to script powering on/off thousands of VMs

    adamjg Enthusiast

      Hi all, I'm looking for the fastest way to power off/on say 2000 VMs.  We're doing a SAN vendor bake-off for a new VDI project and want to find the fastest way possible to script this.  I ran this to power down VMs this morning:

       

      Get-VM | Where {$_.PowerState -eq "PoweredOn"} | Stop-VM -RunAsync -Confirm:$False

       

      This started to run pretty fast, but still took almost 30 minutes to finish.  This cluster only contains VDI VMs, so I don't have to worry about shutting down vCenter or anything.  So what's the fastest way to get this accomplished?  Disabling HA and setting DRS to manual then sending reboot commands to the hosts themselves would be much faster, but based on future testing scenarios I don't really want to do this and would rather find a way to do this with PowerCLI.  Any ideas?

        • 1. Re: Fastest way to script powering on/off thousands of VMs
          DZ1 Hot Shot

          I'm not sure about a faster way, but you are shutting down the VMs hard,  Shutdown-VMGuest would be better, although, it would take much longer, and a program or user logged on could stop the shutdown process, but it is safer. 

          • 2. Re: Fastest way to script powering on/off thousands of VMs
            adamjg Enthusiast

            DZ1 wrote:

             

            I'm not sure about a faster way, but you are shutting down the VMs hard,  Shutdown-VMGuest would be better, although, it would take much longer, and a program or user logged on could stop the shutdown process, but it is safer.

             

            Yeah, but at the point where we were shutting down/powering off, the VMs might have been in a state where they weren't fully powered up yet.  Basically we're just testing boot storms and the VMs are streamed off of Citrix PVS.  There's no worry about data loss here since it's only to test the storage.  Still, all that said, Shutdown-VMGuest will the same parameters would still take 30 minutes to run, since it would go at the same speed as Stop-VM.  Thanks for the reply.

            • 3. Re: Fastest way to script powering on/off thousands of VMs
              LucD Guru
              vExpertCommunity WarriorsUser Moderators

              You could split up your 2000 VMs in a separate slices, and then use a Start-Job, with the Stop-VM in the code block, for each slice.

              That way you would be running multiple stops in parallel.

              But at one point you will most probably hit the limits of your vSphere server

              Blog: http://lucd.info | Twitter: @LucD22 | PowerCLI Reference co-author: http://tinyurl.com/hkn4glz
              1 person found this helpful
              • 4. Re: Fastest way to script powering on/off thousands of VMs
                adamjg Enthusiast

                LucD wrote:

                 

                You could split up your 2000 VMs in a separate slices, and then use a Start-Job, with the Stop-VM in the code block, for each slice.

                That way you would be running multiple stops in parallel.

                But at one point you will most probably hit the limits of your vSphere server

                 

                Thanks for the reply Luc.  I was thinking more about this and just thought of something and like to run it past you.  In our VDI environment the hosts will be relatively static, especially for the purposes of the SAN test. What if I did a Start-Job for each host and connected directly to the host instead of going through vCenter?  Wouldn't that go much faster since the job is running on a host by host basis?  What do you think?

                • 5. Re: Fastest way to script powering on/off thousands of VMs
                  LucD Guru
                  Community WarriorsUser ModeratorsvExpert

                  That would be a good idea imho.

                  That way you avoid having to run everything through the vCenter.

                   

                  And you can still split up the VMs in several batches on a specific ESXi server.

                  Blog: http://lucd.info | Twitter: @LucD22 | PowerCLI Reference co-author: http://tinyurl.com/hkn4glz
                  1 person found this helpful
                  • 6. Re: Fastest way to script powering on/off thousands of VMs
                    adamjg Enthusiast

                    LucD wrote:

                     

                    That would be a good idea imho.

                    That way you avoid having to run everything through the vCenter.

                     

                    And you can still split up the VMs in several batches on a specific ESXi server.

                     

                    Ok so here's basically what I have so far, without the Start-Job:

                     

                    Connect-VIServer $vcenter | Out-Null

                    $VMHosts = Get-VMHost

                    Disconnect-VIServer $vcenter -Confirm:$false

                    Write-Host

                    Write-Host "Enter the root password to connect directly to each ESXi host in the cluster."

                    $cred = Get-Credential -Credential $null

                    ForEach ($VMHost in $VMHosts) {

                         Connect-VIServer $VMHost.Name -Credential $cred

                         Get-VM -Server $VMHost.Name | Where {$_.PowerState -eq "PoweredOff"} | Start-VM -RunAsync -Confirm:$false | Out-Null

                    }

                     

                    So I connect to the vCenter, grab a list of hosts, then disconnect from the vCenter, prompt for the root credentials on the hosts, then loop through each host and power the VMs on (or off, that's the other script block).

                     

                    I tried putting Start-Job -ScriptBlock in front of the 2nd line of the for loop, but it doesn't do anything.  I have a feeling it's with the way I'm passing the host variable in.  Can you help fix this?  This is basically what I'd want to do:

                     

                    ForEach ($VMHost in $VMHosts) {

                         Connect-VIServer $VMHost.Name -Credential $cred

                         Start-Job -ScriptBlock {Get-VM -Server $VMHost.Name | Where {$_.PowerState -eq "PoweredOff"} | Start-VM -RunAsync -Confirm:$false | Out-Null}  #these two lines should be on one line but the editor split them

                    }

                    • 7. Re: Fastest way to script powering on/off thousands of VMs
                      LucD Guru
                      User ModeratorsCommunity WarriorsvExpert

                      There's a good write-up from Clinton in Multithreading PowerCLI

                      Blog: http://lucd.info | Twitter: @LucD22 | PowerCLI Reference co-author: http://tinyurl.com/hkn4glz
                      • 8. Re: Fastest way to script powering on/off thousands of VMs
                        adamjg Enthusiast

                        Hey Luc, I did check that out, and have what I think is an acceptable context, but the jobs just never run on the hosts.  Here's the code I have now:

                         

                        Connect-VIServer $vcenter | Out-Null

                        $VMHosts = Get-VMHost

                        Disconnect-VIServer $vcenter -Confirm:$false

                        Write-Host

                        Write-Host "Enter the root password to connect directly to each ESXi host in the cluster."

                        $cred = Get-Credential -Credential $null

                        ForEach ($VMHost in $VMHosts) {

                             $ActiveHost = $VMHost.Name

                             Connect-VIServer $ActiveHost -Credential $cred

                             $poweroffjob = {Get-VM -Server $args[0] | Where {$_.PowerState -eq "PoweredOn"} | Stop-VM -RunAsync -Confirm:$false | Out-Null}

                             Start-Job $poweroffjob -ArgumentList $ActiveHost | Out-Null

                        }

                         

                        Sorry for the formatting but the code block button is gone in my editor.  This runs through and the jobs seem to be created and run (if I do a get-job after the script runs), but nothing is ever done on the hosts.  I've tried a number of different formats including the $using:ActiveHost variable, but nothing ever runs on the host.  If I do a Get-Job it shows the job completed and the block of code, but again, nothing on the hosts.  Any ideas?

                        • 9. Re: Fastest way to script powering on/off thousands of VMs
                          adamjg Enthusiast

                          Ahh, I figured it out.  I had to pass the Add-PSSnapin VMware.VimAutomation.Core and my Connect-VIServer lines into the job.  Once I did that it ran on each host.  Thanks for pushing me in the right direction!