10 Replies Latest reply on Aug 12, 2010 11:46 AM by LucD

    Scripted total shutdown

    khughes Virtuoso

       

      I know variations of this question have been asked more than once, and after looking at different threads I've found pieces of my answers but not a complete script.  I'm pretty new to scripting so bare with me please

       

       

       

       

       

      What I'm looking to do is move away from the APC PowerChute Network Shutdown on all VM's/Hosts and possibly move to a scripted shutdown with powershell.  What I've been trying to piece together is a script tha:

       

       

      1) gracefully shuts down all VM's

       

       

       

        • Wait timer for all VM's to shutdown

       

       

      2) Hard shutdown of certain VM's that can't gracefully shut down

       

       

      3) puts the hosts into maintenance mode

       

       

        • wait timer to move into maintenance mode

       

       

      4) shuts down the ESX hosts

       

       

      Any direction of how I can do this (if it is possible?) or references where I should be looking would be extremely helpful. 

       

       

       

       

       

      • Kyle

       

       

        • 1. Re: Scripted total shutdown
          LucD Guru
          Community WarriorsvExpertUser Moderators

          There is one catch to such a script, the Shutdown-VMGuest nor the ShutdownGuest method from the SDK return a status.

          So it is difficult to determine when a guest will not shutdown the soft way.

          A quick-and-dirty solution is to wait a predefined number of seconds.

           

          Anyone know a better solution for this problem ?

           

          There are currently no cmdlets for putting an ESX server in maintenance mode or for shutting down an ESX server.

          Hence the use of the EnterMaintenanceMode_Task and ShutdownHost_Task methods from the SDK.

          An alternative would be to use the VITK Extensions for this but the Extensions require PowerShell v2 CTP3 which might not be admissible in every shop.

           

          $MyHost = <ESX-server>
          $timeout = 0
          $evacuatePoweredOffVms = $false
          $force = $false
          $WaitSeconds = 300
          
          $esx = Get-VMHost $MyHost | Get-View
          
          # Shutdown guests
          Get-VMHost $MyHost | Get-VM | Where-Object {$_.PowerState -eq "PoweredOn"} | Shutdown-VMGuest
          
          # Wait
          sleep $WaitSeconds
          
          # Hard poweroff the remaining guests
          Get-VMHost $MyHost | Get-VM | Where-Object {$_.PowerState -eq "PoweredOn"} | Stop-VM -Confirm:$false
          
          # Enter maintenance mode
          if(-not $esx.runtime.inMaintenanceMode){
               $taskMoRef = $esx.EnterMaintenanceMode_Task($timeout,$evacuatePoweredOffVm)
               $task = Get-View $taskMoRef
               while($task.info.state -eq "running" -or $task.info.state -eq "queued"){$task = Get-View $taskMoRef}
          }
          
          # Shutdown
          $taskMoRef = $esx.ShutdownHost_Task($force)
          $task = Get-View $taskMoRef
          while($task.info.state -eq "running" -or $task.info.state -eq "queued"){$task = Get-View $taskMoRef}
          

           

          • 2. Re: Scripted total shutdown
            khughes Virtuoso

             

            Thanks LucD

             

             

             

             

             

            This gives me a great base to go off of.  I noticed the same thing about not really returning anything when its shutdown but the timer should be just fine as I know about how long it takes for them to shut down.  Our APC has about 30-40min of runtime so I have a lot of give when it comes to how fast it shuts down.  I'll try what you posted and open up another thread if needed later on down the road.

             

             

             

             

             

            Thanks again

             

             

            • Kyle

             

             

            • 3. Re: Scripted total shutdown
              Dave.Mishchenko Guru

              Is there a reason you want the host in maintenace mode?  I haven't tried it lately with ESX, but with ESXi the host comes back up in maintenance mode and you have to then change that status.

              • 4. Re: Scripted total shutdown
                LucD Guru
                Community WarriorsvExpertUser Moderators

                If you have DRS in automatic mode, the switch to maintenance mode will vmotion all your running guests to other servers.

                 

                And yes, in ESX 3.5u3 it comes back in maintenance mode as well.

                • 5. Re: Scripted total shutdown
                  khughes Virtuoso

                   

                  The main reason I was thinking about putting it in maintenance mode is that I figured it was a fail-safe to make sure everything was shutdown before killing the ESX host.  If the APC device triggered a network shutdown, someone would be coming into the building to check out a) why there was a power failure and/or why the generator ran out of gas ... or b) why the server room turned into an oven

                   

                   

                  So them coming back into maintenance mode isn't that big of an issue for me considering we only have 4 hosts.  We would power the VM's on ourselves manually anyways.

                   

                   

                   

                   

                   

                  Also wouldn't the command "Get-VMHost $myhost | Set-VMhost -state mainenance" work as well?  

                   

                   

                   

                   

                   

                  • Kyle

                   

                   

                  • 6. Re: Scripted total shutdown
                    LucD Guru
                    vExpertCommunity WarriorsUser Moderators

                    Correct, that brings it to maintenance mode as well.

                    I'll update my script.

                    • 7. Re: Scripted total shutdown
                      khughes Virtuoso

                       

                      For the $WaitSeconds = 300 variable, I can define like WaitSecondsLong = 300 and WaitSecondsShort = 120 or along those lines?  Or does the sleep command look for a WaitMinutes/Seconds or whatever?

                       

                       

                       

                      Also the script is for a specific host, If I wanted to do a total network shutdown of all of our hosts, which is only 4, should I just eliminate the Get-VMhost $myhost from the script or should I do each section and have$myhost1, $myhost2, etc..

                       

                       

                       

                       

                       

                      I wasn't sure if I do the Get-VM | Where-Object {$_.PowerState -eq "PoweredOn"} | Shutdown-VMGuest   if that would be too much for VirtualCenter to process or would it schedule all the shutdowns and just pound out a couple at a time and queue up the rest?

                       

                       

                       

                       

                       

                      • Kyle

                       

                       

                      • 8. Re: Scripted total shutdown
                        LucD Guru
                        User ModeratorsCommunity WarriorsvExpert

                        I'm not sure I understand your question about the sleep cmdlet.

                         

                        With some changes the script could be changed to stop all VMHosts.

                        Best would perhaps be to create a function that takes as a parameter the VMHostname.

                         

                        You could then pipe the names of all your VMHosts to the function.

                         

                        Something like this:

                        function Shutdown-VMHost{
                          param($MyHost)
                         
                          $timeout = 0
                          $evacuatePoweredOffVms = $false
                          $force = $false
                          $WaitSeconds = 300
                        
                          $esx = Get-VMHost $MyHost | Get-View
                        
                        # Shutdown guests
                          Get-VMHost $MyHost | Get-VM | Where-Object {$_.PowerState -eq "PoweredOn"} | Shutdown-VMGuest
                        
                        # Wait
                          sleep $WaitSeconds
                        
                        # Hard poweroff the remaining guests
                          Get-VMHost $MyHost | Get-VM | Where-Object {$_.PowerState -eq "PoweredOn"} | Stop-VM -Confirm:$false
                        
                        # Enter maintenance mode
                          if(-not $esx.runtime.inMaintenanceMode){
                             $taskMoRef = $esx.EnterMaintenanceMode_Task($timeout,$evacuatePoweredOffVm)
                             $task = Get-View $taskMoRef
                             while($task.info.state -eq "running" -or $task.info.state -eq "queued"){$task = Get-View $taskMoRef}
                          } 
                        
                        # Shutdown
                          $taskMoRef = $esx.ShutdownHost_Task($force)
                          $task = Get-View $taskMoRef
                          while($task.info.state -eq "running" -or $task.info.state -eq "queued"){$task = Get-View $taskMoRef}
                        }
                        
                        Get-VMHost | %{Shutdown-VMHost $_.Name}
                        

                         

                        • 9. Re: Scripted total shutdown
                          VirtualTimm Lurker

                           

                          This is the script I've been working on based off of this example.

                          I've added "tiering" of the VMs to allow for an orderly shutdown based

                          on dependencies. I accomplished this by creating a custom field "Tier" and assigning it a

                          numeric value between 0 and 3 depending on the role of the VM. VMs in

                          Tier 0 get shutdown immediately and only when all VMs in that tier are

                          powered off does the script move on to shutting down the next tier. We

                          use PRTG for monitoring so I've used the web API they provide to pause the top level node before beginning shutdown of the system.

                           

                           

                          I've attached my script, but I couldn't figure out how to insert a script block like you did above.

                           

                           

                          • 10. Re: Scripted total shutdown
                            LucD Guru
                            Community WarriorsUser ModeratorsvExpert

                            Great script, very inventive use of the custom attribute.

                             

                            PS: in the editor under the Plain Text tab, there is a button marked "Plain Text Markup Help". This will show all the markup codes.

                            For the box I used, use the block (without the spaces).

                             

                             

                             

                             

                            ____________

                            Blog: LucD notes

                            Twitter: lucd22