Scripted total shutdown

[khughes]

I know variations of this question have been asked more than once, and after looking at different threads I've found pieces of my answers but not a complete script. I'm pretty new to scripting so bare with me please

What I'm looking to do is move away from the APC PowerChute Network Shutdown on all VM's/Hosts and possibly move to a scripted shutdown with powershell. What I've been trying to piece together is a script tha:

1) gracefully shuts down all VM's

• Wait timer for all VM's to shutdown

2) Hard shutdown of certain VM's that can't gracefully shut down

3) puts the hosts into maintenance mode

• wait timer to move into maintenance mode

4) shuts down the ESX hosts

Any direction of how I can do this (if it is possible?) or references where I should be looking would be extremely helpful.

• Kyle

[LucD · Accepted solution]

There is one catch to such a script, the Shutdown-VMGuest nor the ShutdownGuest method from the SDK return a status.

So it is difficult to determine when a guest will not shutdown the soft way.

A quick-and-dirty solution is to wait a predefined number of seconds.

Anyone know a better solution for this problem ?

There are currently no cmdlets for putting an ESX server in maintenance mode or for shutting down an ESX server.

Hence the use of the EnterMaintenanceMode_Task and ShutdownHost_Task methods from the SDK.

An alternative would be to use the VITK Extensions for this but the Extensions require PowerShell v2 CTP3 which might not be admissible in every shop.

$MyHost = <ESX-server>
$timeout = 0
$evacuatePoweredOffVms = $false
$force = $false
$WaitSeconds = 300

$esx = Get-VMHost $MyHost | Get-View

# Shutdown guests
Get-VMHost $MyHost | Get-VM | Where-Object {$_.PowerState -eq "PoweredOn"} | Shutdown-VMGuest

# Wait
sleep $WaitSeconds

# Hard poweroff the remaining guests
Get-VMHost $MyHost | Get-VM | Where-Object {$_.PowerState -eq "PoweredOn"} | Stop-VM -Confirm:$false

# Enter maintenance mode
if(-not $esx.runtime.inMaintenanceMode){
	$taskMoRef = $esx.EnterMaintenanceMode_Task($timeout,$evacuatePoweredOffVm)
	$task = Get-View $taskMoRef
	while($task.info.state -eq "running" -or $task.info.state -eq "queued"){$task = Get-View $taskMoRef}
}

# Shutdown
$taskMoRef = $esx.ShutdownHost_Task($force)
$task = Get-View $taskMoRef
while($task.info.state -eq "running" -or $task.info.state -eq "queued"){$task = Get-View $taskMoRef}

[khughes]

Thanks LucD

This gives me a great base to go off of. I noticed the same thing about not really returning anything when its shutdown but the timer should be just fine as I know about how long it takes for them to shut down. Our APC has about 30-40min of runtime so I have a lot of give when it comes to how fast it shuts down. I'll try what you posted and open up another thread if needed later on down the road.

Thanks again

• Kyle

[Dave_Mishchenko]

Is there a reason you want the host in maintenace mode? I haven't tried it lately with ESX, but with ESXi the host comes back up in maintenance mode and you have to then change that status.

[LucD]

If you have DRS in automatic mode, the switch to maintenance mode will vmotion all your running guests to other servers.

And yes, in ESX 3.5u3 it comes back in maintenance mode as well.

[khughes]

The main reason I was thinking about putting it in maintenance mode is that I figured it was a fail-safe to make sure everything was shutdown before killing the ESX host. If the APC device triggered a network shutdown, someone would be coming into the building to check out a) why there was a power failure and/or why the generator ran out of gas ... or b) why the server room turned into an oven

So them coming back into maintenance mode isn't that big of an issue for me considering we only have 4 hosts. We would power the VM's on ourselves manually anyways.

Also wouldn't the command "Get-VMHost $myhost | Set-VMhost -state mainenance" work as well?

• Kyle

[LucD]

Correct, that brings it to maintenance mode as well.

I'll update my script.

[khughes]

For the $WaitSeconds = 300 variable, I can define like WaitSecondsLong = 300 and WaitSecondsShort = 120 or along those lines? Or does the sleep command look for a WaitMinutes/Seconds or whatever?

Also the script is for a specific host, If I wanted to do a total network shutdown of all of our hosts, which is only 4, should I just eliminate the Get-VMhost $myhost from the script or should I do each section and have$myhost1, $myhost2, etc..

I wasn't sure if I do the Get-VM | Where-Object {$_.PowerState -eq "PoweredOn"} | Shutdown-VMGuest if that would be too much for VirtualCenter to process or would it schedule all the shutdowns and just pound out a couple at a time and queue up the rest?

• Kyle

[LucD]

I'm not sure I understand your question about the sleep cmdlet.

With some changes the script could be changed to stop all VMHosts.

Best would perhaps be to create a function that takes as a parameter the VMHostname.

You could then pipe the names of all your VMHosts to the function.

Something like this:

function Shutdown-VMHost{
  param($MyHost)
 
  $timeout = 0
  $evacuatePoweredOffVms = $false
  $force = $false
  $WaitSeconds = 300

  $esx = Get-VMHost $MyHost | Get-View

# Shutdown guests
  Get-VMHost $MyHost | Get-VM | Where-Object {$_.PowerState -eq "PoweredOn"} | Shutdown-VMGuest

# Wait
  sleep $WaitSeconds

# Hard poweroff the remaining guests
  Get-VMHost $MyHost | Get-VM | Where-Object {$_.PowerState -eq "PoweredOn"} | Stop-VM -Confirm:$false

# Enter maintenance mode
  if(-not $esx.runtime.inMaintenanceMode){
	$taskMoRef = $esx.EnterMaintenanceMode_Task($timeout,$evacuatePoweredOffVm)
	$task = Get-View $taskMoRef
	while($task.info.state -eq "running" -or $task.info.state -eq "queued"){$task = Get-View $taskMoRef}
  } 

# Shutdown
  $taskMoRef = $esx.ShutdownHost_Task($force)
  $task = Get-View $taskMoRef
  while($task.info.state -eq "running" -or $task.info.state -eq "queued"){$task = Get-View $taskMoRef}
}

Get-VMHost | %{Shutdown-VMHost $_.Name}

[VirtualTimm]

This is the script I've been working on based off of this example.

I've added "tiering" of the VMs to allow for an orderly shutdown based

on dependencies. I accomplished this by creating a custom field "Tier" and assigning it a

numeric value between 0 and 3 depending on the role of the VM. VMs in

Tier 0 get shutdown immediately and only when all VMs in that tier are

powered off does the script move on to shutting down the next tier. We

use PRTG for monitoring so I've used the web API they provide to pause the top level node before beginning shutdown of the system.

I've attached my script, but I couldn't figure out how to insert a script block like you did above.

[LucD]

Great script, very inventive use of the custom attribute.

PS: in the editor under the Plain Text tab, there is a button marked "Plain Text Markup Help". This will show all the markup codes.

For the box I used, use the block (without the spaces).

____________

Blog: LucD notes

Twitter: lucd22