ghettoUPSHostShutdown.pl

    Table of Contents

    • Author
    • Description
    • Category
    • Features
    • Requirements
    • Version Support
    • Usage
    • Types of Operations
    • Supported Use Cases
    • Configuration
    • Email Report
    • Sample Execution

     

     

    Author

    William Lam

    Description

    This script is a follow up to ghettoShutdown.pl/upsVIShutdown.pl which had only been tested and supported on ESX(i) 3.5 hosts. The new  script has been redesigned from the bottom up for robustness and  provides much more capablities and features when shutting down your  VMware environment via a UPS monitoring utility that is supported and  runs on VMware vMA.

    Users now have the option of specifying various shutdown operations for both the hosts ( shutdown/standby ) and VMs ( shutdown/suspend ) which also includes an option to utilize the auto startup/shutdown  manager priotize the shutdown order of each VM on a given host. In  addition, users can also specify a hostlist which will allow a user to  determine the order in which hosts will initiate the power down  sequence.

    This script will support any UPS monitoring utility that can run on vMA such as the open source apcupsd and the script will automatically figure out which host the vMA UPS  monitoring VM is running on and ensure that all VMs are down prior to  taking down the final host which runs the vMA monitoring VM.

    Note: This document will not  cover any specific UPS utility configurations, please follow the  documentation and best practices from the UPS vendor or choice. You just  need to substitute this script as part of the UPS execution process to  ensure your VMware environment will be shutdown.

    Category

    • Health Monitoring

     

    Features

    • Host shutdown operations that include shutdown and standby
    • VM shutdown operations that include shutdown, powerOff and suspend
    • Ability to make use of the auto startup/shutdown manager priority list
    • Dryrun mode to ensure you've configured your shutdown process correctly without actually shutting down your systems
    • Asynchronously process all hosts/VMs and block based on configurable time out value
    • Email backup reports

     

    Requirements

     

    Version Support

    • Supports ESXi 3.5 and 4.0 (licensed version only)
    • Supports ESX 3.5 and 4.0

     

    Issues/Feature Request

    Please join the ghettoUPSHostShutdown group to post comments/discussions

    Usage

    [vi-admin@scofield ups]$ ./ghettoUPSHostShutdown.pl
    Required command option 'host_operation' not specified.
    Required command option 'timeout' not specified.
    Required command option 'ups_vm' not specified.
    Required command option 'vm_operation' not specified.

    Synopsis: ./ghettoUPSHostShutdown.pl OPTIONS


    Command-specific options:
       --host_operation (required)
          Host Operation to perform [shutdown|standby|autoquery|dryrun]
       --hostfile
          Manual list out the ESX(i) hosts in which to prioritize the shutdown process
       --loglevel (default 'debug')
          Log level [info|debug]
       --logoutput (default '/tmp/ghettoUPSHostShutdown.log')
          UPS Log output
       --sendmail (default 'no')
          Email UPS shutdown log [yes|no]
       --timeout (required)
          Timeout value before shutting down all hosts (mintues)
       --ups_vm (required)
          UPS Monitoring VM
       --vm_operation (required)
          VM Operation to perform [suspend|shutdown|auto]

    Common VI options:
       --config (variable VI_CONFIG)
          Location of the VI Perl configuration file
       --credstore (variable VI_CREDSTORE)
          Name of the credential store file defaults to <HOME>/.vmware/credstore/vicredentials.xml on Linux and <APPDATA>/VMware/credstore/vicredentials.xml on Windows
       --encoding (variable VI_ENCODING, default 'utf8')
          Encoding: utf8, cp936 (Simplified Chinese), iso-8859-1 (German), shiftjis (Japanese)
       --help
          Display usage information for the script
       --passthroughauth (variable VI_PASSTHROUGHAUTH)
          Attempt to use pass-through authentication
       --passthroughauthpackage (variable VI_PASSTHROUGHAUTHPACKAGE, default 'Negotiate')
          Pass-through authentication negotiation package
       --password (variable VI_PASSWORD)
          Password
       --portnumber (variable VI_PORTNUMBER)
          Port used to connect to server
       --protocol (variable VI_PROTOCOL, default 'https')
          Protocol used to connect to server
       --savesessionfile (variable VI_SAVESESSIONFILE)
          File to save session ID/cookie to utilize
       --server (variable VI_SERVER, default 'localhost')
          VI server to connect to. Required if url is not present
       --servicepath (variable VI_SERVICEPATH, default '/sdk/webService')
          Service path used to connect to server
       --sessionfile (variable VI_SESSIONFILE)
          File containing session ID/cookie to utilize
       --url (variable VI_URL)
          VI SDK URL to connect to. Required if server is not present
       --username (variable VI_USERNAME)
          Username
       --verbose (variable VI_VERBOSE)
          Display additional debugging information
       --version
          Display version information for the script

     

     

    Types of Operations

    Host:

    • shutdown - Shutdown the host after all VMs have powered off or after timeout value
    • standby - Put host into stanby mode if supported and BMC/IPMI is configured, else operation defaults to shutdown
    • dryrun - Will provide detail log output of execution without performing any actions to host or VMs
    • autoquery - Provides summary of the current configured auto  startup/power down priority list if configured. This mode also aides in  user selecting an appporiate timeout value to set

     

    VM:

    • shutdown - Shutdown the guest if VMware Tools is running, else it's a hard power off
    • suspend - Suspends the guest ( Recommended action, quicker than shutdown )
    • auto - Use the configured auto power down configuration

     

    Supported Use Cases

    When your UPS monitoring utility detects a change in power and issues the shutdown process....

    1) Automatically find all ESX(i) hosts being managed by vMA and  initiaite the shutdown process asynchronously based on the configured  host and vm operations, specific order of hosts is not taken into  consideration

    e.g.

    ./ghettoUPSHostShutdown.pl --host_operation shutdown --vm_operation suspend --timeout 10 --ups_vm vMA-UPS-MONITOR

     

     

    2) Specify a prioritzed list of ESX(i) hosts being managed by  vMA and initiaite the shutdown asynchronously process based on the  configured host and vm operations, and specific order is used.

    e.g.

    Example of hostlist:

    [vi-admin@scofield ups]$ cat hostlist.txt
    esxi4-2.primp-industries.com
    esxi35u4.primp-industries.com
    esxi4-1.primp-industries.com

     

     

    ./ghettoUPSHostShutdown.pl --host_operation standby --vm_operation suspend --timeout 10 --ups_vm vMA-UPS-MONITOR --hostlist hostlist.txt

     

     

    3) Specify the priority and VM operations using ESX(i) auto  startup/shutdown manager which is to be configured manually by the user.  Use case #1 or #2 can be used in conjunction with this option which  allows a user to specify both priority of VMs that'll be shutdown and  host priority if the user choose so

    e.g.

    ./ghettoUPSHostShutdown.pl --host_operation standby --vm_operation auto --timeout 10 --ups_vm vMA-UPS-MONITOR

     

    When using VM operation auto, you'll also have an option to view what's currently configured and get assistances with deciding the timeout value to use. Please take a look at Sample Execution for more examples.

    Configuration

    1. Download and upload ghettoUPSHostShutdown.pl to your vMA 4.0 host

    2. Set the script have execution permission:

    [vi-admin@scofield ups]$ chmod +x ghettoUPSHostShutdown.pl

     

    3. You'll want to configure your UPS monitioring utitilty to execute  this script when there is a degradation with the UPS. Remember, that the  vMA VM does not need to power off prior to shutting down the final host  else the last host will not cleanly power off. 

    4. When executing the shutdown script, one required variable is timeout which is in minutes that specifies how long the system will wait for  all VMs to shutdown before initiating the host shutdown operation. This  allows all systems to start the shutdown process and once all the  operations have been sent, the script will then scan through all hosts  to check the number of poweredOn VMs and block until the timeout before  shutting down the hosts. For those using auto startup/shutdown manager,  you can use --host_operation autoquery IN conjunction with --vm_operation auto to view what is the current amount of time that's been configured and help you select a timeout value. At the end of the day, it's up to the user to use his/her best judgement in selecting an apprpoirate timeout value.

    Here is a sample run if you were to use the auto startup/shutdown  manager and wanted to see what is the amount of time you've configured  for each VM to take to shutdown and recommended timeout value:

    [vi-admin@scofield ups]$ ./ghettoUPSHostShutdown.pl --host_operation autoquery --vm_operation auto --timeout 10 --ups_vm vMA-UPS-MONITOR
            01-28-2010 22:26:56 --  info: =========== DRYRUN MODE ENABLED ghettoUPSHostShutdown.pl ==========
            01-28-2010 22:26:56 --  info: UPS_MONITORING_VM: vMA-UPS-MONITOR
            01-28-2010 22:26:56 --  info: VM_OPERATION: auto
            01-28-2010 22:26:56 --  info: HOST_OPERATION: autoquery
            01-28-2010 22:26:56 --  info: HOST_LIST: AUTOMATIC
            01-28-2010 22:26:56 --  info: ===================================================================

            01-28-2010 22:26:56 --  info: Found host: "reflex.primp-industries.com"
            01-28-2010 22:26:56 --  debug: Main: Login by vi-fastpass to: reflex.primp-industries.com
            01-28-2010 22:26:58 --  info: Host: "reflex.primp-industries.com" is not an ESX(i) host and will be ignored
            01-28-2010 22:26:58 --  debug: Main: Disconnect from: reflex.primp-industries.com

            01-28-2010 22:26:58 --  info: Found host: "esxi35u4.primp-industries.com"
            01-28-2010 22:26:58 --  debug: Main: Login by vi-fastpass to: esxi35u4.primp-industries.com
            01-28-2010 22:26:59 --  info: AUTOSTART MANAGER INFO
            01-28-2010 22:26:59 --  info: --------------------------------------
            01-28-2010 22:26:59 --  info: AUTOSTART_ENABLED = YES
            01-28-2010 22:26:59 --  info: DEFAULT_STOP_ACTION = Suspend
            01-28-2010 22:26:59 --  info: DEFAULT_STOP_DELAY = 2 minute
            01-28-2010 22:26:59 --  info: VM=vm-6   ORDER=1 DELAY=2 minute
            01-28-2010 22:26:59 --  info: TOTAL_STOP_DELAY = 2 minute
            01-28-2010 22:26:59 --  info: --------------------------------------
            01-28-2010 22:26:59 --  debug: Main: Disconnect from: esxi35u4.primp-industries.com

            01-28-2010 22:26:59 --  info: Found host: "esxi4-1.primp-industries.com"
            01-28-2010 22:26:59 --  debug: Main: Login by vi-fastpass to: esxi4-1.primp-industries.com
            01-28-2010 22:27:01 --  info: AUTOSTART MANAGER INFO
            01-28-2010 22:27:01 --  info: --------------------------------------
            01-28-2010 22:27:01 --  info: AUTOSTART_ENABLED = YES
            01-28-2010 22:27:01 --  info: DEFAULT_STOP_ACTION = Suspend
            01-28-2010 22:27:01 --  info: DEFAULT_STOP_DELAY = 2 minute
            01-28-2010 22:27:01 --  info: VM=vm-4   ORDER=1 DELAY=2 minute
            01-28-2010 22:27:01 --  info: VM=vm-1   ORDER=2 DELAY=2 minute
            01-28-2010 22:27:01 --  info: VM=vMA-UPS-MONITOR   ORDER=3 DELAY=1 minute
            01-28-2010 22:27:01 --  info: TOTAL_STOP_DELAY = 5 minute
            01-28-2010 22:27:01 --  info: --------------------------------------
            01-28-2010 22:27:01 --  debug: Main: Disconnect from: esxi4-1.primp-industries.com

            01-28-2010 22:27:01 --  info: Found host: "esxi4-2.primp-industries.com"
            01-28-2010 22:27:01 --  debug: Main: Login by vi-fastpass to: esxi4-2.primp-industries.com
            01-28-2010 22:27:02 --  info: AUTOSTART MANAGER INFO
            01-28-2010 22:27:02 --  info: --------------------------------------
            01-28-2010 22:27:02 --  info: AUTOSTART_ENABLED = YES
            01-28-2010 22:27:02 --  info: DEFAULT_STOP_ACTION = PowerOff
            01-28-2010 22:27:02 --  info: DEFAULT_STOP_DELAY = 2 minute
            01-28-2010 22:27:02 --  info: VM=vm-3   ORDER=1 DELAY=2 minute
            01-28-2010 22:27:02 --  info: VM=vm-5   ORDER=2 DELAY=2 minute
            01-28-2010 22:27:02 --  info: TOTAL_STOP_DELAY = 4 minute
            01-28-2010 22:27:02 --  info: --------------------------------------
            01-28-2010 22:27:02 --  debug: Main: Disconnect from: esxi4-2.primp-industries.com

            01-28-2010 22:27:02 --  info: RECOMMENDED_TIMEOUT_VALUE = > 11 minutes

            01-28-2010 22:27:02 --  info:
            01-28-2010 22:27:02 --  info: =========== DRYRUN MODE COMPLETED ghettoUPSHostShutdown.pl ========

     

    Here you'll see a summary of what is currently configured for the  shutdown process using the autostart manager. At the end of the output,  you'll get a recommended amount of time to set your timer. Users at the  end of the day will ultimately decide the amount of time that his/her  environment takes to shutdown, this is just an estimate based on a  user's configuration. Please test before implementing in production!

    Email Report

    The script has the capablity to email an HTML report on the results of  the backup upon completion, you'll need to fill out the following  variables within the script and then you can enable email functionality  by specifying commandline param --sendemail yes

    #################
    # EMAIL CONF
    #################

    my $EMAIL_HOST = "emailserver";
    my $EMAIL_DOMAIN = "localhost.localdomain";
    my $EMAIL_TO = 'William Lam <william@primp-industries.com.com>';
    my $EMAIL_FROM = 'ghettoUPSHostShutdown <ghettoUPSHostShutdown@primp-industries.com.com>';


     

     

    Sample Execution

    Host Operationdryrun
    VM Operationshutdown
    Timeoutanyvalue


    [vi-admin@scofield ups]$ ./ghettoUPSHostShutdown.pl --host_operation dryrun --vm_operation shutdown --ups_vm vMA-UPS-MONITOR --timeout 5
            01-31-2010 03:47:40 --  info: =========== DRYRUN MODE ENABLED ghettoUPSHostShutdown.pl ==========
            01-31-2010 03:47:40 --  info: UPS_MONITORING_VM: vMA-UPS-MONITOR
            01-31-2010 03:47:40 --  info: VM_OPERATION: shutdown
            01-31-2010 03:47:40 --  info: HOST_OPERATION: dryrun
            01-31-2010 03:47:40 --  info: HOST_LIST: AUTOMATIC
            01-31-2010 03:47:40 --  info: ===================================================================

            01-31-2010 03:47:40 --  info: Found host: "esxi4-6.primp-industries.com"
            01-31-2010 03:47:40 --  debug: Main: Login by vi-fastpass to: esxi4-6.primp-industries.com
            01-31-2010 03:47:42 --  info: Begin shutdown operation on VMs ...
            01-31-2010 03:47:42 --  info: Shutting down vm-1 via VMware Tools...
            01-31-2010 03:47:42 --  info: Found UPS Monitoring VM: vMA-UPS-MONITOR
            01-31-2010 03:47:43 --  info: Shutting down vm-3 via VMware Tools...
            01-31-2010 03:47:43 --  info: shutdown operation complete!
            01-31-2010 03:47:43 --  debug: Main: Disconnect from: esxi4-6.primp-industries.com

            01-31-2010 03:47:43 --  info: Found host: "esxi4-7.primp-industries.com"
            01-31-2010 03:47:43 --  debug: Main: Login by vi-fastpass to: esxi4-7.primp-industries.com
            01-31-2010 03:47:44 --  info: Begin shutdown operation on VMs ...
            01-31-2010 03:47:44 --  info: Hard Powering off vm-4, no VMware Tools found ...
            01-31-2010 03:47:44 --  info: Hard Powering off vm-5, no VMware Tools found ...
            01-31-2010 03:47:44 --  info: Shutting down vm-6 via VMware Tools...
            01-31-2010 03:47:44 --  info: shutdown operation complete!
            01-31-2010 03:47:44 --  debug: Main: Disconnect from: esxi4-7.primp-industries.com

            01-31-2010 03:47:44 --  info: Found host: "esxi4-8.primp-industries.com"
            01-31-2010 03:47:44 --  debug: Main: Login by vi-fastpass to: esxi4-8.primp-industries.com
            01-31-2010 03:47:46 --  info: Begin shutdown operation on VMs ...
            01-31-2010 03:47:46 --  info: Shutting down vm-7 via VMware Tools...
            01-31-2010 03:47:46 --  info: Hard Powering off vm-8, no VMware Tools found ...
            01-31-2010 03:47:46 --  info: Shutting down vm-9 via VMware Tools...
            01-31-2010 03:47:46 --  info: shutdown operation complete!
            01-31-2010 03:47:46 --  debug: Main: Disconnect from: esxi4-8.primp-industries.com

            01-31-2010 03:47:46 --  info: Verifying all VMs are either suspended or powered of - TIMEOUT set to: 5 min

            01-31-2010 03:47:46 --  debug: Login by vi-fastpass to: esxi4-7.primp-industries.com
            01-31-2010 03:47:47 --  info: Host: esxi4-7.primp-industries.com has 3/3 VMs still powered on!

            01-31-2010 03:47:47 --  debug: Login by vi-fastpass to: esxi4-6.primp-industries.com
            01-31-2010 03:47:48 --  info: Host: esxi4-6.primp-industries.com has 2/3 VMs still powered on!

            01-31-2010 03:47:48 --  debug: Login by vi-fastpass to: esxi4-8.primp-industries.com
            01-31-2010 03:47:49 --  info: Host: esxi4-8.primp-industries.com has 3/3 VMs still powered on!

            01-31-2010 03:47:49 --  info: Verification stage completed!

            01-31-2010 03:47:49 --  info: Putting hosts into "dryrun mode"
            01-31-2010 03:47:50 --  info: esxi4-7.primp-industries.com is now shutting down!
            01-31-2010 03:47:52 --  info: esxi4-8.primp-industries.com is now shutting down!
            01-31-2010 03:47:52 --  info: Finally putting UPS Monitoring VM host into "dryrun mode"
            01-31-2010 03:47:53 --  info: esxi4-6.primp-industries.com is now shutting down!
            01-31-2010 03:47:53 --  info:
            01-31-2010 03:47:53 --  info: =========== DRYRUN MODE COMPLETED ghettoUPSHostShutdown.pl ========

     

     

    Host Operationshutdown
    VM Operationshutdown
    Timeout5 min


    [vi-admin@scofield ups]$ ./ghettoUPSHostShutdown.pl --host_operation shutdown --vm_operation shutdown --ups_vm vMA-UPS-MONITOR --timeout 5
            01-28-2010 08:50:21 --  info: ================ STARTING ghettoUPSHostShutdown.pl ================
            01-28-2010 08:50:21 --  info: UPS_MONITORING_VM: vMA-UPS-MONITOR
            01-28-2010 08:50:21 --  info: VM_OPERATION: shutdown
            01-28-2010 08:50:21 --  info: HOST_OPERATION: shutdown
            01-28-2010 08:50:21 --  info: HOST_LIST: AUTOMATIC
            01-28-2010 08:50:21 --  info: ===================================================================

            01-28-2010 08:50:21 --  info: Found host: "esxi4-6.primp-industries.com"
            01-28-2010 08:50:21 --  debug: Main: Login by vi-fastpass to: esxi4-6.primp-industries.com
            01-28-2010 08:50:23 --  info: Begin shutdown operation on VMs ...
            01-28-2010 08:50:23 --  info: Shutting down vm-1 via VMware Tools...
            01-28-2010 08:50:23 --  info: Found UPS Monitoring VM: vMA-UPS-MONITOR
            01-28-2010 08:50:24 --  info: Shutting down vm-3 via VMware Tools...
            01-28-2010 08:50:24 --  info: shutdown operation complete!
            01-28-2010 08:50:24 --  debug: Main: Disconnect from: esxi4-6.primp-industries.com

            01-28-2010 08:50:24 --  info: Found host: "esxi4-7.primp-industries.com"
            01-28-2010 08:50:24 --  debug: Main: Login by vi-fastpass to: esxi4-7.primp-industries.com
            01-28-2010 08:50:26 --  info: Begin shutdown operation on VMs ...
            01-28-2010 08:50:26 --  info: Hard Powering off vm-4, no VMware Tools found ...
            01-28-2010 08:50:28 --  info: Hard Powering off vm-5, no VMware Tools found ...
            01-28-2010 08:50:30 --  info: Shutting down vm-6 via VMware Tools...
            01-28-2010 08:50:30 --  info: shutdown operation complete!
            01-28-2010 08:50:30 --  debug: Main: Disconnect from: esxi4-7.primp-industries.com

            01-28-2010 08:50:30 --  info: Found host: "esxi4-8.primp-industries.com"
            01-28-2010 08:50:30 --  debug: Main: Login by vi-fastpass to: esxi4-8.primp-industries.com
            01-28-2010 08:50:31 --  info: Begin shutdown operation on VMs ...
            01-28-2010 08:50:31 --  info: Shutting down vm-7 via VMware Tools...
            01-28-2010 08:50:31 --  info: Hard Powering off vm-8, no VMware Tools found ...
            01-28-2010 08:50:33 --  info: Shutting down vm-9 via VMware Tools...
            01-28-2010 08:50:33 --  info: shutdown operation complete!
            01-28-2010 08:50:33 --  debug: Main: Disconnect from: esxi4-8.primp-industries.com

            01-28-2010 08:50:33 --  info: Verifying all VMs are either suspended or powered of - TIMEOUT set to: 5 min

            01-28-2010 08:50:33 --  debug: Login by vi-fastpass to: esxi4-7.primp-industries.com
            01-28-2010 08:50:34 --  info: Host: esxi4-7.primp-industries.com has 1/3 VMs still powered on!

            01-28-2010 08:50:34 --  debug: Login by vi-fastpass to: esxi4-6.primp-industries.com
            01-28-2010 08:50:35 --  info: Host: esxi4-6.primp-industries.com has 2/3 VMs still powered on!

            01-28-2010 08:50:35 --  debug: Login by vi-fastpass to: esxi4-8.primp-industries.com
            01-28-2010 08:50:37 --  info: Host: esxi4-8.primp-industries.com has 2/3 VMs still powered on!

            01-28-2010 08:50:37 --  info: Count: 1 - Sleeping for 60secs...
            01-28-2010 08:51:37 --  debug: Login by vi-fastpass to: esxi4-7.primp-industries.com
            01-28-2010 08:51:38 --  info: Host: esxi4-7.primp-industries.com has 0/3 VMs powered on and is ready!

            01-28-2010 08:51:38 --  debug: Login by vi-fastpass to: esxi4-6.primp-industries.com
            01-28-2010 08:51:39 --  info: Host: esxi4-6.primp-industries.com has 0/3 VMs powered on and is ready!

            01-28-2010 08:51:39 --  debug: Login by vi-fastpass to: esxi4-8.primp-industries.com
            01-28-2010 08:51:40 --  info: Host: esxi4-8.primp-industries.com has 0/3 VMs powered on and is ready!

            01-28-2010 08:51:40 --  info: Verification stage completed!

            01-28-2010 08:51:40 --  info: Putting hosts into "shutdown mode"
            01-28-2010 08:51:42 --  info: esxi4-7.primp-industries.com is now shutting down!
            01-28-2010 08:51:44 --  info: esxi4-8.primp-industries.com is now shutting down!
            01-28-2010 08:51:44 --  info: Finally putting UPS Monitoring VM host into "shutdown mode"
            01-28-2010 08:51:46 --  info: esxi4-6.primp-industries.com is now shutting down!
            01-28-2010 08:51:46 --  info:
            01-28-2010 08:51:46 --  info: ================ COMPLETED ghettoUPSHostShutdown.pl ================