ESX/ESXi APC/APCUPSD Host Shutdown VI Perl Toolkit Script (ghettoShutdown.pl/upsVIShutdown.pl) -- [ DEPRECATED ]

Version 3

    Description

    Many parties have looked for the possibility of monitoring power continuity and subsequently reacting to power outages with information supplied by a UPS agent installed in a virtual machine running on a free version of ESXi. Currently, one free solution How to configure ESXi to shutdown using an APC SmartUPS created by Joseph Holland facilitates shutdown/suspend procedures for online virtual machines in the event of a power outage given that there is some type of physical UPS system in place to provide backup power to the ESXi server. Joseph’s solution utilizes VMware VIMA and the open source utility apcupsd however, one needs to enable the unsupported SSH console in ESXi and setup paired SSH keys between VIMA and host to utilize it. Enabling the unsupported SSH console may not be the preferred method for some end users with respect to their support contracts held with VMware.

     

    The following solution outlined in this document is similar to Joseph’s work although it will not necessitate enabling the unsupported SSH console on the ESXi hosts. This is made possible by utilizing the VI Perl Toolkit to communicate with the ESXi hosts. The only possible drawback however is that next revision of ESXi may disallow write operations and require that the user purchase ESXi license(s) to make full use of the VI API (read/write).

     

    It must be pointed out that this implementation is built on top of initial findings made by Joseph Holland (APC VIOPS documentation) and Andrew Sullivan (VMware VIMA VI Fastpass disection). Please do visit Andrew’s blog at get-admin. Thanks to Duncan Epping for nudging the challenge via Twitter ;)...

     

    Note: Please ensure proper testing has been performed in a staging or development environment prior to implementing this solution in a production environment.

     

    Requirements

    ESX/ESXi 3.5+

    VMware VIMA

    VI Perl Toolkit (included in VIMA)

    APC SmartUPS or other "smart" UPS (may require another monitoring utility)

    apcupsd-3.14.5-1.el5.x86_64.rpm (VMware VIMA is 64bit RHEL VM)

     

    Usage

    [vi-admin@vima-ups-dev ~]$ ./ghettoShutdown.pl
    Required command option 'ups_vm' not specified.
    
    Synopsis: ./ghettoShutdown.pl OPTIONS
    
    
    Command-specific options:
       --sleep
          The amount of time (secs) to wait after a guestOS shutdown (default 15 secs)
       --ups_vm (required)
          The name of VM that is monitoring your UPS
    
    Common VI options:
       --config (variable VI_CONFIG)
          Location of the VI Perl configuration file
       --encoding (variable VI_ENCODING, default 'utf8')
          Encoding: utf8, cp936 (Simplified Chinese), iso-8859-1 (German), shiftjis (Japanese)
       --help
          Display usage information for the script
       --passthroughauth (variable VI_PASSTHROUGHAUTH)
          Attempt to use pass-through authentication
       --passthroughauthpackage (variable VI_PASSTHROUGHAUTHPACKAGE, default 'Negotiate')
          Pass-through authentication negotiation package
       --password (variable VI_PASSWORD)
          Password
       --portnumber (variable VI_PORTNUMBER)
          Port used to connect to server
       --protocol (variable VI_PROTOCOL, default 'https')
          Protocol used to connect to server
       --savesessionfile (variable VI_SAVESESSIONFILE)
          File to save session ID/cookie to utilize
       --server (variable VI_SERVER, default 'localhost')
          VI server to connect to. Required if url is not present
       --servicepath (variable VI_SERVICEPATH, default '/sdk/webService')
          Service path used to connect to server
       --sessionfile (variable VI_SESSIONFILE)
          File containing session ID/cookie to utilize
       --url (variable VI_URL)
          VI SDK URL to connect to. Required if server is not present
       --username (variable VI_USERNAME)
          Username
       --verbose (variable VI_VERBOSE)
          Display additional debugging information
       --version
          Display version information for the script
    

     

     

     

     

    Overview of the scripts:

    ghettoShutdown.pl - This script initiates the shutdown of all VM(s) within an ESX/ESXi host excluding the virtual machine that's monitoring the UPS device and then shutdowns the host. It accepts two commandline parameters: --sleep the duration in seconds to wait after a VM has initiated the shutdown before moving onto the next VM (shutdownVM() is non-blocking function) and --ups_vm the name of the displayName of your VM that is monotiring the UPS device \[more details to come later\].

     

    upsVIShutdown.pl - This script is a wrapper which will hold the configurations of the order of hosts to shutdown. It may be used inconjunction with other UPS monitoring utility, though with our example, it'll be placed in the apccontrol script to execute upon a power interuption.

     

    Scenario

    2 ESXi 3.5u3 (free unlicensed edition):

     

    superion.primp-industries.com

    SQL-PROD-01

    Powered On

    Quentin

    Powered Off

    air_raid

    Powered On

    skydive

    Suspended

    fireflight

    Powered Off

    slingshot

    Powered On

     

    devastator.primp-industries.com

    VIMA-UPS-DEV

    Powered On

    bonecrusher

    Powered Off

    master_mix

    Suspended

    quickMigrate-X

    Powered Off

    scavenger

    Suspended

    hook

    Powered On

    long_haul

    Powered On

     

    Simulate a power interuption and initiate shutdown of superion and it's VM(s) then devastator and its VM(s) using VMware VIMA: VIMA-UPS-DEV. (note, due to the lack of our UPS being configured, this will be a simulation of the UPS utility initiate the shutdown)

     

    Since the script is written using the VI Perl Toolkit, this potentially could function on a Windows VM installed with the VI Perl Toolkit but you will not able to use VI Fastpass for authentication and credentials will need to be passed to ghettoShutdown.pl via the execution.

     

    Setup

    1. Ensure your ESX/ESXi host(s) are being managed by VMware VIMA

     

    2. Download ghettoShutdown.pl and save to /home/vi-admin and make sure it has executable permissions set (chmod +x ghettoShutdown.pl)

     

    3. Download upsVIShutdown.pl and save to /home/vi-admin and make sure it has executable permissions set (chmod +x upsVIShutdown.pl)

     

    4. Edit the following sections in upsVIShutdown.pl:

     

    A) Insert your ESX/ESXi host(s), use the hostname that you used to add to VIMA managment interface, if you're unsure run sudo vifp listservers

     

    Remember to make sure the host that is running your VIMA VM is listed as the last entry.


    my @hosts = ("superion.primp-industries.com","devastator.primp-industries.com");B) Modify the logoutput of the shutdown process if you like, default will go to /tmp/upsShutdown.log


    my $log_output = "/tmp/upsShutdown.log";C) Insert the displayName of VIMA or VM that is monitoring your UPS device (case sensitive)

    Very important step else you could end up shutting down the VM that is executing this script, we wouldn't want that


    my $ups_vm_name = "VIMA-UPS-DEV";D) Insert the number of seconds to delay after a guestOS has initiated to be shutdown, this will vary depending on the type of applications the VM(s) may be running, tweak this value as needed.

    Note, the shutdownVM() is a non-blocking function, if you set the delay to be too short the host could be powered down before the VM(s) have completed powering off (default 15 secs)


    my $sec_to_sleep = 15;Next, you'll want to install the apcupsd client on your VMware VIMA host with the following command:

    sudo rpm -ivh apcupsd-3.14.5-1.el5.x86_64.rpm
    

     

     

    You'll then need to configure your apcupsd.conf and you use the following document

     

    Next, instead of setting up a SSH paired keys and inserting the ssh command in /etc/apcupsd/apccontrol, you'll be appending the following:

     

     

     

    doshutdown)
         echo "UPS ${2} initiated Shutdown Sequence" | ${WALL}
         /home/vi-admin/upsVIShutdown.pl 
         ${SHUTDOWN} -h now "apcupsd UPS ${2} initiated shutdown"
    ;;
    

     

     

    I don't think the 3rd line is really ncessary since the VM will be on the same host that'll be shutting down.

     

     

     

    Sample output

    1. We'll now manually initate the upsVIShutdown.pl script as if a power interuption was detected by apcupsd client.

     

    [vi-admin@vima-ups-dev ~]$ ./upsVIShutdown.pl
    

     

     

    2. As you can see from above...nothing too fancy, though the process has been kicked off and both host(s) and their VM(s) have been powered down in the order of our configuration file. To verify the output, we will power on our hosts, log into VIMA-UPS-DEV (the VM that initiated the shutdown process) and inspect the logs containing the shutdown process at /tmp/upsShutdown.log.

     

     

     

    [vi-admin@vima-ups-dev ~]$ cat /tmp/upsShutdown.log
    04-25-2008 11:17:52 -- Found ESX/ESXi host: superion.primp-industries.com!
            04-25-2008 11:17:52 -- Begin shutdown process ...
            04-25-2008 11:18:07 -- VM: slingshot shutdown succssfully.
            04-25-2008 11:18:22 -- VM: air_raid shutdown succssfully.
            04-25-2008 11:18:37 -- VM: silverbolt shutdown succssfully.
            04-25-2008 11:18:37 -- VM: skydive is already suspended.
            04-25-2008 11:18:37 -- VM: fireflight is already off.
            04-25-2008 11:18:44 -- VM: SQL-PROD-01 hard poweroff succssful (No VMware Tools, bad admin).
            04-25-2008 11:18:44 -- VM: Quentin is already off.
            04-25-2008 11:18:44 -- HOST: superion.primp-industries.com is shutting down!
    
    04-25-2008 11:18:49 -- Found ESX/ESXi host: devastator.primp-industries.com!
            04-25-2008 11:18:49 -- Begin shutdown process ...
            04-25-2008 11:18:49 -- VM: scavenger is already suspended.
            04-25-2008 11:18:49 -- VM: bonecrusher is already off.
            04-25-2008 11:18:49 -- VM: master_mix is already suspended.
            04-25-2008 11:18:56 -- VM: scrapper hard poweroff succssful (No VMware Tools, bad admin).
            04-25-2008 11:19:04 -- VM: long_haul hard poweroff succssful (No VMware Tools, bad admin).
            04-25-2008 11:19:11 -- VM: hook hard poweroff succssful (No VMware Tools, bad admin).
            04-25-2008 11:19:11 -- VM: quickMigrate-X is already off.
            04-25-2008 11:19:12 -- Shutting down final host: devastator.primp-industries.com and UPS VM: VIMA-UPS-DEV!
    ==============================================================================================