VMware Communities > VMTN > VMware vSphere™ > Automation Tools > vSphere™ Management Assistant (vMA) > Documents

ESX/ESXi APC/APCUPSD Host Shutdown VI Perl Toolkit Script (ghettoShutdown.pl/upsVIShutdown.pl)

VERSION 2 Published

Created on: Feb 16, 2009 12:51 AM by lamw - Last Modified:  Sep 23, 2009 7:40 PM by lamw

Description

Many parties have looked for the possibility of monitoring power continuity and subsequently reacting to power outages with information supplied by a UPS agent installed in a virtual machine running on a free version of ESXi. Currently, one free solution How to configure ESXi to shutdown using an APC SmartUPS created by Joseph Holland facilitates shutdown/suspend procedures for online virtual machines in the event of a power outage given that there is some type of physical UPS system in place to provide backup power to the ESXi server. Joseph’s solution utilizes VMware VIMA and the open source utility apcupsd however, one needs to enable the unsupported SSH console in ESXi and setup paired SSH keys between VIMA and host to utilize it. Enabling the unsupported SSH console may not be the preferred method for some end users with respect to their support contracts held with VMware.

The following solution outlined in this document is similar to Joseph’s work although it will not necessitate enabling the unsupported SSH console on the ESXi hosts. This is made possible by utilizing the VI Perl Toolkit to communicate with the ESXi hosts. The only possible drawback however is that next revision of ESXi may disallow write operations and require that the user purchase ESXi license(s) to make full use of the VI API (read/write).

It must be pointed out that this implementation is built on top of initial findings made by Joseph Holland (APC VIOPS documentation) and Andrew Sullivan (VMware VIMA VI Fastpass disection). Please do visit Andrew’s blog at get-admin. Thanks to Duncan Epping for nudging the challenge via Twitter ;)...

Note: Please ensure proper testing has been performed in a staging or development environment prior to implementing this solution in a production environment.

Requirements

ESX/ESXi 3.5+
VMware VIMA
VI Perl Toolkit (included in VIMA)
APC SmartUPS or other "smart" UPS (may require another monitoring utility)
apcupsd-3.14.5-1.el5.x86_64.rpm (VMware VIMA is 64bit RHEL VM)

Usage

[vi-admin@vima-ups-dev ~]$ ./ghettoShutdown.pl
Required command option 'ups_vm' not specified.

Synopsis: ./ghettoShutdown.pl OPTIONS


Command-specific options:
   --sleep
      The amount of time (secs) to wait after a guestOS shutdown (default 15 secs)
   --ups_vm (required)
      The name of VM that is monitoring your UPS

Common VI options:
   --config (variable VI_CONFIG)
      Location of the VI Perl configuration file
   --encoding (variable VI_ENCODING, default 'utf8')
      Encoding: utf8, cp936 (Simplified Chinese), iso-8859-1 (German), shiftjis (Japanese)
   --help
      Display usage information for the script
   --passthroughauth (variable VI_PASSTHROUGHAUTH)
      Attempt to use pass-through authentication
   --passthroughauthpackage (variable VI_PASSTHROUGHAUTHPACKAGE, default 'Negotiate')
      Pass-through authentication negotiation package
   --password (variable VI_PASSWORD)
      Password
   --portnumber (variable VI_PORTNUMBER)
      Port used to connect to server
   --protocol (variable VI_PROTOCOL, default 'https')
      Protocol used to connect to server
   --savesessionfile (variable VI_SAVESESSIONFILE)
      File to save session ID/cookie to utilize
   --server (variable VI_SERVER, default 'localhost')
      VI server to connect to. Required if url is not present
   --servicepath (variable VI_SERVICEPATH, default '/sdk/webService')
      Service path used to connect to server
   --sessionfile (variable VI_SESSIONFILE)
      File containing session ID/cookie to utilize
   --url (variable VI_URL)
      VI SDK URL to connect to. Required if server is not present
   --username (variable VI_USERNAME)
      Username
   --verbose (variable VI_VERBOSE)
      Display additional debugging information
   --version
      Display version information for the script

Overview of the scripts:

ghettoShutdown.pl - This script initiates the shutdown of all VM(s) within an ESX/ESXi host excluding the virtual machine that's monitoring the UPS device and then shutdowns the host. It accepts two commandline parameters: --sleep the duration in seconds to wait after a VM has initiated the shutdown before moving onto the next VM (shutdownVM() is non-blocking function) and --ups_vm the name of the displayName of your VM that is monotiring the UPS device [more details to come later].

upsVIShutdown.pl - This script is a wrapper which will hold the configurations of the order of hosts to shutdown. It may be used inconjunction with other UPS monitoring utility, though with our example, it'll be placed in the apccontrol script to execute upon a power interuption.

Scenario

2 ESXi 3.5u3 (free unlicensed edition):

superion.primp-industries.com
SQL-PROD-01 Powered On
Quentin Powered Off
air_raid Powered On
skydive Suspended
fireflight Powered Off
slingshot Powered On


devastator.primp-industries.com
VIMA-UPS-DEV Powered On
bonecrusher Powered Off
master_mix Suspended
quickMigrate-X Powered Off
scavenger Suspended
hook Powered On
long_haul Powered On


Simulate a power interuption and initiate shutdown of superion and it's VM(s) then devastator and its VM(s) using VMware VIMA: VIMA-UPS-DEV. (note, due to the lack of our UPS being configured, this will be a simulation of the UPS utility initiate the shutdown)

Since the script is written using the VI Perl Toolkit, this potentially could function on a Windows VM installed with the VI Perl Toolkit but you will not able to use VI Fastpass for authentication and credentials will need to be passed to ghettoShutdown.pl via the execution.

Setup

1. Ensure your ESX/ESXi host(s) are being managed by VMware VIMA

2. Download ghettoShutdown.pl and save to /home/vi-admin and make sure it has executable permissions set (chmod +x ghettoShutdown.pl)

3. Download upsVIShutdown.pl and save to /home/vi-admin and make sure it has executable permissions set (chmod +x upsVIShutdown.pl)

4. Edit the following sections in upsVIShutdown.pl:

A) Insert your ESX/ESXi host(s), use the hostname that you used to add to VIMA managment interface, if you're unsure run sudo vifp listservers

Remember to make sure the host that is running your VIMA VM is listed as the last entry.
my @hosts = ("superion.primp-industries.com","devastator.primp-industries.com");B) Modify the logoutput of the shutdown process if you like, default will go to /tmp/upsShutdown.log

my $log_output = "/tmp/upsShutdown.log";C) Insert the displayName of VIMA or VM that is monitoring your UPS device (case sensitive)

Very important step else you could end up shutting down the VM that is executing this script, we wouldn't want that
my $ups_vm_name = "VIMA-UPS-DEV";D) Insert the number of seconds to delay after a guestOS has initiated to be shutdown, this will vary depending on the type of applications the VM(s) may be running, tweak this value as needed.

Note, the shutdownVM() is a non-blocking function, if you set the delay to be too short the host could be powered down before the VM(s) have completed powering off (default 15 secs)
my $sec_to_sleep = 15;Next, you'll want to install the apcupsd client on your VMware VIMA host with the following command:

sudo rpm -ivh apcupsd-3.14.5-1.el5.x86_64.rpm

You'll then need to configure your apcupsd.conf and you use the following document

Next, instead of setting up a SSH paired keys and inserting the ssh command in /etc/apcupsd/apccontrol, you'll be appending the following:

doshutdown)
	echo "UPS ${2} initiated Shutdown Sequence" | ${WALL}
	/home/vi-admin/upsVIShutdown.pl 
	${SHUTDOWN} -h now "apcupsd UPS ${2} initiated shutdown"
;;

I don't think the 3rd line is really ncessary since the VM will be on the same host that'll be shutting down.

Sample output

1. We'll now manually initate the upsVIShutdown.pl script as if a power interuption was detected by apcupsd client.

[vi-admin@vima-ups-dev ~]$ ./upsVIShutdown.pl

2. As you can see from above...nothing too fancy, though the process has been kicked off and both host(s) and their VM(s) have been powered down in the order of our configuration file. To verify the output, we will power on our hosts, log into VIMA-UPS-DEV (the VM that initiated the shutdown process) and inspect the logs containing the shutdown process at /tmp/upsShutdown.log.

[vi-admin@vima-ups-dev ~]$ cat /tmp/upsShutdown.log
04-25-2008 11:17:52 -- Found ESX/ESXi host: superion.primp-industries.com!
        04-25-2008 11:17:52 -- Begin shutdown process ...
        04-25-2008 11:18:07 -- VM: slingshot shutdown succssfully.
        04-25-2008 11:18:22 -- VM: air_raid shutdown succssfully.
        04-25-2008 11:18:37 -- VM: silverbolt shutdown succssfully.
        04-25-2008 11:18:37 -- VM: skydive is already suspended.
        04-25-2008 11:18:37 -- VM: fireflight is already off.
        04-25-2008 11:18:44 -- VM: SQL-PROD-01 hard poweroff succssful (No VMware Tools, bad admin).
        04-25-2008 11:18:44 -- VM: Quentin is already off.
        04-25-2008 11:18:44 -- HOST: superion.primp-industries.com is shutting down!

04-25-2008 11:18:49 -- Found ESX/ESXi host: devastator.primp-industries.com!
        04-25-2008 11:18:49 -- Begin shutdown process ...
        04-25-2008 11:18:49 -- VM: scavenger is already suspended.
        04-25-2008 11:18:49 -- VM: bonecrusher is already off.
        04-25-2008 11:18:49 -- VM: master_mix is already suspended.
        04-25-2008 11:18:56 -- VM: scrapper hard poweroff succssful (No VMware Tools, bad admin).
        04-25-2008 11:19:04 -- VM: long_haul hard poweroff succssful (No VMware Tools, bad admin).
        04-25-2008 11:19:11 -- VM: hook hard poweroff succssful (No VMware Tools, bad admin).
        04-25-2008 11:19:11 -- VM: quickMigrate-X is already off.
        04-25-2008 11:19:12 -- Shutting down final host: devastator.primp-industries.com and UPS VM: VIMA-UPS-DEV!
==============================================================================================
Attachments:
Average User Rating
(0 ratings)




Mar 31, 2009 7:19 AM lamw  says:

Note, VMware ESXi 3.5u4 was just released and the internal API seems to have been fixed and exposed VI API will have only read-only access when using the free licensed version of ESXi. This script will most likely not work on U4, please be advised if you're going to upgrade. I have not had a chance to verify but I believe this will be the case: http://tinyurl.com/cxayy6 U3 will continue to have this hole in the API to provide both read/write access to the VI API.


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

Aug 10, 2009 1:20 PM Sully1900  says:

What's the proper "document" link for setting up the apcupsd.conf file? The current link points to an rpm file. Thanks.

Aug 10, 2009 2:23 PM lamw  says: in response to: Sully1900

It should be: http://viops.vmware.com/home/docs/DOC-1341

I've updated the document, this document primarily covers the setup of vMA and the shutdown scripts. The apcupsd conf is covered in Joesph's VIOPS documentation, it utilizes this script.


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
VMware Code Central - Scripts/Sample code for Developers and Administrators
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

Aug 13, 2009 8:58 AM DaveHill  says:

This looks like a useful script.

Is there a reason why you do each guest individually (issue shutdown, sleep, power off)?

If you have a lot of VMs then this could take a while, especially if you need to extend the timeout because of (say) an Exchange Server.

Can you shutdown more than one machine at a time - I was thinking along the lines of:

foreach (vm)
issue shutdown
done

sleep (some time to allow all machines to shutdown)

foreach (vm)
power off
done

shutdown host

Aug 13, 2009 3:27 PM duz  says:

hi all...i don't know why but it seems to work on vima 1.0 but not on vma 4.0...
In vma 4.0 i have comunications lost problem..
Any help?

thk
Alessandro

Aug 13, 2009 4:13 PM lamw  says: in response to: duz

This was written when ESX(i) 3.5 and VIMA 1.0 was available (also described in the requirements). I've not had the cycles to verify against ESX(i) 4.0 and vMA, it should work but I've not tested and have not advertised this script to support vMA 4.0


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
VMware Code Central - Scripts/Sample code for Developers and Administrators
http://twitter.com/lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

Aug 25, 2009 8:56 AM vipernicus42  says:

Hey,

So I've got a simple test setup going:

  • vmbox1.domain.com ESXi 3.5u3 host which has one guest
    • "vmguest1.domain.com" - a win2k "workstation"
  • vmbox2.domain.com ESXi 3.5u3 host which has one guest
    • vMA-vmbox2.domain.com - a vMA 4.0 virtual appliance (I can't find a copy of 1.0 to download ANYWHERE!)

I've got vmbox1 added in vMA with no issues, I can initialize with FastPass and run commands like "vicfg-nics -l --vihost vmbox1" with no issues

I can even run the script, it finds the host and outputs exactly what you'd expect, ending with the line "HOST: vmbox1.domain.com is shutting down!"

But it doesn't shut down. It doesn't do -anything-.

I'm actually running the ghettoShutdown.pl command directly, not using the wrapper for the UPS, just to simplify things. Here's the command I issued on the console of vMA, logged in as vi-admin, after doing vifpinit:

./ghettoShutdown.pl --server vmbox1.domain.com --username root --password password --ups_vm vMA-vmbox2 --sleep 30

Any ideas why the target ESXi box would do absolutely nothing?

I know your last post says that vMA 4.0 hasn't been tested, but I can't find a download link for 1.0 anywhere, else I'd be using it.

Jon

Aug 25, 2009 9:18 AM lamw  says: in response to: vipernicus42

As mentioned, vMA 4.0 has not been tested and it probably won't work due to the updated paths to some of the vCLI scripts.

VIMA 1.0 is definitely hidden, since vMA 4.0 is what VMware wants everyone to use. Through some digging, I found the download page: http://www.vmware.com/support/developer/vima/vima10/vima10relnotes.html The download link is in the middle of the page :)


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
VMware Code Central - Scripts/Sample code for Developers and Administrators
VMware Developer Comuunity
Twitter: @lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

Aug 25, 2009 10:15 AM vipernicus42  says: in response to: lamw

Awesome, downloading now.

If I could figure out how to award points for "correct" or "helpful" (as per your sig) I most definitely would!

I'll let you guys know how it works out.

Jon

Aug 31, 2009 12:09 PM vipernicus42  says: in response to: vipernicus42

As promised, I have returned with results.

OK, so I can confirm that this works in both vMA 1.0 -AND- vMA 4.0

The difference is in the version of ESXi that you're trying to do the remote shutdown command on.

ESXi 3.5u3 (build 123629) will respond to the shutdown command with no issues
ESXi 3.5u4 (build 153875) will completely ignore the command

From the vMA console, the output of the perl script is identical. The only difference is the result.

I suspect that this is a licensing thing. I'm using the "Free" license for both u3 and u4, and I figure they probably took away the ability to do that in the "Free" version of u4.

So now you folks have a confirmed case of it working with vMA 4.0 + ESXi 3.5u3, and a confirmed case of it not working with 3.5u4 no matter which vMA you're using!

I hope this helps someone out... And I hope we can find a way to get this working on the later versions or I'm going to have to start hunting for "old" hardware when we want to build new ESXi servers!

Jon

Sep 1, 2009 7:24 AM SirToby  says: in response to: vipernicus42

Hello everyone,

I tried the same under esxi 4.0 single server (i.e. the free version) and

  • followed the how-to here ("http://communities.vmware.com/docs/DOC-9308") and ended with a status:COMMLOST-issue too. (same as "http://vmware-forum.de/viewtopic.php?p=72807" in german)
  • faced the same results when starting the upsVIShutdown.pl directly (it creates the logfile ending "HOST: SERVERNAME is shutting down!" but not shutting down anything, neither the VMs nor the host

IMHO this script doesn't work with ESXi 4.0 single server version.
I'll check with ESXi 4.0 non-free-version and post the results later this week.

Anyhow: thanks to lamw for share!

Michael

Sep 1, 2009 7:33 AM duz  says: in response to: vipernicus42

scripts work, the problem is power chute signal from UPS.
I tried to test my apc 8000 and it doesn't work.
It is a communication problem from vmware-guest and UPS.
I check firewall but it seems ok.

With vime-1.0, all ok.

thk
ciao
Alessandro

Sep 2, 2009 8:19 AM SirToby  says: in response to: SirToby

Hi,
just to produce no cliffhanger:
I decided to swap from ESXi 4.0 to ESX 4.0, because I some other reasons to do so.
PowerChute Network Shutdown 2.2.1 works fine with ESX 4.0.

Michael

Sep 10, 2009 2:45 PM Newftronics  says: in response to: SirToby

Hello

I have followed all of the instruction to the letter but I am not having any luck. From what I have been reading ESXi 3.5.0, 123629 will respond to the two scripts upsVIShutdown.pl and ghettoShutdown.pl. Unfortunatly when I run the command ./upsVIShutdown.pl nothing happens. I do not get anything logged into the log either /tmp/upsShutdown.log.

The first doc that I found was a file called ESXi and APC UPS VIOPS (with lamw scripts).pdf which is different than the second file I found called DOC-9531.pdf which states under the Setup Section " 1. Ensure your ESX/ESXi host(s) are being managed by VMware VIMA " which maybe where my problem lyes. Excuse my ignorance here but I am unsure what this means. I am using the free version of ESXi. Is this simply something that I have to purchase and setup on my ESXi Server?

Thx

Jason

Sep 10, 2009 7:44 PM lamw  says: in response to: Newftronics

Is your ESXi hosts being managed by VIMA 1.0? If not, then this will not work and make sure its VIMA 1.0 and not vMA 4.0

Regarding documentation, the setup of your environment should following this document, anything related to the UPS utilities, take a look at Joesph's comprehensive doc at: http://viops.vmware.com/home/docs/DOC-1341


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
VMware Code Central - Scripts/Sample code for Developers and Administrators
VMware Developer Comuunity
Twitter: @lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

Sep 11, 2009 8:05 AM Newftronics  says: in response to: lamw

Thank you William. I found my error and as it turns out the command sudo vifp addserver <servername> is all I was missing. However as stated above by vipernicus42 ESXi 3.5u3 (build 123629) will respond to the shutdown command with no issues and ESXi 3.5u4 (build 153875) will completely ignore the command and I ave two ESXi Servers with build 153875 and one with build 123629 I only have partial success. That will teach me for loosing my original downloaded ESXi 3.5u3 (build 123629)CD and downloading a new ESXi 3.5u4 (build 153875)CD. :)

Other than backing up my VMs on the ESXi 3.5u4 (build 153875)ESXi Servers and redoing the ESXi 3.5u4 (build 153875)Servers to the older Build of 123629 is there a way to make this work on the ESXi 3.5u4 (build 153875) Servers?

Thx Again
Jason

Sep 11, 2009 8:50 AM lamw  says: in response to: Newftronics

Take a look at my first comment on this document, u4 is not supported.


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
VMware Code Central - Scripts/Sample code for Developers and Administrators
VMware Developer Comuunity
Twitter: @lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".

Oct 22, 2009 8:44 AM tlyczko  says: in response to: lamw

Hello, has anyone had time/opportunity to test this with ESX 4 or 4i and vMA 4.0??

Or should one set up VIMA 1.0 to use this with ESX 4/4i??

Thank you, Tom

Oct 22, 2009 6:43 PM lamw  says: in response to: tlyczko

It 'should' work on ESX(i) 4.0 and you'll need to use vMA 4.0 as VIMA 1.0 does not support ESX(i) 4.0

The code does not use any special API calls that isn't available in the vSphere SDK for Perl, though you will need to have a properly licensed ESXi 4.0, you will not be able to use the free licensed version due to the API lock down from ESXi 3.5u4+ and going forward which includes 4.0

I've not had the time to test but let us know if it does work.


=========================================================================
William Lam
VMware vExpert 2009
VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/
vGhetto Script Repository
VMware Code Central - Scripts/Sample code for Developers and Administrators
VMware Developer Comuunity
Twitter: @lamw

http://engineering.ucsb.edu/~duonglt/vmware/vexpert_silver_icon.jpg

If you find this information useful, please award points for "correct" or "helpful".