APC Network Shutdown and ESX

[J-D]

Hi,

I am trying to install and configure APC's Network Shutdown for VMware version 2.2.1.

APC has a specific rpm for VMware and even has VMware mentioned in their installation guide, so I thought this will be a walk in the park...but no.

I installed the rpm using rpm -ivh which created files under /usr/local/bin/PowerChute and created the necessary S and K scripts in several runlevels. There were no esxcfg-firewall modifications so I added:

esxcfg-firewall -o 3052,tcp,out,APC-ups

esxcfg-firewall -o 3052,tcp,in,APC-ups

esxcfg-firewall -o 3052,udp,out,APC-ups

esxcfg-firewall -o 3052,udp,in,APC-ups

and also esxcfg-firewall - o 80,tcp,out,http

but even without the firewall (-allowIncoming, -allowOutgoing) active I still can't get to the page to configure the APC Agent.

I tried launching the app by /etc/rc.d/init.d/PowerChute start but it just gives info and that's it.

Okay, probably a configuration issue, so I ran /usr/local/bin/PowerChute/PCNSConfig.sh and I tried following this but I keep getting access denied. I am sure though about username/password/passphrase...

What is your experience with APC's UPS agent on ESX?

All info welcome!

TIA

[pauliew1978]

Hi J-D,

I have not used the vmware verion of apc but I have used the windows version to great success. What I actually did was have a windows server on the same ups as the esx server (and on the same network) and essentially trigger the esx server to shutdown from the windows apc powerchute software using a combination of "plink.exe" and some scripts which were created on the esx server. Its not the most ideal way but it does work. I even extended this as I have two sites on a campus network with two esx servers (one in each site). Using scripts my esx servers know if it is a campus wide power outage by using a flagging system and will gracefully power down, if it is only one site then the esx server shuts down the vm's unregisters them and registers them on the other esx server and starts them up.

hope this helps, I think ups's and virtual infrastrcuture is a big pain! someone should write a neat solution. I would if I could program further than simple nix scripts!

Paul

[J-D]

Hi,

yeah, I am using plink also to shutdown a Check Point SecurePlatform based firewall. This is a stripped/hardened RHLinux without any UPS support.

I use a Windows server who starts this commandfile:

c:\bin\plink.exe 192.168.1.1 -l upsuser -pw upspw "shutdown -h now"

I suppose I could use this on an ESX? Will the shutdown command use the configured "Virtual Machine Startup/Shutdown" sequence?

In any case, this is the fallback strategy...I do hope that APC got native support...I know HP UPS have this and it works.

[pauliew1978]

yes, thats exactly what I do. My windows servers are actually datacore sanmelody iscsi sans. As for the startup shutdown sequence, in my script I loop through whatever the returned output of "vmware-cmd -l" is and it stops in that order. However, there is no reason to not "vmware-cmd <server> stop soft" hard coded if you only have a handful of vm's. I could see it getting a bit problematic in a big environment. I guess its just a question of adding it to the setup list of things to do when a new vm comes online (i.e. adding the command to the ups script on esx). Again for the startup sequence you could hard code it. You may find someone else on here can give you a better answer in terms of the shutdown/sequence and being able to control it from the comandline on esx.

cheers,

paul

[J-D]

Tnx for this info. But I do hope somebody got the native APC software working?

Anyone?

[Han]

Have you searched the forums. I have it working and have posted a 'manual' several times.

Han.

[mittim12]

I have this working on serveral different ESX host without any problems at all. I would go back and verify the admin user, password, and autheniticaiton phrase.

[J-D]

Hi,

I found links and info on the forum but I am not really happy with it. Some point to the vmguru document but surely that can't be the way? I don't intend to create a Linux VM, install the agent in there and copy over the configfiles...

This should work out of the box. Apparently mittim12 got it working. I am pretty sure the username/password/authentication phrase is correct...so weird.

I assume I will not be able to connect to the ESX on port 3052 as long as that part isn't set up...

Tomorrow I'll try again from a Windows server and change the authentication phrase there to see if I am typing the correct one.

[J-D]

okay, this new version is working....apparently the authentication phrase was not what our company is using by default. At this location the auth phrase was APC's default: "admin user phrase"

After that I could startup the APC daemon and connect to it with http://...:3052

APC seems to have done a nice job for ESX's...the only thing they didn't mention in their online pdf is that you should configure the esxcfg-firewall.

What are you guys taking as time for the ESX to shutdown? If you 5 VM's on 1 ESX and the default time between stopping each VM is 120 secs, this means at least 10 minutes...

[Han]

In case of an emergency (power outage) you should be less concerned with shutting down your servers (either physical or virtual) in a certain order and thus start shutting down all servers at once. However if your UPS has enough capacity you can take server dependencies into account and shut down in a certain order. You could start simultaneous shutdown of all servers niet being DC's/ESX hosts than shut down the DC's (physical or virtual) and the ESX hosts as the last ones. If you have servers that run databases and other servers running applications dependant on those databases than shut down the application servers first and than the database servers.

Han.

[mittim12]

At our branch offices we have a runtime of 30 minutes for the UPS during a power loss. I have it setup where a script kicks off and starts shutting down VM's if the UPS is on battery for longer then 10 minutes. The last part of the script shuts down the ESX host

[J-D]

I've just done a test with a plink ssh script that just did "shutdown -h now". I had one VM running. After the ESX booted up again, I checked the Windows VM. In the eventlog I saw that a "legacy API" had instructed a shutdown. That must've been the VMware tools who instructed the VM to power down.

This is how I had configured it in the Virtual Machine Startup/Shutdown configuration of the ESX. After the reboot of the ESX, the VM powered up as expected.

I read on the forum that there were many issues on 2.5 but are those issues still there with 3.0.X? You seem to be using your own script to stop the VM's and then the ESX...is it because you don't trust the VMware sequence/config?

After you VMotion a lot, it's true these settings become a mess and an own script might be more interesting.

[mittim12 · Accepted solution]

I have not yet done the necassary testing on the Startup/Shutdown feature of ESX to know if I can trust it or not. It seems that since you are forced to configure it per ESX Host that the settings would not carry over as DRS moves the VM's around but again I have not done the testing to confirm that. I guess I need to put that on the list.

Here is a link to the script that I am using

[J-D]

That script looks awesome! Thanks a lot!

I'll try to use it the next time. I am wondering if I'll increase the shutdown time though. 5 minutes might not be enough to shutdown my Virtual Exchange 2007. Do you use 5 min? (respond and I'll assign you 10 points )

Do you know if the entire VMware shutdown sequence would halt/hang if stopping 1 VM hangs?

[mittim12]

I do use the five minute timer and I have yet to experience any problems with VM's shutting down. We have a Exchange 2007 on a VM in a non production environment and when shutting it down with the vmware-cmd XXXXXXXXXX stop soft command it takes two to three minutes.

I have never had this happen so I can't be 100% positive but I think it would skip the VM and move on to the next.

[Tooms]

That script looks awesome! Thanks a lot!

I'll try to use it the next time. I am wondering if I'll increase the shutdown time though. 5 minutes might not be enough to shutdown my Virtual Exchange 2007. Do you use 5 min? (respond and I'll assign you 10 points )

Do you know if the entire VMware shutdown sequence would halt/hang if stopping 1 VM hangs?

My script is design to not hang if some vm's are hanging

The script will first tell all the vm's to shutdown nicely and then wait for max 300 sec, and if there is any vm's on after the time run out then the script will just turn them off the hard way

[J-D]

Hi Tooms,

thanks for writing that script!

I am sorry, I wasn't clear in my question. I understand the script will hard power down the VM's after 5 minutes. So your script is dealing with hanging VM's but I was wondering if VMware's default implementation can do that too. I mean in the GUI the default delay is 120 seconds before it is instructing another VM to shutdown. I was wondering if one VM wasn't stopping, will the others be shutdown?

I found the answer in the help. I guess they will. I am a bit afraid for the hard shutdown after 5 minutes, I'll probably increase that as some VM's can take a while to shutdown.

Copy-paste help:

Enter a Default Shutdown Delay value if you want to delay shutdown for each virtual machine by a certain amount of time.

Note: This shutdown delay only applies if the virtual machine has not already shut down before the delay period elapses. If the virtual machine shuts down before that delay time is reached, the next virtual machine starts shutting down.

[vhb-technician]

Note: This shutdown delay only applies if the virtual machine has not already shut down before the delay period elapses. If the virtual machine shuts down before that delay time is reached, the next virtual machine starts shutting down.

But does this really imply that if a vm hangs at the shutdown, that the ESX will continue after the period with the next vm ??

It would be good to know, if this is working in the default implementation of VMWare.