VMware Cloud Community
FrostyatCBM
Enthusiast
Enthusiast

Eaton 9130 + ESXi 5.0 + automated shutdown

Hoping that I can find someone in Melbourne (Australia), or even just anyone in Australia, who is using ESXi 5.0 with Eaton UPS and has successfully configured Eaton's virtual appliance to automatically shutdown VMs and ESXi hosts in the event of a power failure.

I have gotten as far as:

-- downloading and installing the Eaton VA

-- configuring the VA via its web interface

---- node settings (UPS, 2 x ESXi hosts)

---- power source (test Eaton UPS)

---- virtualisation settings

---- shutdown settings

Am using two test ESXi hosts, each running two test VMs. 

I thought I had everything configured like this:

-- ESX1 to go into shutdown after 300 seconds on battery

-- ESX2 to go into shutdown after 60 seconds on battery

---- both were set in the VA to shutdown VMs as well as the ESXi host

Unfortunately, when I tested this, what happened was:

-- nothing after 60 seconds on battery

-- nothing after 300 seconds on battery

-- no VMs were shutdown on either ESX1 or ESX2

-- neither ESX1 nor ESX2 hosts themselves shut down

-- after a while the IPM v1.26 virtual appliance shut itself down (!)

-- a few minutes later I ran out of battery and both ESX1 and ESX2 crashed

I have missed something really obvious and critical.  Hoping someone who has gotten this working with ESXi 5.0 can help my work out what bit I have missed, because I have been looking through all different kinds of documentation and am just not making any headway with it.

Reply
0 Kudos
20 Replies
spravtek
Expert
Expert

Hi, not from Australia, sorry ... But do have some experience with Eaton UPS and shutdown ... Not a lot, but trying to help anyway Smiley Wink

Some questions though, since I don't see it mentioned:

- Are you using, or planning to use vCenter?

- Are you using, or planning to use a Cluster/ HA/ DRS?

Because, if you're just planning on using the ESXi hosts without these features, you can just as well use IPP and vMA to gracefulle shutdown your VM's and hosts ...

Reply
0 Kudos
FrostyatCBM
Enthusiast
Enthusiast

Production is 3 x ESXi 5.0 hosts cluster, but no HA or DRS currently in use.

Yes, we have vCenter for our production systems.

But I am not prepared to deploy stuff into Production unless I have tested and proven it first.

But for my test environment I have 2 x ESXi 5.0 hosts, no HA/DRS, no vCenter, just the hosts themselves.

I have installed the Eaton virtual appliance and have configured the settings.

But since there is no vCenter in the test systems, I just have the IPM VA and the hosts to work with.

I am getting the impression that what I am doing requires vCenter to be installed in order to work.

Reply
0 Kudos
spravtek
Expert
Expert

Yes normally there's a plugin to be installed in vCenter... If you don't use vCenter you'll have to resort to the IPP and the vMA to orderly shut down all VM's and hosts...

You probably know this, but thought I would point to the place where you can download the user guide http://pqsoftware.eaton.com/explore/eng/ipm/default.htm?lang=en

There's more information about the vCenter plugin in there to....

Just for the people interested Smiley Wink

Reply
0 Kudos
FrostyatCBM
Enthusiast
Enthusiast

OK, so I will have to do an install of vCenter into my Test systems, add the hosts, etc etc, add the Eaton vCenter Plugin, then re-test.

I will post again after I have tried that.

Yes, I did have that documentation ... I must say however that some of the documentation is a bit confusing and would be better if it were rewritten with a focus on clarity of explanation ...

Reply
0 Kudos
spravtek
Expert
Expert

That is true, I found it to be confusing to... You actually need to read two documents, one for the appliance and one for the IPM ... The document of the appliance doesn't really contain much interesting info.

In the IPM user guide there's also information on what to do when you're using DRS/HA... It's somewhere at the back of the document... While the info on the vCenter plugin is more to the middle... To make it more easy Smiley Wink

Good luck testing!

Reply
0 Kudos
FrostyatCBM
Enthusiast
Enthusiast

This morning I have:

-- created a new VM in the test environment and installed Windows Server 2008 R2

-- installed and configured vCenter (connected the two ESXi hosts to it)

-- deleted the test ESXi hosts from the Virtualisation area of IPM on the VA

-- added the vCenter server into the Virtualisation area of IPM on the VA

---- checked that it is functioning right with correct credentials and showing as Connected (it is)

-- checked vCenter to ensure that the IPM plugin was installed and working (it is)

-- repeated my test from yesterday by removing the mains power from the UPS

-- the UPS ran for nearly 10 minutes before anything significant was observed

-- the VA shut itself down about 3-4 minutes before the UPS battery was due to run out

-- VMs with VMware Tools installed did shut themselves down gracefully

-- the errors observed in the Log in IPM yesterday were not apparent

But ... as best I can tell ...

-- the ESXi hosts did not seem to shut down gracefully; they seemed to run on and on until the UPS battery died

---- I had the console for the ESXi right in front of me and there was no evidence there of any graceful shutdown

---- I wonder if this is because there were VMs running on them which did not have VMware Tools installed?

-- the vCenter VM also did not seem to shut itself down gracefully

Still, that is some significant progress.

Reply
0 Kudos
FrostyatCBM
Enthusiast
Enthusiast

I noticed one thing just now in some of the printed documentation that is floating around my desk:

"Solution for more than one hypervisor ESX/ESXi

Configuration:

Download and install IPM (we do recommend to install IPM in the same machine as vCenter)"

Now that's not what I have been testing with.  I have been using the virtual appliance version of IPM.  Is that where the problems lie?

I understood based on previous discussions / correspondence that I would need 3 x VA versions of IPM, one for each host+UPS.  But these docs seem to recommend something else:  one install of IPM directly in the vCenter server, which in turn manages the 3 hosts and the 3 UPS units?

Now I am bloody confused.  Would appreciate some suggestions as to the right way to protect:

3 x ESXi hosts (vCenter cluster)

3 x physical Windows servers

with

3 x 9130R UPS units

(each ESXi host connects to its own UPS, but the 3rd UPS also drives the physical Windows servers)

Previously I had installed IPM to one of the physical Windows Servers and IPP to the other two physical servers.

Can I just have one IPM install on the vCenter VM and get rid of IPM on the physical server (replace that with IPP instead and manage everything through vCenter)?

Reply
0 Kudos
FrostyatCBM
Enthusiast
Enthusiast

I am afraid that I have finally, regretfully, given up on the Eaton VMware integration as a viable solution.

The symptoms that I am getting now:

-- with the test UPS fully charged and running on mains

-- I boot my test ESX hosts

-- they start up and also start up my VMs

-- for some unknown reason the IPM VA now shuts itself down (and also shuts down the VMs)

-- the hosts then shut themselves down (fair enough, once no VMs are running)

Unfortunately it is doing this with the UPS running on mains power.  Its not on battery.

I cannot afford to risk putting this into Production, because if it did this to me in Production it would be disastrous.

Nor can I keep investing hours and hours of my time in trying to work out all the issues in the various interlocking systems.

For me to rely on this stuff it needs to be both simple and reliable ... and I am absolutely, completely, utterly frustrated with it because it is neither simple nor reliable.

Its a real shame, because the idea of it is great and if it worked simply and reliably it would be a massive help.

Reply
0 Kudos
spravtek
Expert
Expert

Hi,

Sorry to hear you have a hard time getting it all to work, I'm not with Eaton by the way, just for all things to be clear ... I just configured it for a customer.

In order to gracefully shutdown a VM, and the Eaton application to be able to shutdown the VM, the VMware tools indeed need to be installed.

Now, I can't really tell why your IPM shuts down the VMs and hosts without apparent power failure ... I would need to be physically present I think Smiley Happy ...

Did you have a read through the Appendix 3 and 4 (page 88 and on) of the IPM User manual? There's more information there regarding using IPM and HA/DRS ... With some tips and tricks ...

I'm thinking you are using the Remote Shutdown feature ... Like not using the IPP agent on the VMware Host ?

Anyway ... I can understand if you don't want to invest more time into testing this solution, it can use up a lot of time when it's not working from the first time Smiley Happy

Reply
0 Kudos
FrostyatCBM
Enthusiast
Enthusiast

Yes, I read through all the IPM documents, several times.

I normally pride myself on being able to understand new stuff pretty quickly.

But this software + architecture has defeated me I think...

One question I have ... when you configured your solution ... did you use the IPM virtual appliance? ... or did you install IPM on the vCenter server (which seems to be what is actually recommended in the documentation)?

I am using the virtual appliance, because that is what one of the techs at Eaton here in Australia recommended.

What is not clear to me:  if I were to install IPM to the vCenter server in Production, I would have one (1) IPM, 3 x 9130 UPS, 3 x ESXi hosts + 3 x physical servers ... and its not clear whether that would be an effective design.

What did you do?

Reply
0 Kudos
spravtek
Expert
Expert

I'll keep it short because I'm on the road ... Don't worry I'm not driving Smiley Wink

The last implementation I did I went back to how I used to configure it when the Eaton appliance wasn't available yet...

I installed a vMA with an IPP on each ESXi host and configured it using a script to shut down the hosts and its VMs... These are then managed by an IPM on another physical server. The UPS devices were reachable over the network, so not entirely the same setup as yours...

Another customer is using the appliance, also with a UPS device connected to the network... This is working like it should but took quite the time to configure, so I'm definitely with you on the fact they don't make it easy...

If you're using the IPP you don't really need to use the vCenter integration, it works fine without as well...

In your setup ... Are the physical servers also directly connected to the UPS devices? Maybe your setup might benefit from the IPP approach for the ESXi hosts and use IPM to shutdown the physical hosts remotely ... Just thinking out loud here... That way you can control the physical and virtual environment kind of separate ... And might give more "freedom" using scripts to shut down the virtual environment ...

It does require some time again of course, to get it all set up.

Reply
0 Kudos
FrostyatCBM
Enthusiast
Enthusiast

Wondering if you would mind exchanging some correspondence directly via email.

I can give you a bit more detail about the infrastructure but I am reluctant to post it publicly.

My email address ought to be accessible to you via my profile, but just in case it isn't:

sfrost at cbm dot org dot au

Reply
0 Kudos
FrostyatCBM
Enthusiast
Enthusiast

Ahhhh ... one mystery is explained at least ... it seems that because I had some VMs which did NOT have Vmware Tools installed, they could not be shutdown gracefully ... this led to the host not shutting itself down gracefully either ... and after trying to start everything back up on restoration of power, the IPM virtual appliance was detecting that the host had not shutdown, so IPM immediately went back into trying to do shutdowns ... and so on, in an endless loop.

The fix, for me, was (1) to prevent the IPM VA from auto-starting with the ESXi host; (2) to ensure that all VMs had VMware Tools installed; (3) to start the IPM VA and let it do a complete shutdown of all the VMs; (4) to start everything back up again.

Now the IPM VA is back in a stable state and I can work with it again. I have two further issues:

1. configured my test environment so that ESXi2 + VMs shut down first and then a minute later ESXi1 + VMs should shut down ... if I remove mains power from the UPS to test this, after the nominated time, the VMs and my ESXi2 shut down as expected ... but when it gets to host ESXi1 (2 x test VMs, plus vCenter VM plus the IPM VA) ... the VMs shut down OK, the vCenter shuts down OK, the IPM VA shuts down OK ... but the ESXi host itself does NOT shut down and it keeps on running, and running, and running ... so I either have to wait for the battery to run out or I have to manually shut it down.

Eaton have suggested to me that this could be because I have configured IPM in Settings, System, Shutdown and that this actually tells IPM to shut itself down ... they reckon that if I turn that shutdown settings OFF then the ESXi host will be free to shut down the IPM VA ... I have yet to test this but will try to do so on Monday.

2. when power is restored to the UPS and I then restart the IPM virtual appliance again, it is restarting all the VMs on the ESXi host ... even though the configuration of the host has those VMs configured for Manual startup ... this seems to be a bug ... but I am wondering whether the settings for the ESXi Host in Configuration, Startup/Shutdown can be somehow overridden at the VM level ... it certainly seems that way to me.

Reply
0 Kudos
spravtek
Expert
Expert

Hi Frosty ...

Sorry ... I thought you figured that one out earlier ... As I read one of your posts about VM's needing the VM tools, you mentioning that with VM tools they don't have that problem ... Must have misread that :smileyconfused:

Now imagine having to install that to a production environment and somehow the appliance brings down some VM's because they don't have VM tools installed, small chance of that happening but you never know.

Anyway ...

they reckon that if I turn that shutdown settings OFF then the ESXi host  will be free to shut down the IPM VA ... I have yet to test this but  will try to do so on Monday.

Don't really get this part, I get turning the shutdown settings to OFF ... But then let the ESXi host do the rest? From what will the ESXi host get the command to shut down then?

2. when power is restored to the UPS and I then restart the IPM virtual  appliance again, it is restarting all the VMs on the ESXi host ... even  though the configuration of the host has those VMs configured for Manual  startup ... this seems to be a bug ... but I am wondering whether the  settings for the ESXi Host in Configuration, Startup/Shutdown can be  somehow overridden at the VM level ... it certainly seems that way to  me.

Are you using the "Maintenance Mode" option of the IPM? In other words letting the ESXi hosts go to maintenance? I gues not because then the VM's wouldn't restart normally, I'm just asking because things might be different or not functioning like I think ... Not  knowing your test setup completely... The "Maintenance Mode" would only be interesting if you have multiple UPS devices with different up-times/mains etc ... So you could put 1 ESXi host in maintenance mode when its powersupply fails and it switches to UPS ... Then it will fail-over to other ESXi hosts in the cluster on another mains/UPS device ... If you don't want this feature, disable the "Maintenance Mode" ...

I'll send you a mail if you want to continue this conversation later on. Smiley Wink

Reply
0 Kudos
nicksta
Contributor
Contributor

Hi Spravtek,

How can you configure the Eaton IPM (Configured within vCenter via Plugin) to enter Maintenance Mode without DRS?

Thanks in advance.

Reply
0 Kudos
spravtek
Expert
Expert

Hi nicksta,

How much UPS device do you have to protect your cluster?

I ask because (haven't checked on in this in a while, could've changed) when using only 1 UPS you can't use the maintenance mode setting in IPM.(Eaton recommends it to be turned off).

Are you using HA? Is the Shutdown command working and you need only Maintenance mode?

Can you elaborate on what it is exactly you're trying to do?

Thanks.

Reply
0 Kudos
nicksta
Contributor
Contributor

Thanks for your reply.

There is currently one UPS device. I've configured the IPM within vCenter to enter maintenance mode when the configured trigger is activated.

The IPM does it's part and puts the vCenter into maintenance mode but I then realised the vCenter cluster is not licensed for DRS. Therefore, entering maintenance mode is stuck at 2%.

I need the VMs guests to vMotion to the other ESXi host because one of the guests needs to online at ALL times for the SAN Quorum to stay online.

Customer's Infrastructure Layout:

  • 2 Sites (Primary and DR)
  • Each site has an ESXi Host and a HP P4000 SAN Node
  • ESXi Hosts are part of the same cluster and P4000 SAN Nodes are clustered (There is a HP Failover Manager hosted within the VMware environment using the SAN VMFS Datastore. This isn't best practice but we have to make do.)
  • Primary site runs ALL the VM Guests
  • Current results of DR scenario tests
    • DR site goes down --> All VMware infrastructure is working as per normal
    • Primary site goes down --> Quorum is lost because the HP Failover Manager is hosted on the SAN, Therefore HA doesn't work.
  • Requirement: I need the Eaton IPM to at least vMotion the HP Failover Manager VM guest to the other ESXi host to keep the quorum alive. Even if the other VMs are not migrated across, the Quorum will be active for HA to work.

Let me know if you require further details. Thanks again.

Reply
0 Kudos
spravtek
Expert
Expert

Tricky... Just thinking outside the box and out loud...

Maybe you could try to use the SRM way of moving the VM's ... I haven't really played with it yet, I just know of its existance.

There's a PDF here: http://pqsoftware.eaton.com/install/win32/ipm/UserGuide_VMWareSiteRecoveryManagerIPM.pdf

It mentions the use of a script, I think that will be the best bet, using a script to migrate the VM's to the DR site.

There are scripts included in the EATON SRM module, maybe you can adjust them?

Of course, it might not work if you don't have SRM (which I think you don't have) because it will miss certain modules

You can also just run a script through IPM (triggered through a bat file and using Powershell or something) to vmotion the VM's when the UPS gets triggered...

Something like this (moves just 1 VM):

$vm = Get-VM -Name YOUR_VM
$chost = ($vm | Get-VMHost).Name
if ($chost -eq "ESXi01.company.com")
{
     Move-VM -VM YOUR_VM -Destination ESXi02.company.com
}
else
{
     Move-VM -VM YOUR_VM -Destination ESXi01.company.com
}
Reply
0 Kudos
nicksta
Contributor
Contributor

Correct, we don't have the luxury of SRM.

I'll will test the suggested script method. Will I required to install VMware PowerCLI?

Thanks again.

Reply
0 Kudos