Glitchy
Contributor
Contributor

"Find entity by IP" when PCNS gives shutdown command.

I recently installed PCNS 3.1 on vMA 5.5. This in an ESXi version 5.5.

I installed vMA and added my ESXi  with 'vifp addserver server.domain.name' .

with 'vifp listserver'I can see that it is succesfully added.

Then, I installed Powerchute and configured it like this:

-IPV4

-Standalone VMware Host

-information to the ESXi server (IP address, username and password)

-Single UPS configuration.

-login information to the ups

the PCNS succesfully regsitered the UPS' management card.

The problem

------------------

I now go to the UPS' web ui and comamnd it to reboot the UPS (to test my PCNS obviously).

So I go to the event log in the PCNS and it show 'shutting down host IP ADDRESS'.

But when I view my ESXi it says 'Find entity by IP' and it doesn't do anything.

Both the PCNS and ESXi view these events every 30 seconds.

What can I do to make it actually work?

Thanks in advance.

0 Kudos
28 Replies
dgrehan
Enthusiast
Enthusiast

Hi,

PCNS 3.1 no longer requires you to add the ESXi host using  vifp addserver command.

Can you enable debug logging in PCNS 3.1 as follows:

1. sudo service PowerChute stop.

2. Edit /opt/APC/PowerChute/group1/log4j.xml using sudo vi and change:

<root>

        <priority value="error" />

        <appender-ref ref="rollingFile" />

    </root>

to

<root>

        <priority value="debug" />

        <appender-ref ref="rollingFile" />

    </root>

3. Save log4.xml and re-start the daemon - sudo service PowerChute start.

4. Try to shutdown the UPS again.

Attach a copy of /opt/APC/PowerChute/group1/error.log

0 Kudos
dgrehan
Enthusiast
Enthusiast

Can you check if the following errors appear in /opt/APC/PowerChute/group1/error.log:

ERROR Thread-35 com.apcc.m11.components.Shutdowner.ShutdownSequencer - performShutdownForESXiStandaloneConfiguration

com.vmware.vim25.RestrictedVersion

ERROR pool-2-thread-1 com.apcc.m11.components.Shutdowner.CheckIfCriticalEventActive - processCriticalEventExceptClusterMode

com.vmware.vim25.RestrictedVersion

This error - "com.vmware.vim25.RestrictedVersion" usually points to a licensing issue e.g. attempting to shutdown the Free/Unlicensed version of ESXi.

https://www.vmware.com/support/developer/vc-sdk/visdk41pubs/ApiReference/vim.fault.RestrictedVersion...

http://sourceforge.net/p/vijava/discussion/823526/thread/0224ed4c

http://sourceforge.net/p/vijava/discussion/826592/thread/3ee70cd7

http://www.virtuallyghetto.com/2011/06/dreaded-faultrestrictedversionsummary.html

Can you confirm if it is the free version of ESXi 5.5 or a licensed version? If licensed, can you specify the license type?

PowerChute Network Shutdown does not support the free version of ESXi. The following Kbase provides details on how to perform ESXi host shutdown for the free version:

shutdownHostViaSOAPAPICall.pl for ESX(i) licensed and free version

0 Kudos
dgrehan
Enthusiast
Enthusiast

We have discovered an issue in PCNS 3.1 that affects Standalone ESX/ESXi hosts only. NB: This does not occur if you are using the Managed by vCenter Server option.

PowerChute opens an API session to communicate with the ESX/ESXi host that later times out and PowerChute fails to re-establish the session. This prevents PowerChute from issuing a shutdown command to the Host when a critical UPS event occurs. A fix has been identified and an update to address this issue will be published on the APC Web site shortly. v3.0.1 should be used for Standalone ESXi hosts until the update has been released.

0 Kudos
stanj
Enthusiast
Enthusiast

I am running ESXi 5.1 with a vMA along with vCenter Server 5.1.

I have PCNS 3.0.1 installed on the vMA.

I do not recall seeing or selecting an option for Managed by vCenter Server.

The current installation is working as when there is a critical battery event, the VMs do an orderly shutdown.

My goal is to upgrade to vSphere 5.5.

APC support indicted that vSphere / ESXi 5.5 is not supported with PCNS but didn’t say it would not work.

Is it possible to upgrade to 5.5 and use PCNS to shutdown the VMs if the batteries reach the critical threshold?


thanks

0 Kudos
dgrehan
Enthusiast
Enthusiast

Yes it should be possible to upgrade to vSphere 5.5 and use PCNS to shutdown the VMs. You should be able to use PCNS 3.0.1 to achieve this.

PCNS 3.1 is the latest version and provides support for HA Clusters.

0 Kudos
defkev
Contributor
Contributor

As much as i hate resurrecting old topics but as of today, running:

* ESXi 5.5 (licensed)

* vMA 5.5

* PCNS 3.10

the exact same Problem still exists and the announced patch has yet to be released, or at least i am not seeing anything over at apc.com...

The problem just hit me on a setup i did a while ago, and never truly tested (yeah, yeah, shame on me...never ever trust a Whitepaper) with the site having a one hour lasting power-outtake this morning and none of the configured VMs were gracefully shutdown and the host just turning off after sucking the batteries dry.

The vSphere Client is only reporting "Find entity by IP" and "Find entity by DNS" again and again once the shutdown gets called by PCNS but nothing actually happens, no Host shutdown gets initiated thus no VM gets shutdown either.

As a "workaround" i have now setup PCNS to call the actual shutdown bash file /opt/APC/PowerChute/group1/bin/shutdown explicit as a Run command under Shutdown Settings in the PCNS frontend...and i am pretty certain that this shouldn't be required to get it to work.

Cheers!

0 Kudos
stanj
Enthusiast
Enthusiast

I just deployed PCNS 3.1 Appliance (not powered up yet)  and also have a vMA running on ESXi 5.5 U1 build 1881737 with plans to move to ESXi U2.

I was going to shutdown the vMA and bring the PCNS Appliance on-line to handle the automated shutdown.

What problem are you referring to that requires a patch?

thanks

0 Kudos
dgrehan
Enthusiast
Enthusiast

The issue is outlined in the following Kbase FAQ - FA222090

It has since been fixed - This update was included as part of the main product on the Software/Firmware downloads page on February 17th 2014 so any downloads since this date are not exposed to this issue.

0 Kudos
stanj
Enthusiast
Enthusiast

ok,

Looks like I need to then download and apply the patch PCNS3.1.0EN.tar.gz listed in the FAQ?


0 Kudos
dgrehan
Enthusiast
Enthusiast

If you downloaded the Appliance before Feb 17th then yes you'll need to install the patch or download the Appliance again to replace your existing copy.

0 Kudos
dgrehan
Enthusiast
Enthusiast

@defkev: FindByIP/FindByDNSName: Could you attach a copy of error.log, pcnsconfig.ini and EventLog.txt please? These are located in /opt/APC/PowerChute/group1 folder on vMA.

The /opt/APC/PowerChute/group1/bin/shutdown is no longer used in 3.1 for shutting down the ESXi host.

0 Kudos
stanj
Enthusiast
Enthusiast

The one i have unnzippd is dated 2/4/14.

What is in the logs??

0 Kudos
dgrehan
Enthusiast
Enthusiast

The affected version is build 4512 (See About Box in PowerChute UI).

The fixed version is build 4844. The fixed version of the Appliance is available to download from the APC Software/Firmware downloads page.

The request for the logs was in response to the comments posted by user defkev  on Sep 11, 2014 1:12 AM.

0 Kudos
stanj
Enthusiast
Enthusiast

ok,

i see the new appliance is dated 9/10/14 at the APC site

It would be nice if APC would let users know when thee are new patches or potential issues once they are discovered

0 Kudos
dgrehan
Enthusiast
Enthusiast

My apologies,  I should have updated this thread when the fixed version was released. We added entries on the APC Forums and VMware Forums when the issue was discovered.

0 Kudos
stanj
Enthusiast
Enthusiast

no problem,


I am glad you updated the forum because I did receive the email alert to check..

Where in the appliance can I find the build number?

0 Kudos
dgrehan
Enthusiast
Enthusiast

The build number is shown in the PowerChute UI->Help->About.

For the next and future releases we will look at including the Build Number in the Appliance Name on the Downloads page so as to make it easier to distinguish if there are updates required in future.

0 Kudos
defkev
Contributor
Contributor

As i originally wrote i am running PCNS on vMA (as a service, installed using pcns310ESXi.tar downloaded on July 7 2014) and not the PCNS appliance, as i make use of the vMA anyway.

The build number is 4844

If 3.1 doesn't use /opt/APC/PowerChute/group1/bin/shutdown to initiate the host shutdown, what should i be looking for within PCNS instead? What script/binary calls the shutdown? Does it even use the vi-fastpass target any longer?

Find attached snippets of the log files during a NMC initiated UPS restart, which didn't shutdown the host/prior to adding the (now obsolete) shutdown script as a Run command.

Eventlog.txt:

...

09/11/2014      01:23:16        Communication has been established.     .3.5.1.5.6.1

09/11/2014      01:23:16        Communication established with Environmental Monitor.   .3.5.1.7.20

09/11/2014      01:23:18        PowerChute successfully opened UDP port 3052.   .3.4.9.9

09/11/2014      01:23:18        PowerChute Network Shutdown version 3.1.0 monitoring started.   .3.5.1.5.6.10

09/11/2014      01:23:19        PowerChute successfully opened TCP port 3052.   .3.4.9.9

09/11/2014      01:23:19        PowerChute successfully opened TCP port 6547.   .3.4.9.9

09/11/2014      01:43:53        UPS turn off has been initiated.        .3.5.1.4.6

09/11/2014      01:43:53        UPS critical event: <b>UPS turn off initiated</b> occurred on Host: <b>esxi.fully.qualified.domainname</b>.     .3.4.9.9

09/11/2014      01:46:58        UPS critical event: <b>UPS turn off initiated</b> resolved on Host: <b>esxi.fully.qualified.domainname</b>.     .3.4.9.9

errorlog.txt

...

ERROR main com.apcc.m11.components.Shutdowner.Shutdowner - bad interface ID: .1.3.7

ERROR main com.apcc.m11.components.Shutdowner.Shutdowner - bad interface ID: .1.3.8

ERROR pool-1-thread-6 com.apcc.m11.components.Omaha.OmahaComponent - com.apcc.m11.arch.CommunicationException

ERROR pool-1-thread-18 com.apcc.m11.components.AutoUpdate.SoftwareUpdateChecker_HTTPS - http 404

ERROR 1366201742@qtp-1777521044-3 com.apcc.m11.components.AutoUpdate.SoftwareUpdateChecker_HTTPS - http 404

ERROR Thread-31 com.apcc.m11.components.CommandFileRunner.DelayedExecutor - No event formatter

ERROR main com.apcc.m11.components.EventLogger.Logger.TextFileLogger - java.io.FileNotFoundException: EventLog (No such file or directory)

Furthermore the pcnsconfig.ini is still pointing the "shutdownCommand" to "/opt/APC/PowerChute/group1/bin/shutdown"

pcnsconfig.ini

[NetworkManagementCard]

port = 80

protocol = http

enrolWithNMC = true

acceptCerts = false

IP_1 = the.nmc.ipv4.address

mode = single

IP_1_Outlet = 0

[EventLog]

logsize = 1000

logfile = EventLog.txt

[Shutdown]

shutdownCommandDuration = 60

turnOffUps = false

turnOffSOG = false

shutdownDelay = 10

event_ShutdownStarting_enableLogging = true

shutdownCommand = /opt/APC/PowerChute/group1/bin/shutdown

runtimeRemainingThreshold = 0

shutdownCommandFile = /opt/APC/PowerChute/group1/bin/shutdown

shutdownCommandFileDelay = 0

shutdownCommandFileDuration = 0

shutdownCommandFileEnabled = true

...

[AdvancedConfig]

SinglePowerFailed_enableLogging = true

SinglePowerFailed_turnOffUps = false

SinglePowerFailed_turnOffUpsDelay = 0

SinglePowerFailed_enableCommandFile = false

[HostDetails]

ESXiConfigurationMode = Standalone Host.

ESXi_IP = esxi.fully.qualified.domainname

So who or what am i supposed to believe now?

Thanks for looking into this.

Regards

PS: Looks like the Update Check also isn't working right now, the server only returns a 404 querying https://updates.apc.com/pcns/autoupdate.cfm

0 Kudos
dgrehan
Enthusiast
Enthusiast

Hi Defkev,

Starting in version 3.1, PowerChute no longer uses /opt/APC/PowerChute/group1/shutdown script which called shutdownhost.pl. The is a PERL script that used VMWare PERL SDK and vi-fastpass functionality to connect to the ESXi host to perform the shutdown.

In 3.1 PowerChute uses VI-Java which is a wrapper library that interacts with the VMware Web SDK. In order to troubleshoot the issue you are having I would need to have a look at the debug logs from PowerChute. If you'd like to send me a private message I can provide instructions on how to enable debug logging and you can send me the logs directly rather than posting them publicly.

I'm guessing that there may be an issue trying to resolve the ESXi hostname or a permissions issue but I cannot tell for sure without access to the debug log.

0 Kudos