How to configure and monitor ActiveMQ 5.3

[dg_w]

ActiveMQ 5.3 is not officially supported by Hyperic HQ 4.2. So is not auto discovered. Is there a way to monitor ActiveMQ 5.3 in Hyperic 4.2 ?

Thank you

[admin]

I see that ActiveMQ 5.2 support was added although the documentation is outdated HHQ-3091. 5.3 might work. You should try manually adding 5.3 via the HQ GUI.

[edan_hyperic]

I added my ActiveMQ 5.3 manually and it still didn't work, but it wasn't all HQ's fault 🙂

First, you have to enable "jmxremote" over a network port so HQ can browse the JMX, at least as far as I can tell. If you're using a stock 5.3 install, you can edit the startup script apache-activemq-5.3.0/bin/activemq and uncomment the line for "SUNJMX"

SUNJMX="-Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

I also added useJMX="true" to the broker config in activemq.xml, but I think that's actually on by default and not needed.

My second problem was that ActiveMQ was using JDK 1.6_016 and the agent was running an embedded 1.5._012. Once I had enabled jmxremote on port 1099, it still failed to autodiscover:

2010-01-06 17:52:47,769 ERROR [Thread-1] [RuntimeAutodiscoverer] Unexpected error running autodiscoverer for plugin: ActiveMQ 5.1: Cannot connect to server
org.hyperic.hq.product.PluginException: Cannot connect to server
[ ... ]
Caused by: java.rmi.UnmarshalException: error unmarshalling return; nested exception is:
java.io.InvalidObjectException: javax.management.MalformedObjectNameException: Unterminated key property part
at sun.rmi.server.UnicastRef.invoke(Unknown Source)

Using the same JDK 1.6 as HQ_JAVA_HOME fixed that though, and it auto-discovered all of my queues, topics, etc. The only problem is that the "openwire localhost Connector" doesn't work, because the statistics are not available.

The problem is: Invalid configuration: Error retrieving value: Attribute not found [org.apache.activemq:Type=Connector,BrokerName=localhost,ConnectorName=openwire:DequeueCount]: javax.management.AttributeNotFoundException: No such attribute: DequeueCount

I did some digging around and I believe the ActiveMQ team disabled connector statistics in 5.2. If you look at the JMX MBeans with JConsole the only attribute available is "StatisticsEnabled" 🙂 If I have time, I might poke around at the plugin XML to try to remove that metric so it looks "clean."

I also modified the latest activemq-plugin.xml from SVN to add "5.3" as a version, and I installed it both on my 4.1.1 server and agent. Mostly I just like being able to pick "ActiveMQ 5.3" from the drop down even though it doesn't really do anything differently... Oh, ok, it does look for activemq-5.3.*.jar instead of 5.2, but, not much else 🙂

Hope this helps. I've attached the slightly modified plugin xml that I'm using to monitor now.

[edan_hyperic]

Not sure where my attachment went, but here it is anyway FWIW.

[zerolagtime]

In HHQ 4.5.2, it auto-discovered ActiveMQ 5.3 and pulls up the basic statistics about it. It fails, however, to pick up any of the services: brokers, connectors, queues, or topics.

After much pain, I figured out how to add JMX support to the container, which is running under Windows 2008R2. Every document I've read says to modify the startup script, but the version that was installed on my server uses the Java Service Wrapper. I had to modify wrapper.conf with additional parameters to the JVM. The arguments are the same (e.g. -Dcom.sun.management.jmxremote...), but it is pretty painful if you don't know where you're going.

Anyways, I have JMX responding and tested on a port of my choosing with authentication all set up and working just like I do for Tomcat. HHQ 4.5.2 talks to Tomcat and auto-discovers its internal workings just fine.

Not so for ActiveMQ. The agent running on the host that discovered ActiveMQ 5.3 in the first place, reports:

Unexpected error running autodiscoverer for plugin: ActiveMQ 5.3: jmx.url==null
...(MxServerDetector.java:4777)

The configuration parameters for ActiveMQ do not allow me to specify a JMX URI, so how is it supposed to do any inspections? I dug through the source code (fisheye.springsource.org/browse) and saw that the plugin seems to be nothing more than the MxServerDetector.

I'm stuck. Can anyone help me decode the auto-discovery process for ActiveMQ so that I can figure out what isn't configured properly? What assumptions are made? Where is the code that actually find the brokers, connectors, etc?

For the record, we installed the stand-alone ActiveMQ 5.3.0 package for Windows 32-bit. It used the Java Service Wrapper (Tanuki Software). Tomcat 6.0 is also on the machine, but there is no sharing of resources between the two containers.

If you've even read this and think you might have some idea, drop a reply that says you read this. Thanks!

[laullon]

Hi, I'm working on this problem.

I think this could be a permissions problem, but I need some logs to confirm it.
Could you setup the agent log level to debug, and restart the agent?
agent.logLevel=DEBUG

Also.
Could you execute the plugin by command line? (as administrator)
%HQ_HOME%\jre\bin\java.exe -ea -jar %HQ_HOME%\bundles\agent-XXXX/pdk/lib/hq-pdk-XXXXX.jar -p activemq -Dplugins.include=activemq -Dlog=debug -m discover -a metric

change %HQ_HOME% and XXXX for the correct values.

[zerolagtime]

Mr. Laullon,
Thanks for giving me a hand. I am unable to attach the log directly as I would have to type it all by hand. I'll include some of the highlights below, but the obvious message seems to be that the plugin is unable to log in to the JMX instance for ActiveMQ. I set up a generic JMX Server in HQ that is able to talk to it as can JConsole. I think that the plugin should be asking for me to provide the following configuration parameters: jmx.url, jmx.username, and jmx.password. I set my site up to use a user in a read-only role with a password. Works just find under Tomcat6.

I am on v4.5.2 of the agent and server (July 8, 2011).
- Auto detector picked up ActiveMQ 5.\d and settled on version 5.3
- DEBUG [Thread-0] [AutoScan] Running AutoServerDetector for: ActiveMQ
5.3
- DEBUG [Thread-0] [MxServerDetector]
State.Name.sw=java,Args.*sw=-Dactivemq.home= matched 1 processes
- DEBUG [Thread-0] [MxServerDetector]
VERSION_FILE=.*activemq-.*-(5.\d*).*.jar matches -> C:\Program Files
(x86)\apache-activemq-5.3.0\bin\win32\..\..\activemq-all-5.3.0.jar
- DEBUG [Thread-0] [MxServerDetector] classpath unchanged using:
C:\Program Files (x86)\apache\activemq-5.3.0\bin\win32\..\..
- DEBUG [Thread-0] [MxServerDetector] Cannot configure local jmx.url:
C:\Program Files (x86)\apache\activemq-5.3.0\bin\win32\..\..
java.io.IOException: Could not map vmid to user name
At org.hyperic.hq.product.jmx.MxUtil.getUrlFromPid(MxUtil.java:397)
At org.hyperic.hq.product.jmx.MxServerDetector.configureLocalMxURL(MxServer Detector.java:126)
At org.hyperic.hq.product.jmx.MxServerDetector.getServerResrouce(MxServerDetector.java:351)
At org.hyperic.hq.product.jmx.MxServerDetector.getServerResrouce(MxServerDetector.java:379)
...

- It goes on identifying the sigar values for things like Availability, Process Virtual Memory Size, ..., Process Open File Descriptors, and Process Threads

- DEBUG [main] [PluginDiscoverer] Runtime discover for <HOST> ActiveMQ 5.3, config=log=debug, jmx.port=6970, plugins.include=activemq, installpath= C:\Program Files (x86)\apache\activemq-5.3.0\bin\win32\..\.., jmx.url=service:jmx:rmi:///jndi/rmi://localhost:6970/jmxrmi, process.query=State.Name.sw=java, Args.*.eq=-Dactivemq.home= C:\Program Files (x86)\apache\activemq-5.3.0\bin\win32\..\..
- DEBUG [main] [ProductPluginManager] No ProductPlugin found for: ActiveMQ 5.3
- ERROR [main] [PluginDiscoverer] ActiveMQ 5.3 Win32-->org.hyperic.hq.product.PluginException: Authentication failed! Username or password is null
org.hyperic.hq.product.PluginException: Authentication failed! Username or password is null at
org.hyperic.hq.product.jmx.MxServerDetector.discoverServices(MxServerDetector.java:477)
org.hyperic.hq.product.ServerDetector.discoverResources(ServerDetector.java)
...
Caused by: java.lang.Security.Exception:
at
com.sun.jmx.remote.security.JMXPluggableAuthenticator.AuthenticationFailure(Unknown)
...
org.hyperic.hq.product.jmx.MxUtil.getMBeanConnector(MxUtil.java:457)

It looks like there was an attempt to compute the URL for JMX, which failed. In any case, it was never going to tell Hyperic the username/password for access, so I still think the plugin should have required more configuration options or else connect irectly in as I seem to be able to do with JConsole.

Thanks for taking a look and happy hunting.

[laullon]

Send me the logs to my email: glaullon@vmware.com

Message was edited by: laullon

[admin]

Is your active MQ process running as the same user as the HQ agent?

That is what this error means "java.io.IOException: Could not map vmid to user name". I believe the jmx.url error message is misleading.

-Jason

[zerolagtime]

No, my ActiveMQ process is running as a domain user and the HQ Agent is running as the local SYSTEM account. The host is Windows 2008 R2.

This is an interesting idea though. I could have sworn that I had ActiveMQ running as local SYSTEM for a while and that there was no difference. It still failed to discover the broker/queue. I might try creating a local, non-SYSTEM account to run ActiveMQ though.

If this is true, I wonder if a Group Policy Object could be interfering. Thus a permission problem as first believed by laullan.

[zerolagtime]

I checked on our development string where they've installed ActiveMQ 5.4 with no changes that the installer or developer remember making. Still running Hyperic 4.5.2 on server and agent under Windows 2008 R2 on both sides. Again, no auto-discovery of brokers, queues, connections, etc.

I don't have ready access to the logs and will have to meet up with our sysadmin on the development string.

[AlpacaSteve]

"DEAR PEOPLE FROM THE FUTURE: Here's what we've figured out so far..."

I know it has been a couple months since anyone has posted on this topic, but I ran into the same issue as zerolagtime and was able to solve it.

I also use the Tanuki wrapper to run ActiveMQ and ran into this issue with agents installed on both AIX and Windows. Turns out, I didn't increment the additional java option index numbers when modifying the Tanuki wrapper.conf file.

Example (wrong version):
# Uncomment to enable remote jmx
wrapper.java.additional.n=-Dcom.sun.management.jmxremote.port=1616
wrapper.java.additional.n=-Dcom.sun.management.jmxremote.authenticate=false
wrapper.java.additional.n=-Dcom.sun.management.jmxremote.ssl=false

Example (correct version)
# Uncomment to enable remote jmx
wrapper.java.additional.10=-Dcom.sun.management.jmxremote.port=1616
wrapper.java.additional.11=-Dcom.sun.management.jmxremote.authenticate=false
wrapper.java.additional.12=-Dcom.sun.management.jmxremote.ssl=false[b][/b]

Hope this helps someone.

[michael123]

I'm seeing the same jmx.url==null error when I try to discover all of our queues, here's the log:

2012-10-07 18:36:32,091 INFO  [Thread-1] [RuntimeAutodiscoverer] Running runtime autodiscovery for ActiveMQ 5.6

2012-10-07 18:36:32,103 DEBUG [Thread-1] [MxServerDetector] [discoverServices] serverConfig=platform.id=10110,platform.log_track.level=Warn,server.log_track.include=,platform.log_track.enable=true,server.log_track.enable=false,server.config_track.enable=false,installpath=/opt/activemq,server.log_track.level=Error,server.log_track.files=var/activemq.log,platform.ip=10.100.71.108,platform.name=prd.queue01.ec2.xtify.com,server.config_track.files=,platform.config_track.enable=true,server.log_track.exclude=,platform.fqdn=prd.queue01.ec2.xtify.com,platform.type=Linux,process.query=State.Name.sw=java,Args.*.eq=-Dactivemq.home=/opt/activemq,

2012-10-07 18:36:32,103 ERROR [Thread-1] [RuntimeAutodiscoverer] Unexpected error running autodiscoverer for plugin: ActiveMQ 5.6: jmx.url==null

org.hyperic.hq.product.PluginException: jmx.url==null

at org.hyperic.hq.product.jmx.MxServerDetector.discoverServices(MxServerDetector.java:477)

at org.hyperic.hq.product.ServerDetector.discoverResources(ServerDetector.java:219)

at org.hyperic.hq.autoinventory.agent.server.RuntimeAutodiscoverer.doRuntimeScan_internal(RuntimeAutodiscoverer.java:272)

at org.hyperic.hq.autoinventory.agent.server.RuntimeAutodiscoverer.doRuntimeScan(RuntimeAutodiscoverer.java:205)

at org.hyperic.hq.autoinventory.ScanManager.mainRunLoop(ScanManager.java:165)

at org.hyperic.hq.autoinventory.ScanManager.access$000(ScanManager.java:41)

at org.hyperic.hq.autoinventory.ScanManager$1.run(ScanManager.java:107)

Caused by: java.net.MalformedURLException: jmx.url==null

at org.hyperic.hq.product.jmx.MxUtil.getMBeanConnector(MxUtil.java:435)

at org.hyperic.hq.product.jmx.MxServerDetector.discoverServices(MxServerDetector.java:474)

... 6 more

HOWEVER, when I run the discover manually, it discovers our queues:

Server: prd.queue01.ec2.xtify.com ActiveMQ 5.6 [/opt/apache-activemq-5.6.0]

AIID...../opt/apache-activemq-5.6.0

config...

product..process.query=State.Name.sw=java,Args.*.eq=-Dactivemq.home=/opt/activemq,jmx.url=ptql:State.Name.sw=java,Args.*.eq=-Dactivemq.home=/opt/activemq,

metric...

control..null

DEBUG [main] [PluginDiscoverer] Runtime discover for prd.queue01.ec2.xtify.com ActiveMQ 5.6, config=plugins.include=activemq,process.query=State.Name.sw=java,Args.*.eq=-Dactivemq.home=/opt/activemq,jmx.url=ptql:State.Name.sw=java,Args.*.eq=-Dactivemq.home=/opt/activemq,log=debug,installpath=/opt/apache-activemq-5.6.0,

DEBUG [main] [ProductPluginManager] No ProductPlugin found for: ActiveMQ 5.6

DEBUG [main] [MxServerDetector] [discoverServices] serverConfig=platform.log_track.level=Warn,platform.type=Linux,plugins.include=activemq,platform.ip=127.0.0.1,platform.name=localhost,platform.log_track.enable=true,process.query=State.Name.sw=java,Args.*.eq=-Dactivemq.home=/opt/activemq,platform.config_track.enable=true,jmx.url=ptql:State.Name.sw=java,Args.*.eq=-Dactivemq.home=/opt/activemq,log=debug,platform.fqdn=prd.queue01.ec2.xtify.com,installpath=/opt/apache-activemq-5.6.0,

DEBUG [main] [MxUtil] ptql:State.Name.sw=java,Args.*.eq=-Dactivemq.home=/opt/activemq resolved to vmid=6950, jmx.url=service:jmx:rmi://127.0.0.1/stub/rO0ABXNyAC5qYXZheC5tYW5hZ2VtZW50LnJlbW90ZS5ybWkuUk1JU2VydmVySW1wbF9TdHViAAAAAAAAAAICAAB4cgAaamF2YS5ybWkuc2VydmVyLlJlbW90ZVN0dWLp/tzJi+FlGgIAAHhyABxqYXZhLnJtaS5zZXJ2ZXIuUmVtb3RlT2JqZWN002G0kQxhMx4DAAB4cHc4AAtVbmljYXN0UmVmMgAADTEwLjEwMC43MS4xMDgAAIa7bK5YhktP1QkkRaVXAAABOjx0bn6AAQB4

DEBUG [main] [MxServerDetector] [discoverMxServices] url=ptql:State.Name.sw=java,Args.*.eq=-Dactivemq.home=/opt/activemq

DEBUG [main] [MxServerDetector] [discoverMxServices] objName=null

DEBUG [main] [MxServerQuery] [findServices] servicePlugins={ActiveMQ 5.6 Topic=null, ActiveMQ 5.6 Broker=null, ActiveMQ 5.6 Queue=null, ActiveMQ 5.6 Connector=null}

DEBUG [main] [MxServerQuery] [findServices] name=org.apache.activemq:Type=Topic,*

DEBUG [main] [MxServerQuery] [findServices] services=(20)[org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Producer.Queue.CHRONICLE, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Consumer.Queue.SCHEDULER.ACTION, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Consumer.Queue.PM.TIMING, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Consumer.Queue.OUT.LOCATION, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Consumer.Queue.STATS.INACTIVE, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Consumer.Queue.PUSH.BULK, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Consumer.Queue.PUSH.IPHONE, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Consumer.Queue.PUSH.ANDROID, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Consumer.Queue.PUSH.GCM, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Consumer.Queue.PUSH.C2DM, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Consumer.Queue.STATS.SDK_ACTION, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Consumer.Queue.PUSH.BLACKBERRY, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Consumer.Queue.PM.LOCATION, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Queue, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Consumer.Queue.STATS.CLIENT, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.NetworkBridge, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Consumer.Queue.PM.STATISTICS, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Consumer.Queue.CHRONICLE, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Connection, org.apache.activemq:BrokerName=prd.queue01,Type=Topic,Destination=ActiveMQ.Advisory.Consumer.Queue.STATS.ACTION]

What am I missing here? 

[michael123]

Ok, I was running hq agent as root. I changed it to run as activemq (the same user activemq is running as), and auto discovery found all of my queues.  All working.