jCollectd - How to integrate with Hyperc HQ

[tnist]

We have a SOA implementation where we have numerous services which expose mbeans which capture statistical and counter based information which we would like to expose to Hyperic HQ for monitoring. We would prefer not to have to write a custom plugin for each service we want to monitor if possible. We are looking at jCollectd as a possible solution. We have been able to have it find all the mbeans which we expose, and deliver the messages to an instance of jcollectd running in receiver / listener mode; this works great.

The problem we have it figuring out how to get the information to Hyperic HQ; is this possible with jCollectd. Do we need to expose / embed an instances of collectd or jcollectd inside of Hyperic and publish messages to this listener or is there a better way of doing this or is this even possible? If we have to embed an instance of collectd / jCollectd is there a good guide to follow?

TIA for the assistance.

Regards,
Todd

[mcmesser]

I can't speak to the jCollectd side of the question, I'll leave that for smarter folks, but you might be interested in approaching the problem from a different direction:

http://support.hyperic.com/display/DOC/Instrumenting+Java+Applications+for+Management

Granted it requires code modification to add the required instrumentation, but makes for a very nice developer-driven model.

[tnist]

Marty,

Thanks for the suggestion, much cleaner and what I thought should be available. However, this appears to be a Spring 3.x feature as the @ManagedMetric is not available in 2.5.x unless one is using the instrumented Spring Jars provide by SpringSource. I'm sure we can obtain them but was trying to avoid to much change at one time if possible. With that said, I thought I would try your suggestion.

I have tried to get a simple POC to work with 3.0.0 of Spring and Hyperic 4.2.0.7-EE. I have annotated the mbean as shown below:

/**
* POC - Server manager implementation.
*/
@Component
@ManagedResource(objectName="com.tsintech.crm.monitoring.jmx:name=ServerManager,type=ServerManager, name=serverManager",
description="Server manager.")
public class ServerManagerImpl implements ServerManager {

final Logger logger = LoggerFactory.getLogger(ServerManagerImpl.class);

private String serverName = "springServer";
private boolean serverRunning = true;
private int minPoolSize = 5;
private int maxPoolSize = 10;
private int currentSize = 0;
private int invocationCount = 0;

/**
* Gets Current pool size.
*/
@ManagedMetric(category="utilization", displayName="DB Connection Pool Size",
description="The size of the DB Connection Pool",
metricType = MetricType.COUNTER, unit="connections")
public int getCurrentPoolSize() {
return currentSize;
}

/**
* Gets Invocation count.
*/
@ManagedMetric(category="utilization", displayName="Invocations per second",
description="The total count of invocations",
metricType = MetricType.COUNTER, unit="invocations")
public int getInvocationCount() {
return invocationCount;
}

...

I start the application as follows:

java -Dspring.managed.application.name=ServerManager -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=6969 -Dcom.sun.management.jremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -jar spring-jmx-1.0.0.jar

The mbean shows up fine via jConsole and VisualVM.

When I run the following from the hyperic-agent-home:

java -jar ./bundles/agent-*/pdk/lib/hq-product.jar -Dlog=DEBUG -m discover -a metrics > disover.txt

I see the following entry in the discover.txt file:

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Server: localhost ServerManager Spring Application []
AIID.....localhost ServerManager Spring Application
config...
product..{ptql=State.Name.eq=java,Args.*.ct=spring.managed.application.name, jmx.username=system, jmx.url=service:jmx:rmi:///jndi/rmi://localhost:6969/jmxrmi, jmx.port=6969}
metric...{}
control..null
DEBUG [main] [PluginDiscoverer] Runtime discover for localhost ServerManager Spring Application, config={ptql=State.Name.eq=java,Args.*.ct=spring.managed.application.name, jmx.username=system, jmx.url=service:jmx:rmi:///jndi/rmi://localhost:6969/jmxrmi, jmx.port=6969, installpath=}
DEBUG [main] [ProductPluginManager] No ProductPlugin found for: Spring Application
DEBUG [main] [SpringStandaloneApplicationDetector] discovered 0 services
DEBUG [main] [SpringStandaloneApplicationDetector] discovered 0 services

Runtime Resource Report...
Platform=Linux, fqdn=theseus.hiw.com (Fedora 13)
config...
product..null
metric...{platform.log_track.level=Warn, platform.log_track.enable=true, platform.config_track.enable=true}
control..null
cprops...{cpuSpeed=8 @ 1734 MHz (8x16), secondaryDNS=10.161.19.10, defaultGateway=10.161.25.1, arch=x86_64, ram=16016 MB, vendor=Fedora, vendorVersion=13, primaryDNS=127.0.0.1, ip=10.161.25.175, version=2.6.33.6-147.fc13.x86_64}
localhost ServerManager Spring Application cprops: {}
[No services discovered]

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

The agent does discover the service and makes it available to Hyperic HQ to be added to the inventory. But when I click on add to inventory I get the following exception:

Unable to import platform : java.rmi.ServerException: RuntimeException; nested exception is: org.hyperic.hq.common.SystemException: javax.ejb.TransactionRolledbackLocalException


So I have several questions:

1. Is there something wrong with the mbean class?
2. The discover.txt file seems to indicate that while it saw the service that 0 services were discovered, or am I reading this wrong:
DEBUG [main] [SpringStandaloneApplicationDetector] discovered 0 services
(See above for complete entry)
3. The agent does make the service available for adding as a resource in Hyper but then throws an exception when I try to add the resource; is there something obvious here that I am missing
4. Is there any way to achieve the same in 2.5.x by adding some attributes to the @ManagedAttribute tag by applying some naming convention?

TIA for any assistance.

Regards,
Todd

[jencompgeek]

Hello,
I'm not sure about the exception you are seeing adding to inventory, but there is currently a few restrictions on @ManagedResource in order for Hyperic to discover it and add it as a Service.

1) The ObjectName domain needs to be spring.application (this is to be made more flexible in a future release)
2) The "type" and "name" key properties must be specified. All MBeans with the same "type" must have the same metrics, attributes, and operations (as though you were defining a <service> type in plugin XML)

You are correct, @ManagedMetric is only available in open source Spring 3.0 or Instrumented Spring 2.5.x. It is possible to bypass the usage of annotations if you are willing to populate all the ModelMBean (and right now it does only work with ModelMBeans, no MXBean support yet) descriptor info yourself. Full info on the ModelMBean discovery is available here: http://support.hyperic.com/display/DOC/Instrumenting+Java+Applications+for+Management

That document links to a detailed description of the metadata read from the ModelMBeans: http://support.hyperic.com/display/DOC/Model+MBean+Naming+and+Descriptor+Data

Regards,
Jennifer

[dougm_hyperic]

@ManagedMetric is the right path for integrating your MBeans without having to write any xml. jcollectd was part of a different project.. we did have plans to integrate with HQ, but the idea there was to use the listener side of jcollectd to pull in data from machines running collectd where there was no JVM/HQ-Agent running.

[tnist]

Marty, Jennifer and Doug,

Thanks for the pointers, they have helped a lot. Still having an exception when trying to get the discovered mbean into HypericHQ, but we have a ticket opened on that now.

I do have one more question and am not sure if it is best posted here or on the Spring Forum, but I will start here.

We have downloaded and installed into our maven repo the spring instrumented files for version 2.5.7.SR01 and 3.0.3.RELEASE. I will need to get this to work with 2.5.6 and the 2.5.7.SR01 instrumented classes, but for now I am stuck on 2.5.7.SR01. When I try to compile the application it complains "can not find symbol: ManagedMetric", along with one for metricType, ... I looked in the org.springframework.context.instrumented-2.5.7.SR01.jar and there is no class org.springframework.jmx.support.MetricType located there. I thought this annotation became available in 2.5.6 instrumented jars? I see it there in the 3.0.3.RELEASE jar, is it somewhere else under 2.5.7? I looked through the jars and could not find it; what am I missing?

Since I could not find the dependency in 2.5.x for the ManagedMetric annotation, I stuck with version 3.0.3 for this test.

Based on your suggestions I modified the code to use the appropriate objectName "spring.aplication:application=ServerManager, name=ServerManagerImpl, type =ServerManager". It shows up fine along with some nicely instrumented information about my spring application context and the like. The agent now discovers it correctly, but as stated above still have an issue with importing the resource into Hyperic HQ.

In addition, I have several classes annotated with @Component and @Service which show up as well, but the only attribute they show is "MonitoringEnabled" with a value of true? I was under the impression with the Spring instrumented jars I would see additional statistics like ExeutionsPerSecond, InvocationCount, MaximumExecutionTime, MinimumExecutionTime, ... clearly there is something I am missing here but I am not sure what it is. The ApplicationContext and BeanFactory show valuable information, but nothing under the Components or Services of any value.

Attached is the simple little POC that we are using, perhaps it may provide some insight as to what I am doing incorrectly. I believe I have followed all the steps outlined at http://static.springsource.com/projects/instrumentation/instrument/html/, but I must be missing something.

Any assistance is greatly appreciated, and again thanks for your help to date.

Regards,
Todd

Message was edited by: tnist

Message was edited by: tnist

[mcmesser · Accepted solution]

Todd,

This topic might be crossing the streams a bit too much, and I know moving this question over to the Spring forums is a manual pain, but I do believe you'll get much better results from that community than this one.

Thanks,
Marty

[tnist]

Marty,

Thanks for the follow-up. I have moved it over the late yesterday morning as I felt that it was more of a Spring Framework related issue now. The folks on this forum have pointed us in the right direction with the ManagedMetric annotation, so I think this is closed on the Hyperic side with the exception of getting the resources imported via Hyperic HQ server without an exception.

Again, thanks to all for your assistance.

Regards,
Todd