First JMX plugin

[BC_hyperic]

Hi,

I want to write my first JMX plugin. Thanks to Charles and Doug my tests with SQL query plugins work fine. I have a simple JMX complaint program, with a get/set and reset. I have tested the program with JConsole. My goal is to see the MBean get/set and reset in hypericHQ. I have gone over the documentation on
http://support.hyperic.com/confluence/display/DOCSHQ27/JMX+Plugin, which I will refer to as the JMX Remote Monitoring page.

I have the following questions.

1) What version of Java should we use and what version should go into the plugin xml file? I have Java 1.5 on my workstation. The JMX Remote Monitoring page shows version=1.5. But the version installed under hyerpic server 2.7.0\jre\bin is Java 1.4.2. The workstation where the agent is has 1.3.?. I can install any version on the agent workstation.

2) Does the simple program need to register and start an RMI connector, or is this done by the Hyperic agent?

3) Does the MBean need to be in a jar or can it stay as a .class file?

4) The JMX Remote Monitoring page shows the path, what is the path relative to?
<config>
<option name="host"
description="Host name"
default="localhost"/>
<option name="path"
description="Path"
default="/jsp-examples"/>
</config>
jsp-examples is relative to what?

5) To see the metrics with the SQL plugin, I added a “New Platform Service” do I add a “New Platform Service” to see the metrics from the JMX plugin?

Thanks,
Bob

[dougm_hyperic]

On Aug 4, 2006, at 9:31 AM, BobCrothers wrote:

> Hi,
>
> I want to write my first JMX plugin. Thanks to Charles and Doug my
> tests with SQL query plugins work fine. I have a simple JMX
> complaint program, with a get/set and reset. I have tested the
> program with JConsole. My goal is to see the MBean get/set and
> reset in hypericHQ. I have gone over the documentation on
> http://support.hyperic.com/confluence/display/DOCSHQ27/JMX+Plugin,
> which I will refer to as the JMX Remote Monitoring page.
>
> I have the following questions.
>
> 1) What version of Java should we use and what version should go
> into the plugin xml file? I have Java 1.5 on my workstation. The
> JMX Remote Monitoring page shows version=1.5. But the version
> installed under hyerpic server 2.7.0\jre\bin is Java 1.4.2. The
> workstation where the agent is has 1.3.?. I can install any version
> on the agent workstation.
>

The Java version running the HQ Agent can either be 1.4 or 1.5,
provide the server you are talking to is JSR-160 compliant. Which it
would be by default if the server you are talking to is running under
a 1.5 JRE, otherwise it depends on the server's JMX implementation.

> 2) Does the simple program need to register and start an RMI
> connector, or is this done by the Hyperic agent?
>

What server implementation are you using to expose your MBeans?
There a links under the heading "Configuring JMX-enabled Applications
for Remote Connections" you should follow to setup the remote RMI
connector.

For example, if your server is running under a 1.5 JRE, you can
enable the remote connector using the following properties:
java -Dcom.sun.management.jmxremote.port=1099 -
Dcom.sun.management.jmxremote.authenticate=false -
Dcom.sun.management.jmxremote.ssl=false ...other args...
your.server.MainClass


> 3) Does the MBean need to be in a jar or can it stay as a .class file?
>

The MBean implementation and packaging is entirely up to you, it
makes no difference from HQ's perspective using the JMX remote
connector.

> 4) The JMX Remote Monitoring page shows the path, what is the path
> relative to?
> <config>
> <option name="host"
> description="Host name"
> default="localhost"/>
> <option name="path"
> description="Path"
> default="/jsp-examples"/>
> </config>
> jsp-examples is relative to what?
>

These properties are specific to the documentation example. You'll
need different properties to compose the OBJECT_NAME of your MBean
(s). If you could provide the ObjectName(s) of your MBeans, we can
provide an example based on that.

> 5) To see the metrics with the SQL plugin, I added a ?New Platform
> Service? do I add a ?New Platform Service? to see the metrics from
> the JMX plugin?
>

If you want to use the "Sun JVM 1.5" server type, click the "New
Server" link from the Platform view to create it and edit
"Configuration Properties" to enter the jmx.url and user+pass if you
have authentication enabled. The services you define in your plugin
will then be auto-discovered (driven by OBJECT_NAME and <plugin
type="autoinventory"/>) or you can click the "New Service" link from
the "Sun JVM 1.5" server view to create services by hand.

[BC_hyperic]

Thanks for the information.

Doug you wrote,
These properties are specific to the documentation example. You'll
need different properties to compose the OBJECT_NAME of your MBean
(s). If you could provide the ObjectName(s) of your MBeans, we can
provide an example based on that.

-----

The ObjectName is created with
ObjectName irrTestName = new ObjectName("App1:name=irrtest");

The URL to connect to the bean is "service:jmx:rmi:///jndi/rmi://localhost:9995/jmxrmi".

Could you give the config properties that go into the plugin?

Thanks
Bob

[dougm_hyperic]

On Aug 8, 2006, at 11:57 AM, BobCrothers wrote:
>
> The ObjectName is created with
> ObjectName irrTestName = new ObjectName("App1:name=irrtest");
>

Below is an example plugin (irr-plugin.xml) to get you started based
on this. You'll need to change the metric names to your MBean's
attributes. The whitespace of the attribute names will be stripped
out before calling MBeanServer.getAttribute. So let's say your MBean
has the following attributes:

public interface ExampleMBean {
public Long getRegisteredUsers();

public Long getInvalidLogins();
}

Your irr-plugin.xml would define metrics like so:

<metric name="Registered Users" .../>

<metric name="Invalid Logins" .../>

You can also use any name you choose if you specify the metric alias
to be that of the MBean attribute name, like so:

<metric name="Number of IRR Registered Users"
alias="RegisteredUsers"
.../>

<metric name="Number of Failed Logins"
alias="InvalidLogins"
.../>

With the plugin deployed on the agent side, you can start by testing
from the command line in the agent directory:

.\jre\bin\java -jar pdk\lib\hq-product.jar -p irr -t "Sun JVM 1.5
IRR" -Djmx.url=service:jmx:rmi:///jndi/rmi://localhost:9995/jmxrmi

After you verifying the plugin from the command line, deploy the
plugin to the server.
Click the "New Server" link from the platform view in HQ, create one
of type "Sun JVM 1.5".
Then edit Configuration Properties to add the jmx.url and user/pass
if required.
After submitting the configuration, you should see the services
defined by your plugin displayed in HQ.

<plugin>
<!-- attach a new service type to the Sun JVM 1.5 server type -->
<service name="IRR"
server="Sun JVM" version="1.5">

<!-- define ObjectName template used for auto-discovery and
monitoring -->
<property name="OBJECT_NAME"
value="App1:name=*"/>

<!-- format the auto-discovered service names -->
<property name="AUTOINVENTORY_NAME"
value="%platform.name% IRR %name%"/>

<!-- instance attribute values; will be auto-discovered -->
<config>
<option name="name"
description="IRR instance name"
default="irrtest"/>
</config>

<!-- enable auto-discovery -->
<plugin type="autoinventory"/>

<!-- plugin for monitoring -->
<plugin type="measurement"
class="org.hyperic.hq.product.jmx.MxMeasurementPlugin"/>

<!-- monitor template applied to all metrics below -->
<filter name="template"
value="${OBJECT_NAME}:${alias}"/>

<!-- Availability metric for red/green lights -->
<metric name="Availability"
indicator="true"/>

<!-- change these to the attributes you want to collect -->
<metric name="Attribute Name"
indicator="true"/>

<metric name="Another Attribute Name"
indicator="true"/>
</service>
</plugin>

[BC_hyperic]

Problem #1
I get the following error when I test the plugin on the agent side

D:\hyperic-hq-agent-2.7.0>.\jre\bin\java -jar pdk\lib\hq-product.jar -p irr -t "Sun JVM 1.5 IRR" -Djmx.url=service:jmx:r
mi:///jndi/rmi://localhost:9995/jmxrmi
Exception in thread "main" java.lang.NullPointerException
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.hyperic.hq.product.ProductPlugin.loadClass(ProductPlugin.java:91)
at org.hyperic.hq.product.ProductPlugin.getPlugin(ProductPlugin.java:114)
at org.hyperic.hq.product.PluginManager.getPluginExtension(PluginManager.java:217)
at org.hyperic.hq.product.PluginManager.getPlugin(PluginManager.java:253)
at org.hyperic.hq.product.PluginManager.getConfigSchema(PluginManager.java:125)
at org.hyperic.hq.product.util.PluginDumper.getConfig(PluginDumper.java:507)
at org.hyperic.hq.product.util.PluginDumper.getPluginConfig(PluginDumper.java:524)
at org.hyperic.hq.product.util.PluginDumper.fetchMetrics(PluginDumper.java:756)
at org.hyperic.hq.product.util.PluginDumper.invoke(PluginDumper.java:365)
at org.hyperic.hq.product.util.PluginMain.main(PluginMain.java:247)

I have tried setting HQ_JAVA_HOME to the same as JAVA_HOME
I have tried setting HQ_JAVA_HOME to D:\hyperic-hq-agent-2.7.0\jre

I have tried replacing localhost with the IP

Problem #2
I added Sun JVM 1.5 server to HQ, this worked fine. I added my new service, but there are no metrics. I hope solving problem #1 will solve this problem

The plugin, called irr-plugin.xml, is below
<plugin>
<!-- attach a new service type to the Sun JVM 1.5 server type -->
<service name="IRR"
server="Sun JVM" version="1.5">

<!-- define ObjectName template used for auto-discovery and monitoring -->
<property name="OBJECT_NAME"
value="App1:name=*"/>

<!-- format the auto-discovered service names -->
<property name="AUTOINVENTORY_NAME"
value="%platform.name% IRR %name%"/>

<!-- instance attribute values; will be auto-discovered -->
<config>
<option name="name"
description="IRR instance name"
default="irrtest"/>
</config>

<!-- enable auto-discovery -->
<plugin type="autoinventory"/>

<!-- plugin for monitoring -->
<plugin type="measurement"
class="org.hyperic.hq.product.jmx.MxMeasurementPlugin"/>

<!-- monitor template applied to all metrics below -->
<filter name="template"
value="${OBJECT_NAME}:${alias}"/>

<!-- Availability metric for red/green lights -->
<metric name="Availability"
indicator="true"/>

<!-- change these to the attributes you want to collect -->
<metric name="Count One"
indicator="true"/>

<metric name="Count Two"
indicator="true"/>
</service>
</plugin>


Thanks
Bob

[BC_hyperic]

Some more information, I added –Dlog=debug to the script to test the plugin on the agent side. I get the following information.


[DEBUG] ControlPluginManager - Created plugin for service 'Sun JVM 1.5 IRR' from server 'Sun JVM 1.5'
[DEBUG] ControlPluginManager - Using null control plugin from Sun JVM 1.5 for Sun JVM 1.5 IRR
Exception in thread "main" java.lang.NullPointerException
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.hyperic.hq.product.ProductPlugin.loadClass(ProductPlugin.java:91)
at org.hyperic.hq.product.ProductPlugin.getPlugin(ProductPlugin.java:114)
at org.hyperic.hq.product.PluginManager.getPluginExtension(PluginManager.java:217)
at org.hyperic.hq.product.PluginManager.getPlugin(PluginManager.java:253)
at org.hyperic.hq.product.PluginManager.getConfigSchema(PluginManager.java:125)
at org.hyperic.hq.product.util.PluginDumper.getConfig(PluginDumper.java:507)
at org.hyperic.hq.product.util.PluginDumper.getPluginConfig(PluginDumper.java:524)
at org.hyperic.hq.product.util.PluginDumper.fetchMetrics(PluginDumper.java:756)
at org.hyperic.hq.product.util.PluginDumper.invoke(PluginDumper.java:365)
at org.hyperic.hq.product.util.PluginMain.main(PluginMain.java:247)

The java version is
D:\hyperic-hq-agent-2.7.0>java -version
java version "1.5.0_07"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_07-b03)
Java HotSpot(TM) Client VM (build 1.5.0_07-b03, mixed mode, sharing)


“Using null control plugin from Sun JVM 1.5 for Sun JVM 1.5 IRR” looks bad, what do I need to do?

Bob

[dougm_hyperic]

The control NullPointerException was a bug in 2.7.0 which is fixed in
the current 2.7.3 release. You can workaround the bug in 2.7.0 by
adding the following to the plugin anywhere within the <service ...>
tag:

<plugin type="control"
class="org.hyperic.hq.product.jmx.MxControlPlugin"/>

[BC_hyperic]

Thank You!

Adding
<plugin type="control"
class="org.hyperic.hq.product.jmx.MxControlPlugin"/>
fixed my test on the agent side. Works great.

I can't test the HQ server now. We are installing a new database etc. I'll let you know when I get a chance.

Thanks
Bob

[BC_hyperic]

I finally got a chance to try the changes on the server. Now I have a new problem.

Version 2.7.0
Hyperic HQ is on Windows Server 2003
The agent machine is on WindowsServer 2003

I created a new “Sun JVM 1.5” server using the following url service:jmx:rmi:///jndi/rmi://10.22.1.2:9995/jmxrmi
This look fine. The availability is green. The metrics for loaded classes, and thread count look okay.

I can run jconsole on the server and connect to the bean on the agent using the above URL.

I have created a New Service which does not work.
Click on the resource (New Service) which was created. Click on the inventory tab. Click on Edit configuration properties. Click ok. I get the following error

The configuration has not been set for this resource due to : Invalid configuration: Plugin not found: measurement plugin name=Sun JVM 1.5 IRR not found

Server.log contains the following line
2006-08-17 15:57:54,765 ERROR [org.hyperic.hq.ui.action.resource.common.inventory.EditConfigPropertiesAction] invalid config org.hyperic.hq.appdef.shared.InvalidConfigException: Invalid configuration: Plugin not found: measurement plugin name=Sun JVM 1.5 IRR not found
[1]-NESTED-EXCEPTION:
org.hyperic.hq.measurement.monitor.LiveMeasurementException: Plugin not found: measurement plugin name=Sun JVM 1.5 IRR not found

If I click on the resource, then metric data, show all metrics, I see the correct metric headings. The availability is yellow and the collection interval for all metrics is NONE

The plugin on the Server is in
\server-2.7.0\hq-engine\server\default\deploy\hq.ear\hq-plugins

The name of the plugin is irr-plugin.xml

The text of irr-plugin.xml is
<plugin>
<!-- attach a new service type to the Sun JVM 1.5 server type -->
<service name="IRR"
server="Sun JVM" version="1.5">

<!-- define ObjectName template used for auto-discovery and monitoring -->
<property name="OBJECT_NAME"
value="App1:name=*"/>

<!-- format the auto-discovered service names -->
<property name="AUTOINVENTORY_NAME"
value="%platform.name% IRR %name%"/>

<!-- instance attribute values; will be auto-discovered -->
<config>
<option name="name"
description="IRR instance name"
default="irrtest"/>
</config>

<!-- Workaround for bug in 2.7.0, fixed in 2.7.3 -->
<plugin type="control"
class="org.hyperic.hq.product.jmx.MxControlPlugin"/>


<!-- enable auto-discovery -->
<plugin type="autoinventory"/>

<!-- plugin for monitoring -->
<plugin type="measurement"
class="org.hyperic.hq.product.jmx.MxMeasurementPlugin"/>

<!-- monitor template applied to all metrics below -->
<filter name="template"
value="${OBJECT_NAME}:${alias}"/>

<!-- Availability metric for red/green lights -->
<metric name="Availability"
indicator="true"/>

<!-- change these to the attributes you want to collect -->
<metric name="Count One"
indicator="true"/>

<metric name="Count Two"
indicator="true"/>
</service>
</plugin>


The plug in the agent machine is in the following directory
D:\hyperic-hq-agent-2.7.0\pdk\plugins

If I run the following test on the agent machine
\jre\bin\java -jar pdk\lib\hq-product.jar -p irr -t "Sun JVM 1.5 IRR" -Djmx.url=service:jmx:r
mi:///jndi/rmi://10.22.1.2:9995/jmxrmi

The test looks good


Thanks
Bob

[dougm_hyperic]

On Aug 17, 2006, at 4:49 PM, BobCrothers wrote:

> I finally got a chance to try the changes on the server. Now I have
> a new problem.
>
> Version 2.7.0
> Hyperic HQ is on Windows Server 2003
> The agent machine is on WindowsServer 2003
>
> I created a new ?Sun JVM 1.5? server using the following url
> service:jmx:rmi:///jndi/rmi://10.22.1.2:9995/jmxrmi
> This look fine. The availability is green. The metrics for loaded
> classes, and thread count look okay.
>
> I can run jconsole on the server and connect to the bean on the
> agent using the above URL.
>
> I have created a New Service which does not work.
> Click on the resource (New Service) which was created. Click on the
> inventory tab. Click on Edit configuration properties. Click ok. I
> get the following error
>
> The configuration has not been set for this resource due to :
> Invalid configuration: Plugin not found: measurement plugin
> name=Sun JVM 1.5 IRR not found
>

Everything else looks right. Did you restart the agent service after
deploying the plugin? Hot-deployment is currently supported on the
server side only, but the agent requires a restart to pickup new
plugins and plugin changes.

[BC_hyperic]

Success! I deleted the Sun JVM server and the IRR service from resources. I then stopped and started everything. I then added the Sun JVM server and the service.

Previously I think I made a mistake setting up the Sun JVM sever. The Sun server was hosted by the wrong platform. Someone else setup the server and agents. The names do not reflect what machine the agents are running on. I added a Sun JVM server to the wrong host machine. The availability was green, it showed metrics for the JVM, my IRR service was in the drop down list. I assumed everything was okay. The error message, “Invalid configuration: Plugin not found: measurement plugin name=Sun JVM 1.5 IRR not found” was from the wrong machine, saying the plugin does not exist on that machine, which is correct. Does this make sense?

Thanks
Bob

[dougm_hyperic]

On Aug 18, 2006, at 11:13 AM, BobCrothers wrote:

> Success! I deleted the Sun JVM server and the IRR service from
> resources. I then stopped and started everything. I then added the
> Sun JVM server and the service.
>

Great news!

> Previously I think I made a mistake setting up the Sun JVM sever.
> The Sun server was hosted by the wrong platform. Someone else setup
> the server and agents. The names do not reflect what machine the
> agents are running on. I added a Sun JVM server to the wrong host
> machine. The availability was green, it showed metrics for the JVM,
> my IRR service was in the drop down list. I assumed everything was
> okay. The error message, ?Invalid configuration: Plugin not found:
> measurement plugin name=Sun JVM 1.5 IRR not found? was from the
> wrong machine, saying the plugin does not exist on that machine,
> which is correct. Does this make sense?

Yes, makes sense, that explains the problem.