Breaking the Mold - Websphere

[bjchip_hyperic]

With some difficulty I have created a set of classes that define a remote admin client and can query the state of selected embedded queues on a running Websphere Application Server. None of this is a plugin, and there's the rub. It runs most excellently anywhere.

If I put it on the agent on the AIX server I am going to have to rebuild the jar in that environment unless I am dead lucky. This causes trouble because the IBM AIX P5 servers that are the target are not so common.

If I do not, then it is a matter of getting access to the WAS security information. This has its own set of associated puzzles but if it is done this way it does not matter what platform the WAS is running on at all. This is a big big advantage.

The question is, what precedent (if any) is there for putting such a resource gathering tool on something OTHER than the target platform. I am imagining putting it on the server in some way but it isn't clear how this fits into the model at all. It is a "proxy-agent" ? I don't know.

I want this to work into Hyperic proper because the tool has a lot to offer... but how to do it correctly is absolutely unclear to me.

respectfully
BJ

[admin]

Sounds like you've got something quite useful.

It will be hard for us to advise you on how to model this unless we
have a bit more information.

How are you currently implementing this if not a plugin?
What is the protocol for communicating with WAS?
We support live-exec + JMX, is that enough?
Which parts of your feature present things to the UI, and which are
for querying the remote product?
Are you collecting metrics from the WAS? Or do you just want to
control it from HQ?

-- Jon



On Apr 23, 2009, at 9:43 PM, Bj wrote:

> With some difficulty I have created a set of classes that define a
> remote admin client and can query the state of selected embedded
> queues on a running Websphere Application Server. None of this is
> a plugin, and there's the rub. It runs most excellently anywhere.
>
> If I put it on the agent on the AIX server I am going to have to
> rebuild the jar in that environment unless I am dead lucky. This
> causes trouble because the IBM AIX P5 servers that are the target
> are not so common.
>
> If I do not, then it is a matter of getting access to the WAS
> security information. This has its own set of associated puzzles
> but if it is done this way it does not matter what platform the WAS
> is running on at all. This is a big big advantage.
>
> The question is, what precedent (if any) is there for putting such a
> resource gathering tool on something OTHER than the target
> platform. I am imagining putting it on the server in some way but
> it isn't clear how this fits into the model at all. It is a
> "proxy-agent" ? I don't know.
>
> I want this to work into Hyperic proper because the tool has a lot
> to offer... but how to do it correctly is absolutely unclear to me.
>
> respectfully
> BJ

[bjchip_hyperic]

It communicates using the WAS Admin-Client on the standard SOAP interface exposed by the WAS.

I have not configured it as a monitor... I intended it as a local plugin for WAS to deliver the queue depth and state of embedded JMS queues. At present it is a set of tested classes that do that and a bit more. It is the underlying utility services not the monitor. I could write monitoring in a heartbeat but I had intended to query from the agent and monitored through Hyperic

It was only after working with it and trying to understand how to integrate it into the plugin form that I recognized that I had built something that probably is more useful from the server.

As a result nothing presents information to the UI (yet). The tests show that the information is captured and available, and there are tested classes that return it, but I have nothing that integrates with the UI.

I am new to Hyperic and I haven't looked at live-exec... I will look at that now. Sounds promising if it runs on the server and allows integration.

I am wanting to collect metrics. The core functionality of the admin-client is not limited to this, but our customers already have perfectly good tools for admin.

The WAS is not particularly competent when it comes to tracking the state of its internal queues but a project that uses them needs to get an alert when the plumbing gets clogged. Integrating this with Hyperic is very desirable as a WSMQ component on the same server is part of the plumbing and other monitoring is required, not just this.

HermesJMS can be made to work with them but it doesn't monitor.

FYI - I am based in NZ.

respectfully
BJ

[bjchip_hyperic]

live-exec does a transient check of current data. ("The data is delivered on-demand and is not stored.") which somewhat defeats the purpose of monitoring. We really need automated queue depth monitoring and alerting.

I could configure a monitor that dumps information that the log-tracker could pick at, but I expected to do better than that.

BJ

[bjchip_hyperic]

This category of Server (AppServers, LDAP servers...) has admin layers of its own. It is another layer and all of them have their own utility interfaces for admin accessible over the net.

If we use an agent plugin to interface with the server, the agent plugin code is going to have issues relating to the platform and resources to do this.

Monitoring the AppServer platform, process and JVM health from the agent is good.

The internal state of an AppServer and its processes and queues is possibly more universally done by an external agent that runs on the HQserver and uses the AppServer's admin interface to dig, but hands the data back to HQserver to be monitored just as though it were on the remote agent. This is a useful extension of the model that follows the underlying structure better.

respectfully
BJ

[bjchip_hyperic]

If it is set up as a foreign-agent however, the connectedness of the Admin-Client has to be monitored. I hadn't thought of that until this instant.

BJ

[bjchip_hyperic]

At this point it looks like a plugin that runs on the agent that runs on the HQ-Server. This would give closest conformance to the model, and would keep the integration simple. All the standard plugin-development tools could be applied (within reason). The plugin thus configured would have the following disadvantages:

1. It would require another port open to the admin server.
2. It would need to be monitored itself for its connectivity to the server.
3. It could need special handling in terms of security configuration.

It would have the following advantages.
1. It would not be affected by differences in the AppServer platform or JVM.
2. Changes to the remote agent and redeployments there are not required to alter function.

The advantage of platform independence is hard to overstate. As this pattern would leave the remote agent and its plugins to do the basic jobs of monitoring the OS and Platform health, pulling logs and all the rest, it serves more as an extension of the pattern than a breaker of it. It allows the remote agent maintenance to remain simple no matter what AppServers are deployed.

respectfully
BJ

[bjchip_hyperic]

Now I want to know how to have an WSadmin client that maintains its connection to the Websphere Application Server and does not have to be restarted every time an update of the queue status is required.

I am willing to restart it if it has gone missing but restarting is going to require reinstating the queues to be monitored. This is a somewhat involved and annoying process. Also needed is a way to display a set of strings (the queue names available for monitoring) back at the UI.


WebsphereAppServer------WSAdminClient-----JMSQueueService

There is a method that gives back a list of queues. The format can be altered to suit.

There are methods to define which queues to monitor and...

There is a method to get just the monitored queues status... or just one monitored queue's status.

Not much on going one-at-a-time here...:-) However, the experience of writing a SIMPLE plugin has made it clear that something that defines more than one queue is going to be a PITA of some magnitude.

However, the concern is the effort of getting a new WSAdmin client and creating a new queues server each and every time the queries are made.

BJ

[bjchip_hyperic]

I am going to have a go at extending the ControlPlugin to incorporate the initialization and maintenance of the AdminClient and my queues utility classes. Seems like the wrong way (just feels awkward) to implement the utility, but not so bad that I won't try it. It gives me a way to isolate the IBM specific stuff and it gives me a way to actually control the two things I want to keep isolated and running.

The measurement plugin has to access this. Seems like a plugin in a plugin sort of structure. How that works is going to be a real treat to find out.

...but that part DOES feel right.

Properties files will be used in the first instance, to set things up. Then move it all to Windows and make it work there. Then put security back on the server and make it work anyway.

respectfully
BJ

[bjchip_hyperic]

Still not there. Architecturally this is extremely difficult to figure out..

If I do figure out how to get it to start a "server" (actually my WASAdminClient and jmsQueues utilities), I am still at a loss as to how to get the various measurement plugins to talk to THAT server as opposed to each starting a new one. There MUST be a way that this works, but the documentation for it doesn't hint at the methods used or isn't where I've been looking.

What happens when I specify a SERVER in the plugin xml? Does that get started ? Does it get started if I "discover" it manually... as is done with the JVM server? I think its time to toss the decaf into the sink. This is hard to understand.

respectfully
BJ

[bjchip_hyperic]

For those who need a picture of where this is going ( I know I did ) 🙂

respectfully
BJ

[bjchip_hyperic]

"....obtaining the adminclient is not as simple as passing the username and password. It also requires the ssl trustStore and keyStore files from a WebSphere installation. The other thing is that you just can’t get these files out of nowhere. They need to be validly generated from a WebSphere installation. Note: The WebSphere installation doesn’t need to be the server where you want to connect."

From Alvin Abad's site. Bolding mine.

http://alvinabad.wordpress.com/2009/02/15/automating-websphere-using-jmx/