How can I programmatically disable an alert

[mbaker3]

We are building a self-service portal for our customers to allow them to do simple management of their hosted environment. For instance they want to be able to restart a WebLogic managed server. However when they do this it will cause Hyperic to raise an alert as the server will become unavailable.

Is there some way that my restart routine can disable the Hyperic alerts for this managed server while the restart takes place? I don't see an API for this but I assume we could modify the database to set the alerts to disabled and then back to enabled.

Thanks . . . Murray

[admin]

Hi Murray,

The concept of maintenance windows is something that we are thinking
about developing for future versions of HQ. Currently, there's no
way to "temporarily disable" an alert programmatically. However, you
can achieve this by using the some additional configurations to the
alert. If you set the alert to fire for an unavailability of at
least the duration of the restart, rather than every time, then you
won't have false alerts firing while you are restarting the server.

You should not modify the database to affect this behavior. Object
data from the database is usually cached and maintained in our
application server, and you have to guarantee that a direct
modification to the database will be read by the HQ application.

Charles

[mbaker3]

Charles,

Thanks. The maintenance window idea would be very useful and would solve part of the problem. That is something we would definitely use.

We are also looking at ad-hoc situations which are a bit trickier. The problem with extending the duration before you alert is, of course, that you delay the alert when a real problem happens. It would be useful to extend the concept of a maintenance window a little further. For example to set the alert differently for different times of day (business hours vs overnight). That way we could have a short time during the day when we can investigate it easily and longer at night when it is less critical and we have to get people out of bed.

Thanks for your help. In the short term we may just have to go with the extended duration.

Cheers . . . Murray

[mbaker3]

Charles,

If we were to use the Hyperic control function to perform the self-service start/stop would that help? That is, if I stop WebLogic via the control function does it, or can it be made to, disable the alerts?

Thanks . . . Murray

[admin]

Hi Murray,

That isn't a built in function of the product. This certainly sounds like an interesting feature. In all honesty, it's probably not a difficult action to add to the product. We have a concept of "Recovery Alert" in the Enterprise product that does the opposite of what you are detailing. You'd actually set up another alert that could watch the control functions being performed and "enable" another alert when that happens. However, I just don't see a way of setting this up so that you can disable an alert temporarily through an automated process.

Charles

[mbaker3]

Charles,

For my purposes the shell would be a perfectly good way to do it. Currently you have "alertdef list", "alertdef add" and "alertdef delete". What I need is "alertdef modify" with the ability to enable/disable the alert. So then when I'm stopping the resource I disable the alert(s) and when I restart it I enable the alert.

On the same subject I have looked at the doco at http://support.hyperic.com/confluence/display/DOCSHQ30/Alertdef+CLI+Command. Is it possible to do the add via a single command or is it only available via the dialogue? If there is a single line command then I could delete the alert and add it again. (Crude but I guess it would work.)

Thanks . . . Murray

[admin]

Hi Murray,

So you'd rather do this manually rather than getting an alert that
tells you if something was restarted?

I suppose adding a shell command to enable/disable alert definition
wouldn't be too difficult. I wouldn't imagine that you'd want to
delete the alert and add it again because of the number of properties
you will have to set.

Charles

[mbaker3]

Charles,

In our case it wouldn't really be manual. Turning the alert on again would be done in our management framework once it determined that WebLogic was back up. That way we can let our customers start/stop their own environments without an alert coming to us.

What we need is just a function to disable the alert and then enable it again, as you can do through HQAdmin.

In the long term product development it should be made possible to do anything through the shell that can be done through HQAdmin. That would allow is to fully automate the monitoring set-up to complement our automation of things like WebLogic domain builds.

Cheers . . . Murray

[admin]

On Fri, 2007-03-16 at 17:18 -0700, Charles Lee wrote:
> Hi Murray,
>
> So you'd rather do this manually rather than getting an alert that
> tells you if something was restarted?
>
> I suppose adding a shell command to enable/disable alert definition
> wouldn't be too difficult. I wouldn't imagine that you'd want to
> delete the alert and add it again because of the number of properties
> you will have to set.
>
> Charles
>

I concur that to be able to enable/disable the alerts through the shell
could be very useful. I assume (but haven't looked into it yet) that I
can check the status (enabled/disabled) of alerts through the shell, as
well as list (in some way) all the alert definitions (if not, that could
be VERY useful as well, to check for consistency or misconfigurations).

Also agree that EVERYTHING that can be done through the gui should be
available in the shell. Sometimes when things happen we just don't have
direct access to a web browser and/or fast connection.

Just suggestions, not complaints!

John

[admin]

Okay, I will look into adding that functionality into the CLI. Although, EVERYTHING that the GUI can do in the CLI may be next to impossible, though :-(.

Charles