Limit tags when sent to PagerDuty

[ericnipro]

Limit tags when sent to PagerDuty

This question is Assumed Answered.

  

tkarraker Jul 12, 2016 5:12 PM   

When we ship data over to Wavefront, we include a series of tags about the state of the metric being collected. This allows us to aggregate along all sorts of things later on. However, this has caused some information overload when receiving pagerduty alerts. For the sake of this example, let's say we have a metric called database.isrunning, and the metric is a simple 0/1 answer. I set up this alert:

ts("database.isrunning", db_state=active) < 1

The problem here is that when the metrics are sent over, it includes all sorts of unnecessary information for this alert:

[WARN] database-server (database.isrunning) [db_pod=pod01][collector_source=somehost][env=live][db_role=slave][dc=datacenterX][server_type=dba][db_state=active]

Is it possible to trim the alert information to only include information that I deem necessary when creating the alert? Think of it in terms of SQL, this is the failure I want to be sent to PD:

SELECT HOSTNAME,DB_POD,DB_ROLE FROM ALERTS WHERE ISRUNNING < 0 AND DB_STATE=ACTIVE.

I don't need collector_source, env, dc, etc.

Thanks in advance!

[ericnipro]

Hi Troy,

Thanks for reaching out to us with this request. We have heard a similar request from few of our other customers as well . ( See  this thread for reference ) . Our product team is evaluating if we can enable this feature to have customized Pager Duty alert templates.

In meantime, could you try using Webhooks to customize the PD integration with Wavefront Alerts? Here is an article on how to  -Managing Webhooks

Please let me know if this helps.

Thanks,

Salil D