How to: set up Alert Reminders

[corey_r]

In Wavefront, the default behavior for alerts is to send an email when an alert fires and when it resolves, and only send updates in-between when something changes - when new sources/labels are affected or when it's partially resolved.

However, for some non-critical alerts that don't require immediate attention but that still need to be solved in a relatively short timeframe, you may want to choose to receive reminder emails if these alerts are not resolved or acknowledged (snoozed) after a certain amount of time. Even though this functionality is not supported directly in the product, it's very easy to do using the below script (also in the attachment) that uses Wavefront Public API to interact with the system - just change the settings and schedule it to run on a regular basis (every 5 minutes is a good starting point).

The script configuration is very simple and only needs a few settings modified: Wavefront API URL (api_url), API access token (token), and settings for your SMTP server for sending reminder emails (email_from, smtp_server).

By default, it sends reminders every 30 minutes for all "SEVERE" alerts as long as they are firing, and it can be also configured to send reminders only for alerts that have specific tags (shared_tags and user_tags settings).

Script code:

#!/usr/bin/env python

import json

import urllib2

import smtplib

import time

import os.path

from email.mime.text import MIMEText

api_url = "https://[domain].wavefront.com/api"
token = "[token]"
email_from = "reminder@company.domain"
smtp_server = "smtp.company.domain"
reminder_interval = 30 * 60  # 30 minutes

# only send reminders for these severity levels
severity = ["SEVERE"]  # ["WARN", "SEVERE"]

# optional: only send reminders for alerts that have any of these tags. leave empty to include all alerts
shared_tags = []  # ["mission-critical-alert-tag-1", "mission-critical-alert-tag-2"]
user_tags = []  # ["user-alert-tag-1", "user-alert-tag-2", "user-alert-tag-3"]

if os.path.isfile("alert_reminder.json"😞

   with open("alert_reminder.json", "r") as infile:

      alerts = json.loads(infile.read())

else:

   alerts = {}

req = urllib2.Request("{}/alert/?view=active".format(api_url))

req.add_header('X-AUTH-TOKEN', token)

response = json.loads(urllib2.urlopen(req).read())

for alert_id in alerts.keys():

   if len([alert for alert in response if str(alert["created"]) == str(alert_id)]) == 0:

      print "Alert {} is no longer firing".format(alerts[alert_id]["name"])

      del alerts[alert_id]

for alert in [item for item in response

              if (len(shared_tags) == 0 or (set(shared_tags) &set(item["customerTagsWithCounts"].keys()))) and
              (len(user_tags) == 0 or (set(user_tags) & set(item["userTagsWithCounts"].keys()))) and
              item["severity"] in severity]:

   alert_id = str(alert["created"])

   name = alert["name"]

   last_fire_ts = alert["event"]["startTime"]

   alert_severity = alert["severity"]

   alert_targets = [item.strip().lower() for item in alert["target"].split(",") if "@" in item]

   if alert_id not in alerts:

      print "New alert firing {}".format(alert_id)

      alerts[alert_id] = {"name": name, "last_fire_ts": last_fire_ts, "last_notify_ts": last_fire_ts}

   else:

      if alerts[alert_id]["last_notify_ts"] + reminder_interval * 1000 < time.time() * 1000:

      print "Sending reminder for alert '{}'".format(name)

      body = "Reminder: Alert '{}' is still firing!".format(name)

      msg = MIMEText(body)

      msg['Subject'] = '[{}] Alert still firing: {}'.format(alert_severity, name)

      msg['From'] = email_from

      msg['To'] = ",".join(alert_targets)

      s = smtplib.SMTP(smtp_server)

      s.sendmail(email_from, alert_targets, msg.as_string())

      s.quit()

      alerts[alert_id]["last_notify_ts"] = int(time.time() * 1000)

with open("alert_reminder.json", "w") as outfile:

  json.dump(alerts, outfile)