Positive alerts in Prometheus

At work, we use Prometheus to monitor our services. Every few seconds, Prometheus scrapes a /metrics endpoint and records the data in a time series database. The metrics are then used to trigger alerts when services aren’t behaving correctly.

Recently we had a flapping alert where an overloaded service was intermittently failing. We wanted to silence/adjust the alert until we could fix the service, to avoid alert fatigue, but it would be easy to forget to revert the adjustment once the service was fixed.

We solved this using a “positive alert” that would tell us when the service was consistently passing the original alert – that is, when it was no longer flapping.

Let’s walk through an example. One of our alerts checks that Prometheus can successfully scrape our services, which you can check with the automatically generated up time series. A successful scrape gets a 1, a failed scrape gets a 0.

Here’s an alert that warns if more than 5% of scrapes fail within a half hour window:

groups:
  - name: services
    rules:
    - alert: PrometheusTargetIntermittentlyDown
      expr: avg_over_time(up[30m]) < 0.95
      for: 1h
      labels:
        severity: warning
      annotations:
        summary: Prometheus is struggling to scrape metrics from this service.

One of our services was struggling to serve the /metrics endpoint that Prometheus scrapes. When the service had a lot of work in its queue, the worker thread would starve the web server and the metrics request would fail. The scrape rate would dip below 95%, the alert would fire, then the service would empty its queue and the rate would recover, the alert would clear, and so we went round and round. Dozens of notifications which were nothing but noise.

We use Alertmanager Silences when we want to pause an alert for a short period, but it wasn’t clear if or when we’d be able to fix this service. If the silence expired before the problem was fixed, the noise would return. If the problem was fixed before the silence expired, we could unknowingly regress the fix.

Instead, we modified the alert to exclude the overloaded service, then added a “positive alert” to tell us if the service had more than 95% uptime for a fortnight:

groups:
  - name: services
    rules:
    - alert: PrometheusTargetIntermittentlyDown
      expr: avg_over_time(up{instance!="overloaded_service"}[30m]) < 0.95
      for: 1h
      labels:
        severity: warning
      annotations:
        summary: Prometheus is struggling to scrape metrics from this service.
    
    - alert: PrometheusTargetUpAvg
      expr: avg_over_time(up{instance="overloaded_service"}[30m]) >= 0.95
      for: 14d
      labels:
        severity: info
      annotations:
        summary: This service is no longer overloaded; remove the exception.

This worked exactly as intended. A few weeks ago, one of my colleagues made changes to improve the overloaded service, and Prometheus was able to crawl it consistently. The alert fired at the weekend as it ticked over a fortnight of continuous uptime, and we removed the exception.