Dashboards and alerts that work at 3 a.m.

The only test that matters

An alert is worth sending if a competent engineer, woken at 3 a.m., can act on it. Everything else is a tax you pay in attrition, and the interest compounds: once a rotation learns that most pages are noise, it starts treating the real ones as noise too.

Four questions, and a page needs a yes to all of them:

1. Is a user affected, or about to be? If not, it is a ticket at best.
2. Is it urgent? If it can wait until 9 a.m., it must wait until 9 a.m.
3. Is it actionable? If the response is 'watch it', there is nothing to page for.
4. Is it novel? If this fires every week and the fix is always the same, automate the fix instead of paging a person to perform it.

Symptoms page, causes inform

The most durable rule in alerting: page on what the user experiences, not on what you suspect will cause it. Cause-based alerts multiply without limit, because there are unbounded ways for a system to break and only a handful of ways for a user to notice.

The two legitimate exceptions are imminent hard failures and total signal loss. Disk exhaustion causes unrecoverable damage, so predicting it is worth a page. And a service that stopped reporting entirely is indistinguishable from a service that is completely down, so it must page.

The legitimate cause-based alerts

      # Predictive: page on trajectory, not on a threshold
      - alert: DiskWillFillIn4Hours
        expr: |
          predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 4*3600) < 0
          and
          node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes < 0.25
        for: 30m
        labels: { severity: page }
        annotations:
          summary: "{{ $labels.instance }}:{{ $labels.mountpoint }} full in <4h"
          runbook_url: https://runbooks.internal/infra/disk-full

      # Absence of signal is itself a signal
      - alert: TargetDown
        expr: up{job=~"checkout-api|payments-api"} == 0
        for: 5m
        labels: { severity: page }
        annotations:
          summary: "{{ $labels.job }} on {{ $labels.instance }} not scrapeable"

      # The metric stopped existing entirely - absent() catches what
      # a threshold rule cannot, because a missing series never fires
      - alert: NoTrafficReported
        expr: absent(http_requests_total{job="checkout-api"})
        for: 10m
        labels: { severity: page }
        annotations:
          summary: "checkout-api is reporting no request metrics at all"

Everything an alert must carry

The alert body is a handover document written in advance. Assume the recipient has never seen this service, because at 3 a.m. that is functionally true.

A complete alert

      - alert: CheckoutErrorBudgetBurnFast
        expr: |
          sli:checkout_bad:ratio1h > (14.4 * 0.001)
          and
          sli:checkout_bad:ratio5m > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
          team: checkout                  # routing
          service: checkout-api           # routing + grouping
          slo: checkout-availability      # grouping
          environment: production
        annotations:
          summary: "Checkout burning error budget at 14.4x"
          description: >-
            {{ $value | humanizePercentage }} of checkout requests are failing.
            At this rate the 28-day error budget is exhausted in under 2 days.
            Started roughly {{ $labels.alertname }} 2 minutes ago.
          impact: >-
            Customers cannot complete purchases. Revenue impact is immediate
            and proportional to the failure rate.
          runbook_url:   https://runbooks.internal/checkout/error-budget-burn
          dashboard_url: https://grafana.internal/d/checkout-slo?var-env=production
          logs_url:      https://grafana.internal/explore?left={"datasource":"loki","queries":[{"expr":"{app=\"checkout-api\",level=\"error\"}"}]}

The runbook is not optional

An alert without a runbook is an alert that will be handled badly by someone who was not on the team when it was written. Keep runbooks short and specific.

runbooks/checkout/error-budget-burn.md

# Runbook: Checkout error budget burn

## What this means
The checkout API is failing more than 1.44% of requests over the last hour.

## Immediate impact
Customers cannot complete purchases. Assume revenue loss.

## First three checks (in order)
1. Recent deploy?  https://argocd.internal/applications/checkout-api
   -> If a deploy landed in the last 30 min, roll back FIRST, diagnose after.
      `kubectl -n checkout rollout undo deployment/checkout-api`
2. Dependency healthy?  https://grafana.internal/d/checkout-deps
   -> payments-api and inventory-api RED panels. If one is red, escalate to
      that team and declare a SEV-2.
3. Saturation?  https://grafana.internal/d/checkout-resources
   -> Check DB connection pool wait time and CPU throttling ratio.

## If none of the above
Declare a SEV-2 and page the checkout on-call secondary.
Escalation: #checkout-oncall, then @checkout-lead.

## Known false positives
None. This alert requires 2 minutes of sustained 14.4x burn.

Routing, grouping, and inhibition

Alertmanager's job is to turn a storm of firing rules into a small number of useful notifications. Three mechanisms do the work: grouping combines related alerts into one notification, inhibition suppresses alerts that are consequences of a bigger one, and routing sends each to the right destination.

alertmanager.yml

route:
  # Group alerts that share these labels into a single notification
  group_by: [alertname, cluster, service]
  group_wait: 30s            # wait for siblings before the first send
  group_interval: 5m         # wait before sending updates to a group
  repeat_interval: 4h        # re-notify about a still-firing group
  receiver: default-slack

  routes:
    # Pages go to the paging provider, routed by team
    - matchers: [severity = "page"]
      receiver: pagerduty
      group_wait: 10s        # pages should be fast
      routes:
        - matchers: [team = "checkout"]
          receiver: pagerduty-checkout
        - matchers: [team = "platform"]
          receiver: pagerduty-platform

    # Tickets go to a queue, batched generously
    - matchers: [severity = "ticket"]
      receiver: jira
      group_wait: 5m
      group_interval: 1h
      repeat_interval: 24h

    # Non-production never pages anyone, ever
    - matchers: [environment =~ "dev|staging"]
      receiver: slack-dev
      continue: false

inhibit_rules:
  # If the whole cluster is down, do not also page about every service in it
  - source_matchers: [alertname = "ClusterDown"]
    target_matchers: [severity =~ "page|ticket"]
    equal: [cluster]

  # A firing page suppresses the equivalent ticket for the same service
  - source_matchers: [severity = "page"]
    target_matchers: [severity = "ticket"]
    equal: [alertname, cluster, service]

  # If a dependency is down, suppress its consumers' symptom alerts
  - source_matchers: [alertname = "DatabaseDown"]
    target_matchers: [alertname = "CheckoutErrorBudgetBurnFast"]
    equal: [cluster]

receivers:
  - name: pagerduty-checkout
    pagerduty_configs:
      - service_key: ${PD_CHECKOUT_KEY}
        description: "{{ .CommonAnnotations.summary }}"
        details:
          impact:   "{{ .CommonAnnotations.impact }}"
          runbook:  "{{ .CommonAnnotations.runbook_url }}"
          firing:   "{{ .Alerts.Firing | len }}"

  - name: jira
    webhook_configs:
      - url: http://alert-to-jira:8080/webhook

Dashboards structured for triage

Most dashboards are built by someone exploring a problem and then never edited again. That produces a wall of forty panels that is useless under pressure. Build for the sequence an engineer actually follows, in layers.

• Put the answer at the top left. Eyes land there first. It should be the single number that says whether there is a problem.
• Use consistent time ranges and units across panels, or people will misread two graphs as correlated when they are not.
• Annotate deploys. A vertical line at each deploy answers the first triage question without leaving the dashboard.
• Add a text panel with links to the runbook, the repo, the on-call rotation, and the next dashboard down.
• Enable exemplars on latency panels. One click from a spike to a trace is the highest-leverage feature in the entire stack.
• Delete dashboards nobody opens. Grafana usage insights or the API will tell you which. A stale dashboard is worse than none, because someone will trust it.

Deploy annotations and templating

{
  "annotations": {
    "list": [
      {
        "name": "Deploys",
        "datasource": { "type": "prometheus", "uid": "mimir" },
        "enable": true,
        "expr": "changes(kube_deployment_status_observed_generation{deployment=\"$service\"}[5m]) > 0",
        "iconColor": "rgba(255, 96, 96, 1)",
        "titleFormat": "Deploy: {{deployment}}"
      }
    ]
  },
  "templating": {
    "list": [
      { "name": "env",     "type": "query", "query": "label_values(up, environment)" },
      { "name": "service", "type": "query", "query": "label_values(up{environment=\"$env\"}, job)" }
    ]
  }
}

Keeping the alert set honest

Alert rules rot. Services get renamed, thresholds stop matching reality, and rules accumulate for failure modes that were fixed two years ago. Two habits keep the set trustworthy.

Test your rules

promtool rule unit tests

# rules_test.yml - run with: promtool test rules rules_test.yml
rule_files:
  - slo-burn-alerts.yml

evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      - series: 'sli:checkout_bad:ratio1h'
        values: '0+0x30 0.02+0x40'     # healthy 30m, then 2% bad
      - series: 'sli:checkout_bad:ratio5m'
        values: '0+0x30 0.02+0x40'

    alert_rule_test:
      # Should be quiet while healthy
      - eval_time: 20m
        alertname: CheckoutErrorBudgetBurnFast
        exp_alerts: []

      # Should fire after the burn starts plus the "for" duration
      - eval_time: 35m
        alertname: CheckoutErrorBudgetBurnFast
        exp_alerts:
          - exp_labels:
              severity: page
              slo: checkout-availability
              long_window: 1h
            exp_annotations:
              summary: "Checkout is burning error budget at 14.4x (2% consumed in 1h)"

Review what actually fired

Alert review queries

# Which alerts fired most in the last 30 days?
sort_desc(
  count by (alertname) (
    count_over_time(ALERTS{alertstate="firing"}[30d])
  )
)

# Which alerts have never fired? Candidates for deletion or for
# being quietly broken - both are worth knowing about.
count by (alertname) (
  count_over_time(ALERTS[90d])
) == 0

# Pages per week per team - the on-call health metric
sum by (team) (
  count_over_time(ALERTS{alertstate="firing", severity="page"}[7d])
)

Run this monthly with the on-call rotation in the room. For each of the top five alerts, ask one question: did a human do something useful in response? If the answer is no more than half the time, fix the rule or delete it.

Quick answers