Error budgets and burn-rate alerting

The budget is just the complement of the target

If the SLO says 99.9% of valid requests must be good, then 0.1% of them are permitted to be bad. That 0.1% is the error budget for the window. It is the same number expressed three ways, and being fluent in all three matters because different audiences understand different ones.

Three views of the same budget

SLO target        = 99.9%  over 28 days
Error budget      = 0.1%   of valid events

As time:          28d x 24h x 60m x 0.001        = 40.3 minutes of total failure
As requests:      1 bad request in every 1,000
At 2,000 req/s:   2,000 x 0.001                  = 2 bad requests/sec sustained
                  4.84 billion requests x 0.001  = 4.84 million bad requests allowed

Spending it: the error budget policy

The budget only changes behaviour if something happens when it runs low. That something is the error budget policy, and it should be a short document agreed by engineering and product before the first incident, not negotiated during one.

Two details make the difference between a policy that works and one that is quietly ignored. First, the exception path must exist and be documented, because there will always be a genuine emergency that requires shipping during a freeze; an absolutist policy gets bypassed and then abandoned. Second, someone outside engineering must have signed it. A freeze that product did not agree to in advance becomes a political fight at exactly the moment nobody has energy for one.

Burn rate: the number that makes alerting sane

Burn rate is how fast you are consuming the budget, relative to the rate that would exhaust it exactly at the end of the window. A burn rate of 1 means you will finish the window with precisely zero budget left. A burn rate of 2 means you will run out halfway through.

Definition

burn_rate = observed_error_rate / error_budget_rate

# 99.9% SLO over 28 days -> budget rate = 0.001

  observed 0.1% errors  ->  burn rate  1x   -> budget lasts 28 days
  observed 0.5% errors  ->  burn rate  5x   -> budget lasts 5.6 days
  observed 1.0% errors  ->  burn rate 10x   -> budget lasts 2.8 days
  observed 10%  errors  ->  burn rate 100x  -> budget lasts 6.7 hours
  observed 100% errors  ->  burn rate 1000x -> budget lasts 40 minutes

This reframes alerting completely. Instead of 'page when the error rate exceeds 1%', which is a number someone guessed in 2019 and nobody has revisited, you page when the budget is being consumed fast enough to matter. The threshold is derived from the SLO, so it automatically stays correct when the SLO changes.

Why a single threshold does not work

Alert on a short window only and you page on every thirty-second blip, which is how alert fatigue starts. Alert on a long window only and a total outage takes hours to notify anyone. Neither is acceptable, so you need several rules at different sensitivities.

The other half of the problem is reset time. If you alert on a one-hour window alone, a five-minute total outage keeps the alert firing for a full hour after recovery, because the window still contains the bad data. The standard fix is to require that a short window is also burning before firing. The short window clears quickly, so the alert resolves promptly.

The standard multi-window, multi-burn-rate configuration

These four rules together give you fast detection of catastrophic failure, reliable detection of moderate degradation, and a paper trail for slow leaks — without paging anyone for a blip. The short window in each row is one twelfth of the long window, which is the ratio that keeps reset times short.

Implementing burn-rate alerts in Prometheus

Build this in two layers. First, recording rules that compute the bad-event ratio over each window you need. Second, alert rules that compare those ratios against burn-rate multiples of the budget. Splitting it this way keeps the alert expressions readable and keeps evaluation cheap.

prometheus/rules/slo-burn-recording.yml

groups:
  - name: slo-checkout-burn-recording
    interval: 30s
    rules:
      - record: sli:checkout_bad:ratio5m
        expr: 1 - sli:checkout_availability:ratio5m

      - record: sli:checkout_bad:ratio30m
        expr: |
          1 - (
            sum(rate(http_requests_total{job="checkout-api",code!~"5.."}[30m]))
            / sum(rate(http_requests_total{job="checkout-api"}[30m]))
          )

      - record: sli:checkout_bad:ratio1h
        expr: |
          1 - (
            sum(rate(http_requests_total{job="checkout-api",code!~"5.."}[1h]))
            / sum(rate(http_requests_total{job="checkout-api"}[1h]))
          )

      - record: sli:checkout_bad:ratio2h
        expr: |
          1 - (
            sum(rate(http_requests_total{job="checkout-api",code!~"5.."}[2h]))
            / sum(rate(http_requests_total{job="checkout-api"}[2h]))
          )

      - record: sli:checkout_bad:ratio6h
        expr: |
          1 - (
            sum(rate(http_requests_total{job="checkout-api",code!~"5.."}[6h]))
            / sum(rate(http_requests_total{job="checkout-api"}[6h]))
          )

      - record: sli:checkout_bad:ratio1d
        expr: |
          1 - (
            sum(rate(http_requests_total{job="checkout-api",code!~"5.."}[1d]))
            / sum(rate(http_requests_total{job="checkout-api"}[1d]))
          )

      - record: sli:checkout_bad:ratio3d
        expr: |
          1 - (
            sum(rate(http_requests_total{job="checkout-api",code!~"5.."}[3d]))
            / sum(rate(http_requests_total{job="checkout-api"}[3d]))
          )

prometheus/rules/slo-burn-alerts.yml

groups:
  - name: slo-checkout-burn-alerts
    rules:
      - alert: CheckoutErrorBudgetBurnFast
        expr: |
          sli:checkout_bad:ratio1h  > (14.4 * 0.001)
          and
          sli:checkout_bad:ratio5m  > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
          slo: checkout-availability
          long_window: 1h
        annotations:
          summary: "Checkout is burning error budget at 14.4x (2% consumed in 1h)"
          description: >-
            At this rate the 28-day budget is gone in under two days.
          runbook_url: https://runbooks.internal/checkout/error-budget-burn
          dashboard_url: https://grafana.internal/d/checkout-slo

      - alert: CheckoutErrorBudgetBurnModerate
        expr: |
          sli:checkout_bad:ratio6h  > (6 * 0.001)
          and
          sli:checkout_bad:ratio30m > (6 * 0.001)
        for: 5m
        labels:
          severity: page
          slo: checkout-availability
          long_window: 6h
        annotations:
          summary: "Checkout burning error budget at 6x (5% consumed in 6h)"
          runbook_url: https://runbooks.internal/checkout/error-budget-burn

      - alert: CheckoutErrorBudgetBurnSlow
        expr: |
          sli:checkout_bad:ratio1d > (3 * 0.001)
          and
          sli:checkout_bad:ratio2h > (3 * 0.001)
        for: 15m
        labels:
          severity: ticket
          slo: checkout-availability
        annotations:
          summary: "Checkout burning error budget at 3x - investigate this week"
          runbook_url: https://runbooks.internal/checkout/error-budget-burn

      - alert: CheckoutErrorBudgetBurnVerySlow
        expr: |
          sli:checkout_bad:ratio3d > (1 * 0.001)
          and
          sli:checkout_bad:ratio6h > (1 * 0.001)
        for: 1h
        labels:
          severity: ticket
          slo: checkout-availability
        annotations:
          summary: "Checkout on track to exhaust its error budget this window"
          runbook_url: https://runbooks.internal/checkout/error-budget-burn

The low-traffic problem

Burn-rate alerting assumes enough events to make a ratio meaningful. On a service handling two requests a minute, a single failure is a 50% error rate and a 500x burn rate. You will page constantly for nothing.

• Add a minimum traffic guard so the rule only evaluates when there is enough volume to be statistically meaningful.
• Lengthen the windows for genuinely low-traffic services; a 6h/30m pair may be the fastest sensible rule.
• Aggregate related low-traffic services into one SLO where they share a failure domain.
• Use synthetic traffic to establish a reliable baseline volume, and include it in the SLI deliberately.
• Alert on absolute counts instead for very low volume: 'more than 5 failures in 10 minutes' is honest, and pretending otherwise is not.

Minimum traffic guard: at least 1 req/s over the hour

      - alert: CheckoutErrorBudgetBurnFast
        expr: |
          (
            sli:checkout_bad:ratio1h > (14.4 * 0.001)
            and
            sli:checkout_bad:ratio5m > (14.4 * 0.001)
          )
          and
          sum(rate(http_requests_total{job="checkout-api"}[1h])) > 1

Reporting on the budget

On-call needs the burn rate; leadership needs remaining budget as a percentage. That second number is the one that belongs on a wall display and in the monthly review, because it is the only reliability metric a non-engineer can act on.

Error budget remaining

# Remaining budget as a fraction of the total, over a 28-day window.
# 1.0 = untouched, 0.0 = exhausted, negative = SLO missed.

1 - (
  (
    sum(increase(http_requests_total{job="checkout-api",code=~"5.."}[28d]))
    /
    sum(increase(http_requests_total{job="checkout-api"}[28d]))
  ) / 0.001
)

Quick answers