Incidents, postmortems, and what comes next

Incidents need a structure, not heroics

The instinct during an outage is for everyone competent to pile in and start debugging. That produces duplicated work, contradictory changes, a stakeholder channel nobody is updating, and a decision-making process where the loudest voice wins. Incident command exists to prevent all four.

Severity levels that mean something

Write the definitions down and make declaring an incident cheap and blameless. The expensive failure mode is not over-declaring; it is an engineer spending ninety minutes deciding whether something counts as an incident while customers are affected.

The first fifteen minutes

1. Declare. Open the channel, name the IC out loud. Do not wait for certainty about severity — start high and downgrade.
2. Assess impact. Which SLO is burning, how fast, and who is affected? This determines everything that follows.
3. Stop the bleeding before understanding it. Roll back, fail over, shed load, disable the feature flag. Diagnosis is a luxury you buy with mitigation.
4. Communicate. First stakeholder update within fifteen minutes even if it only says 'we are investigating'. Silence is read as incompetence.
5. Only then diagnose. With impact contained, use the triage sequence from Part 5: RED at the edge, walk the call graph, USE at the suspect.

Postmortems that produce change

A postmortem is worth writing only if something is different afterwards. Most are not, and the reasons are consistent: they are written by one person days later, they blame an individual, and their action items have no owner or date.

Blameless means structurally blameless

'Blameless' does not mean politely avoiding names. It means starting from the assumption that everyone acted reasonably given the information they had, and asking why the system made the wrong action look right. If an engineer ran a destructive command against production, the finding is not that they were careless. It is that a destructive command was runnable against production without a confirmation, a review, or a guard.

postmortem-2026-07-14-checkout.md

# Incident 2026-07-14: Checkout unavailable for 34 minutes

**Severity:** SEV-1
**Duration:** 14:02 - 14:36 UTC (34 min)
**Error budget consumed:** 71% of the 28-day budget
**Customer impact:** ~18,400 failed checkout attempts, est. GBP 240k deferred revenue

## Timeline (UTC)
| Time  | Event |
|-------|-------|
| 13:58 | Deploy of checkout-api v2.14.0 begins |
| 14:02 | CheckoutErrorBudgetBurnFast fires (14.4x). Paged. |
| 14:05 | On-call acknowledges; declares SEV-2, opens incident channel |
| 14:09 | Escalated to SEV-1 - error rate at 94% |
| 14:11 | Trace shows connection pool exhaustion on the orders DB |
| 14:18 | Rollback to v2.13.4 initiated |
| 14:31 | Rollback complete; error rate begins falling |
| 14:36 | Error rate normal. Incident resolved. |

## What happened
v2.14.0 added a per-request call to the loyalty service inside the
checkout transaction. Each call held a database connection for the
duration of the external call. Under production concurrency the pool
of 20 was exhausted within four minutes.

## Why it was not caught
- Load testing ran against a staging loyalty service responding in 8ms;
  production responds in 180ms at p50.
- Connection pool saturation was not on any dashboard, and had no alert.
- The change was reviewed by two engineers; neither had context on the
  transaction boundary.

## Why detection took 4 minutes and mitigation 29
- Detection was correct and fast; the alert worked as designed.
- Rollback took 13 minutes because the deploy pipeline had no one-click
  rollback and required a manual Argo sync with an approval step.

## What went well
- Burn-rate alert fired within 4 minutes with an accurate summary.
- Trace-based diagnosis identified the cause in 6 minutes.
- Comms lead posted to the status page at 14:12.

## Action items
| # | Action | Owner | Due | Ticket |
|---|--------|-------|-----|--------|
| 1 | Add connection pool saturation to the service dashboard and a ticket-level alert | @priya | 2026-07-21 | OBS-441 |
| 2 | One-click rollback in the deploy pipeline, no approval for reverts | @sam | 2026-07-28 | PLT-102 |
| 3 | Load test against production-representative dependency latency | @jo | 2026-08-11 | CHK-889 |
| 4 | Lint rule: forbid external calls inside a DB transaction | @priya | 2026-08-11 | CHK-890 |
| 5 | Error budget policy: freeze until budget recovers above 25% | @lead | immediate | - |

• Every action item has one named owner and a date. A team is not an owner.
• Action items go in the normal backlog with the same priority mechanics as everything else. A separate 'postmortem tracker' is where actions go to be forgotten.
• Review completion rate monthly. If under half of postmortem actions ship, the postmortems are theatre and everyone knows it.
• Publish them internally. The learning value is mostly for people who were not in the incident.

On-call as a system to be measured

On-call quality determines retention more than almost anything else in an engineering organisation, and it is measurable. Track these and review them quarterly like any other SLO.

• Compensate on-call, in money or in time off. Unpaid out-of-hours work is a hidden cost that eventually gets paid in resignations.
• Run a proper handover at each shift change: what fired, what is degraded, what is deploying, what is deferred.
• Give the on-call engineer explicit permission to have no sprint commitments. On-call is the work.
• Rotate people through it. An on-call rotation that excludes the developers who write the code removes the feedback loop that makes the code better.

Scaling the platform without scaling the bill

As the estate grows, the observability platform becomes one of the larger line items in the infrastructure budget. Four levers control it, in descending order of effect.

1. Aggregate before long-term storage

Downsampling by aggregation

groups:
  - name: longterm-aggregation
    interval: 60s
    rules:
      # Per-pod detail is valuable for 15 days and worthless for a year.
      # These aggregates typically cut retained series by 20-50x.
      - record: cluster_service:http_requests:rate5m
        expr: |
          sum by (cluster, service, route, code_class) (
            label_replace(rate(http_requests_total[5m]),
              "code_class", "$1xx", "code", "(.)..")
          )

      - record: cluster_service_le:http_request_duration_seconds_bucket:rate5m
        expr: |
          sum by (cluster, service, le) (
            rate(http_request_duration_seconds_bucket[5m])
          )

2. Set retention per signal and per tier

3. Attribute the cost

Multi-tenancy with per-team limits does two things: it contains the blast radius of one team's cardinality accident, and it makes the cost visible to the people who can reduce it. A monthly report of active series and log volume by team changes behaviour in a way that no amount of documentation does.

Who is using what

# Cost attribution queries
# Active series by team
sort_desc(
  count by (team) ({__name__=~".+"})
)

# Log volume by namespace over 24h, in GB
sort_desc(
  sum by (namespace) (
    bytes_over_time({namespace=~".+"}[24h])
  ) / 1e9
)

# The 20 most expensive metrics by series count
topk(20, count by (__name__) ({__name__=~".+"}))

4. Prune what nobody queries

A significant fraction of ingested metrics are never queried by anyone. Grafana usage insights and the Mimir query analyser will tell you which metrics appear in dashboards and rules; anything appearing in neither, for ninety days, is a candidate for a drop rule at the collector.

A maturity path, and where to go next

Most organisations sit between 1 and 2, and the jump to 2 is where nearly all of the value is. It requires no new tooling: it requires deciding what 'working' means, measuring it honestly, and agreeing what happens when you miss.

What this series did not cover

• Chaos engineering. Deliberately injecting failure to verify your assumptions and your alerting actually fire.
• Capacity planning and load testing. Forecasting demand and finding the knee in your latency curve before your users do.
• Production readiness reviews. The checklist a service passes before it is permitted to page a human.
• Progressive delivery. Canary and blue-green deploys gated automatically on SLO burn rate.
• Continuous profiling. Pyroscope and the fourth signal, which closes the loop from 'this service is slow' to 'this function is slow'.

Quick answers