Golden signals, USE and RED

Four questions that cover almost every incident

When a service is misbehaving, the diagnostic space feels infinite. It is not. The Google SRE book reduced it to four golden signals, and after enough incidents you notice that nearly every one is explained by a combination of them.

Utilisation is not saturation, and the difference matters

These two get used interchangeably and they mean different things. Confusing them is why teams keep setting CPU alerts that either fire constantly or never fire when it matters.

• Utilisation is the fraction of a resource's capacity currently in use. CPU at 70%. Disk 60% full. Pool using 12 of 20 connections. It is bounded between 0 and 100%.
• Saturation is the amount of work that cannot be serviced yet and is waiting. Run queue length. Requests queued for a connection. Messages backing up in a topic. It is unbounded, and it is the signal that actually predicts pain.

A CPU at 100% utilisation with a run queue of zero is perfectly healthy: it is doing exactly as much work as arrives, with no waiting. A CPU at 60% utilisation with a run queue of 40 is in trouble, because something else — probably I/O or lock contention — is preventing work from proceeding.

Utilisation vs saturation in PromQL

# Linux: utilisation and saturation are different fields entirely

# CPU utilisation - fraction of time not idle
1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))

# CPU saturation - processes waiting for a CPU that is busy
node_load1 / on(instance) count by (instance) (node_cpu_seconds_total{mode="idle"})

# Memory utilisation - fraction of RAM in use
1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

# Memory saturation - the system is paging, which is the real signal
rate(node_vmstat_pgmajfault[5m])

# Disk utilisation - fraction of time the device was busy
rate(node_disk_io_time_seconds_total[5m])

# Disk saturation - weighted queue time; rising means requests are waiting
rate(node_disk_io_time_weighted_seconds_total[5m])

Pressure Stall Information, available on modern Linux kernels, gives you saturation directly and is far more useful than load average for this purpose:

PSI: saturation without inference

$ cat /proc/pressure/cpu
some avg10=12.45 avg60=8.90 avg300=4.12 total=8871234567

# "some" = at least one task stalled on this resource
# "full" = ALL non-idle tasks stalled (only for memory and io)
#
# avg10 = percentage of the last 10 seconds spent stalled.
# Anything sustained above ~10% for CPU, or above 0 for memory "full",
# deserves investigation.

The USE method: for resources

Brendan Gregg's USE method is a checklist for hardware and OS-level resources. For every resource, check Utilisation, Saturation, and Errors. Its value is completeness — working the list finds the bottleneck faster than following a hunch.

Kubernetes CPU throttling

# CPU throttling ratio by pod - the metric most K8s dashboards forget
sum by (namespace, pod) (
  rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m])
)
/
sum by (namespace, pod) (
  rate(container_cpu_cfs_periods_total{container!=""}[5m])
)

# Sustained above ~0.05 means your limits are the bottleneck,
# not your code.

The RED method: for services

USE is for things you can run out of. RED, popularised by Tom Wilkie, is for request-driven services: Rate, Errors, Duration. It maps neatly onto three of the four golden signals, and it has one enormous practical advantage — it is identical for every service, so you can build it once as a dashboard template and apply it everywhere.

The complete RED dashboard, in six queries

# RATE - requests per second, by route
sum by (route) (rate(http_requests_total{job="$service"}[5m]))

# ERRORS - proportion of requests failing
sum(rate(http_requests_total{job="$service", code=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="$service"}[5m]))

# DURATION - p50, p95, p99 from one histogram
histogram_quantile(0.50, sum by (le) (rate(http_request_duration_seconds_bucket{job="$service"}[5m])))
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="$service"}[5m])))
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="$service"}[5m])))

If every service in your estate exposes http_requests_total and http_request_duration_seconds with consistent label names, one templated Grafana dashboard covers your entire fleet. That consistency is worth more than any individual bespoke dashboard, and enforcing it is one of the highest-leverage things a platform team can do.

Putting them together during an incident

The methods compose into a triage order that works under pressure, when nobody wants to think creatively.

1. Confirm user impact with RED at the edge. Are errors up, is latency up, or both? If neither, the alert may be the problem.
2. Check traffic first. A latency rise with a traffic rise is a capacity story. A latency rise with flat traffic is a dependency or code story. A traffic collapse with no errors means the problem is upstream of you.
3. Walk RED down the call graph. Which downstream service's duration started climbing first? Traces answer this in seconds; see Part 9.
4. At the guilty service, switch to USE. Which resource is saturated? Check saturation before utilisation, because saturation moves first.
5. Check the boring causes. Recent deploy, config change, feature flag, certificate expiry, credential rotation, or a neighbour on the same node. Most incidents are one of these.

The reason to write this down and put it in the runbook is not that it is clever. It is that at 3 a.m. an on-call engineer needs a list, not a methodology.

Instrumenting for these signals from the start

One middleware, all four signals

// Go: a middleware that produces every RED signal at once
var (
    reqTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total HTTP requests.",
        },
        // route must be the TEMPLATE, not the raw path - see Part 7
        []string{"route", "method", "code"},
    )
    reqDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "http_request_duration_seconds",
            Help:    "HTTP request duration in seconds.",
            Buckets: []float64{0.005, 0.01, 0.025, 0.05, 0.1, 0.2, 0.4, 0.8, 1.5, 3, 5, 10},
        },
        []string{"route", "method", "code"},
    )
    inFlight = prometheus.NewGauge(
        prometheus.GaugeOpts{
            Name: "http_requests_in_flight",
            Help: "Requests currently being served (a saturation signal).",
        },
    )
)

func Instrument(route string, next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        inFlight.Inc()
        defer inFlight.Dec()

        start := time.Now()
        rec := &statusRecorder{ResponseWriter: w, status: 200}
        next.ServeHTTP(rec, r)

        code := strconv.Itoa(rec.status)
        elapsed := time.Since(start).Seconds()

        reqTotal.WithLabelValues(route, r.Method, code).Inc()
        reqDuration.WithLabelValues(route, r.Method, code).Observe(elapsed)
    })
}

Quick answers