p50, p95, p99 and the truth about averages

The average is the most confidently wrong number on your dashboard

Take a service handling 10,000 requests. Nine thousand nine hundred of them return in 50ms. One hundred take 30 seconds because they hit a lock contention path.

Worked example

mean latency = (9900 x 0.05 + 100 x 30) / 10000
             = (495 + 3000) / 10000
             = 0.3495 s

Dashboard says:  "349ms average latency"   -> looks acceptable
Reality:         1 in 100 users waits 30 seconds -> they leave

The mean is a single number describing a distribution that is not remotely normal. Latency distributions in real systems are right-skewed with long tails, and the mean sits in a region where almost no actual request lives. It is not a summary of user experience; it is an artefact.

What a percentile actually means

The pth percentile is the value below which p percent of observations fall. If p99 latency is 800ms, then 99% of requests completed in under 800ms and 1% took longer.

Why the tail matters more than it looks

The instinctive response to p99 is that 1% is a rounding error. Two things make that wrong. First, it is not 1% of users, it is 1% of requests — and a single page load may make twenty requests, so far more than 1% of page loads contain at least one slow call. Second, tail latency compounds under fan-out.

Tail amplification

A request that fans out to N backend calls in parallel and must wait
for all of them is slow if ANY of them is slow.

P(request is fast) = (1 - 0.01)^N        # each backend has p99 = 1s

  N =   1  ->  99.0% fast   ->  1 in 100 requests hits the tail
  N =   5  ->  95.1% fast   ->  1 in  20
  N =  10  ->  90.4% fast   ->  1 in  10
  N =  50  ->  60.5% fast   ->  2 in   5
  N = 100  ->  36.6% fast   ->  nearly 2 in 3 requests hit the tail

Your backend p99 has become your frontend p50.

This is why microservice architectures make tail latency a first-class concern rather than a nicety. The more services a request touches, the more the aggregate experience is governed by the worst percentile, not the typical one.

How Prometheus computes percentiles

Prometheus does not store individual observations. A histogram metric stores counts in cumulative buckets, defined at instrumentation time by their upper bound, the le label. Every observation increments every bucket whose bound it falls under.

Histogram anatomy

# One scrape of http_request_duration_seconds

http_request_duration_seconds_bucket{le="0.005"}  120
http_request_duration_seconds_bucket{le="0.01"}   450
http_request_duration_seconds_bucket{le="0.025"}  1890
http_request_duration_seconds_bucket{le="0.05"}   7200
http_request_duration_seconds_bucket{le="0.1"}    9100
http_request_duration_seconds_bucket{le="0.25"}   9800
http_request_duration_seconds_bucket{le="0.5"}    9950
http_request_duration_seconds_bucket{le="1"}      9990
http_request_duration_seconds_bucket{le="+Inf"}   10000
http_request_duration_seconds_sum                 812.4
http_request_duration_seconds_count               10000

# Buckets are CUMULATIVE. 9,100 requests took <= 100ms.
# The number that took between 50ms and 100ms is 9100 - 7200 = 1900.

histogram_quantile() finds the bucket containing the target rank and interpolates linearly within it. That interpolation is the source of every surprising histogram result: the accuracy of your p99 depends entirely on whether you have a bucket boundary near the real p99.

The correct pattern

# Correct: rate the buckets first, sum by le, then compute the quantile
histogram_quantile(
  0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout-api"}[5m]))
)

# Per-route p99 - keep route in the grouping alongside le
histogram_quantile(
  0.99,
  sum by (le, route) (rate(http_request_duration_seconds_bucket{job="checkout-api"}[5m]))
)

Choosing bucket boundaries

Default bucket sets are designed for a generic HTTP service and are wrong for almost everything else. Choose boundaries around the values you actually care about, especially your SLO threshold.

• Put a bucket exactly on your SLO threshold. If the SLO is 'under 400ms', you need an le="0.4" bucket. Without it, your SLI is an interpolation.
• Space them roughly exponentially across the range you expect, since latency spans orders of magnitude.
• Cover the tail. Your highest finite bucket should sit above your worst realistic latency, usually near your client timeout.
• Keep the count modest. Every bucket is a separate time series, multiplied by every other label. Ten to fifteen buckets is usually plenty. Part 7 explains exactly what this costs.

Bucket definition

# Go, prometheus/client_golang
requestDuration := prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name: "http_request_duration_seconds",
        Help: "Request duration in seconds.",
        // Tuned for a service with a 400ms SLO and a 10s client timeout.
        Buckets: []float64{
            0.005, 0.01, 0.025, 0.05, 0.1, 0.2,
            0.4,          // <- the SLO threshold, deliberately present
            0.8, 1.5, 3, 5, 10,
        },
    },
    []string{"route", "method", "code"},
)

Same idea in Python

# Python, prometheus_client
from prometheus_client import Histogram

REQUEST_DURATION = Histogram(
    "http_request_duration_seconds",
    "Request duration in seconds.",
    ["route", "method", "code"],
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.2,
             0.4,   # SLO threshold
             0.8, 1.5, 3, 5, 10, float("inf")),
)

The mistake almost everyone makes: averaging percentiles

Percentiles are not additive. You cannot average them, sum them, or take the maximum across instances and get a meaningful number. This is not a Prometheus quirk; it is a property of quantiles.

Aggregation order matters

# WRONG - computes a p99 per instance, then averages those p99s.
# The result is not the p99 of anything.
avg(
  histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
)

# ALSO WRONG - the same error hidden inside a Grafana panel that
# graphs one series per instance and enables the "mean" legend value.

# WRONG in a different way - a "p99 of p99s" across services
max(service_p99)

# CORRECT - aggregate the underlying buckets, then compute one quantile
histogram_quantile(
  0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

The intuition: consider ten instances, nine idle and one overloaded. The nine idle instances each have a p99 of 10ms; the overloaded one has a p99 of 5s. The average of those p99s is about 510ms. But if the overloaded instance is serving 80% of traffic, the true system-wide p99 is close to 5s. Averaging the percentiles produced a number that is wrong by an order of magnitude and wrong in the reassuring direction.

The same applies over time. Do not average a p99 graph over a day to get a 'daily p99'. Recompute the quantile from buckets aggregated over the day.

Native histograms

Classic histograms force the trade-off between resolution and cardinality: more buckets means more series. Native histograms, available in Prometheus 2.40+ behind a feature flag and stable in Prometheus 3.x, store the whole distribution in a single series with exponentially spaced buckets generated automatically.

• One series per histogram instead of one per bucket, which typically cuts histogram series count by an order of magnitude.
• Resolution is a schema parameter, not a hand-tuned list, so you stop guessing bucket boundaries.
• Buckets adapt to the observed range, so a p99 far above your guesses is still measured accurately.
• Requires protobuf scraping and support in your client library and storage backend, so check the whole chain before adopting.

Enabling native histograms

# prometheus.yml - enable the feature and protobuf negotiation
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: checkout-api
    # Required for native histogram ingestion on 2.x
    # (start Prometheus with --enable-feature=native-histograms)
    static_configs:
      - targets: ["checkout-api:8080"]

# Query syntax is unchanged - the same function works on both types
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds[5m])))

The better move: stop reporting percentiles in your SLO

A latency SLO expressed as 'p99 under 400ms' is harder to reason about than it looks, because a percentile is not an event you can count and therefore does not slot cleanly into an error budget. The threshold-count formulation from Part 2 is strictly better:

Threshold counting beats quantiles for SLOs

# Instead of: "p99 latency must be under 400ms"
# Say:         "99% of requests must complete in under 400ms"
#
# Which is directly measurable as a good/valid ratio:

sum(rate(http_request_duration_seconds_bucket{le="0.4"}[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))

# This gives you:
#   - a ratio that plugs straight into the error budget maths
#   - no interpolation error
#   - the same burn-rate alert template as availability

Keep percentile graphs for debugging, where seeing the shape of the distribution move is exactly what you want. Use threshold ratios for objectives, where you need something countable.

Quick answers