Cardinality: the design problem nobody warns you about

What cardinality actually is

In Prometheus and every system modelled on it, a time series is uniquely identified by its metric name plus the exact set of label key-value pairs attached to it. Change one label value and you have created a new, separate series with its own index entry, its own memory footprint, and its own chunk on disk.

The multiplication

# These are FOUR distinct time series, not one metric with four dimensions.

http_requests_total{route="/checkout", method="POST", code="200"}
http_requests_total{route="/checkout", method="POST", code="500"}
http_requests_total{route="/cart",     method="GET",  code="200"}
http_requests_total{route="/cart",     method="GET",  code="404"}

# Cardinality of a metric = the product of its label value counts.

  routes (40) x methods (4) x codes (12) = 1,920 series
  ... x instances (30)                   = 57,600 series

# Now add one innocent-looking label:

  ... x customer_id (5,000)              = 288,000,000 series

Why it is expensive

Each active series costs memory in the head block, an entry in the inverted index, and storage on disk. The rough figures for Prometheus are worth memorising because they let you sanity-check a design before you deploy it.

The failure mode is not gradual. Prometheus runs fine, then a deploy introduces a new label value, the head block grows past available memory, the process is OOM-killed, it replays the WAL on restart, runs out of memory again, and you now have no monitoring during an incident that you cannot see. Cardinality explosions take out observability precisely when you need it.

Managed backends move this cost from a crash to an invoice. Most vendors bill on active series or custom metrics, so an unbounded label produces a five-figure surprise rather than an outage. Neither outcome is one you want to discover in production.

The labels that cause disasters

Budgeting cardinality before you write the code

Treat cardinality like a resource with a quota, because that is exactly what it is. Do the multiplication at design time, on paper, before the instrumentation is merged.

Doing the arithmetic first

Metric: http_request_duration_seconds  (histogram)

  route            48   (templated, from the router table)
  method            5   (GET, POST, PUT, DELETE, PATCH)
  code             10   (200,201,204,400,401,403,404,409,429,500)
  le               12   (bucket boundaries) + _sum + _count = 14
  instance         24   (pods at peak, HPA max)
  ---------------------------------------------------------------
  48 x 5 x 10 x 14 x 24  =  806,400 series

  ... for ONE metric on ONE service.

Reality check: at ~3.5 KB per series that is ~2.8 GB of head memory
for a single histogram. This design does not survive.

How to bring that number down

1. Do not label buckets by code. You rarely need per-status-code latency percentiles. Split the histogram labels (route, method) from the counter labels (route, method, code). That alone divides the histogram by 10.
2. Reduce bucket count. Twelve buckets is often six buckets' worth of useful resolution. Or move to native histograms, which collapse all buckets into one series.
3. Group status codes. A code_class label with values 2xx, 3xx, 4xx, 5xx reduces ten values to four, and it is what you actually query.
4. Drop instance at aggregation time. You need per-instance data for a short retention window to find a bad pod; you do not need it for a year.

After the fixes

48 routes x 5 methods x 8 buckets(+2) x 24 instances = 57,600 series
                                                          (down from 806,400)

With native histograms:
48 x 5 x 24 = 5,760 series

Finding your existing cardinality problems

Cardinality forensics

# Prometheus ships a cardinality report - the fastest way to start
# UI:  Status -> TSDB Status
# It shows top metric names by series count, and the label pairs
# with the most series.

# Total active series
prometheus_tsdb_head_series

# Series added per second - a step change here is your smoking gun
rate(prometheus_tsdb_head_series_created_total[5m])

# Which metric names dominate (Prometheus 2.47+)
topk(20, count by (__name__)({__name__=~".+"}))

# How many distinct values does one label have?
count(count by (route) (http_requests_total))

# Which label is inflating a specific metric?
topk(10, count by (__name__, job) ({__name__=~"http_.*"}))

promtool tsdb analyze

# promtool analyses a TSDB block offline - no query load on the server
$ promtool tsdb analyze /prometheus/data

Block ID: 01HQ...
Duration: 2h0m0s
Series: 1,482,331
Label names: 87
Postings entries: 4,912,004

Highest cardinality labels:
  482119  pod
  310288  id
  148233  instance
  ...

Highest cardinality metric names:
  612004  container_network_receive_bytes_total
  198221  http_request_duration_seconds_bucket
  ...

Label pairs most involved in churning series:
  148233  job=kubelet
  ...

Defence in depth: stopping it at three layers

1. At instrumentation

Bound the values at the source

// WRONG - r.URL.Path contains the order ID.
// One series per order, forever.
reqTotal.WithLabelValues(r.URL.Path, r.Method, code).Inc()

// RIGHT - the route TEMPLATE from the router
// chi:      chi.RouteContext(r.Context()).RoutePattern()
// gin:      c.FullPath()
// echo:     c.Path()
// express:  req.route.path
// FastAPI:  request.scope["route"].path
reqTotal.WithLabelValues(routePattern, r.Method, code).Inc()

// If you must derive it yourself, use an explicit allow-list and
// bucket everything else. Never trust user input as a label value.
func normaliseRoute(path string) string {
    if pattern, ok := knownRoutes[path]; ok {
        return pattern
    }
    return "other"   // the safety valve
}

2. At scrape time, with relabeling

prometheus.yml

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    metric_relabel_configs:
      # Drop a metric family you never query but that costs a fortune
      - source_labels: [__name__]
        regex: 'container_network_tcp_usage_total|container_tasks_state'
        action: drop

      # Drop a high-cardinality label from a metric you DO want.
      # The series collapse together and Prometheus sums them.
      - source_labels: [__name__]
        regex: 'apiserver_request_duration_seconds_bucket'
        target_label: id
        replacement: ''
        action: replace

      # Keep only the metrics you have actually put on a dashboard.
      # Aggressive, effective, and requires discipline to maintain.
      - source_labels: [__name__]
        regex: '(http_requests_total|http_request_duration_seconds_.*|up|process_.*)'
        action: keep

      # Enforce a hard ceiling per target - the circuit breaker
    sample_limit: 20000
    label_limit: 30
    label_value_length_limit: 128

3. At query and storage time

Aggregate down for long-term storage

groups:
  - name: aggregation
    interval: 60s
    rules:
      # Collapse per-pod series into per-deployment series.
      # Keep raw data for 15 days, keep these for a year.
      - record: deployment:http_requests:rate5m
        expr: |
          sum by (namespace, deployment, route, code_class) (
            label_replace(
              rate(http_requests_total[5m]),
              "code_class", "$1xx", "code", "(.)..")
          )

      - record: deployment_le:http_request_duration_seconds_bucket:rate5m
        expr: |
          sum by (namespace, deployment, le) (
            rate(http_request_duration_seconds_bucket[5m])
          )

Alert on cardinality itself

Monitor the monitoring

groups:
  - name: observability-health
    rules:
      - alert: PrometheusCardinalitySpike
        expr: |
          increase(prometheus_tsdb_head_series[1h]) > 100000
        for: 15m
        labels:
          severity: ticket
        annotations:
          summary: "Active series grew by >100k in an hour"
          description: >-
            Check Status -> TSDB Status for the new top metric or label.
            A recent deploy is the usual cause.
          runbook_url: https://runbooks.internal/observability/cardinality

      - alert: PrometheusScrapeSampleLimitHit
        expr: |
          increase(prometheus_target_scrapes_exceeded_sample_limit_total[15m]) > 0
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "A target is exceeding sample_limit - metrics are being dropped"

      - alert: PrometheusHeadSeriesApproachingCapacity
        expr: |
          prometheus_tsdb_head_series > 4000000
        for: 30m
        labels:
          severity: ticket
        annotations:
          summary: "Prometheus head series above the sizing assumption"

      - alert: PrometheusHighSeriesChurn
        expr: |
          rate(prometheus_tsdb_head_series_created_total[1h])
          / clamp_min(rate(prometheus_tsdb_head_samples_appended_total[1h]), 1)
          > 0.01
        for: 1h
        labels:
          severity: ticket
        annotations:
          summary: "High series churn - a label is probably ephemeral (pod name, build ID)" 

Where the high-cardinality data should go instead

None of this means you cannot have per-user or per-request detail. It means metrics are the wrong place for it. The three signals have different cost models, and the skill is routing each dimension to the signal that can afford it.

Exemplars are the bridge. An exemplar attaches a trace ID to a specific histogram bucket observation, so you can click a spike on a latency graph and land directly in a trace of a request that caused it — getting the drill-down without paying the cardinality cost.

Exemplars in practice

// Go: record an exemplar alongside the observation
observer := reqDuration.WithLabelValues(route, method, code)
if ex, ok := observer.(prometheus.ExemplarObserver); ok {
    ex.ObserveWithExemplar(elapsed, prometheus.Labels{
        "trace_id": span.SpanContext().TraceID().String(),
    })
} else {
    observer.Observe(elapsed)
}

prometheus.yml

# Prometheus must be started with exemplar storage enabled:
#   --enable-feature=exemplar-storage
storage:
  exemplars:
    max_exemplars: 100000

# And the scrape must negotiate OpenMetrics, which Prometheus
# does automatically when the target advertises it.

Quick answers