Metrics fundamentals and PromQL

Four metric types, and the consequences of choosing wrong

Prometheus has a small type system, and nearly every misleading graph in production comes from picking the wrong member of it.

The counter reset problem, and why rate() exists

A counter goes back to zero when the process restarts. Naively subtracting two samples would produce a large negative number every deploy. rate() and increase() detect resets and compensate, which is the entire reason you must never graph a counter's raw value.

rate, irate, and increase

The three of them

# rate() - per-second average over the window, extrapolated to the
# window edges, reset-aware. This is your default.
rate(http_requests_total[5m])

# increase() - the total change over the window.
# Exactly equivalent to rate(x[5m]) * 300, with the same extrapolation.
increase(http_requests_total[1h])

# irate() - instantaneous rate from the last TWO samples only.
# Very responsive, very noisy, and it will miss spikes entirely when
# the graph resolution is coarser than the sample interval.
# Use for high-resolution debugging graphs, never for alerts.
irate(http_requests_total[5m])

Extrapolation surprises

rate() and increase() extrapolate to the window boundaries, which means increase() on a slow-moving counter can return non-integer values like 3.7 events. That is expected behaviour, not a bug. If you need exact counts for billing or reporting, query the raw counter difference rather than using increase().

Aggregation order is not optional

This is the single most common PromQL error, and it produces answers that are wrong rather than obviously broken.

rate before sum

# WRONG - sums the raw counters, then rates the sum.
# Every process restart in the fleet becomes a phantom negative spike,
# and the reset correction cannot work because it has been applied
# to an aggregate that is not a counter.
rate(sum(http_requests_total)[5m:])

# CORRECT - rate each series, then sum the rates.
sum(rate(http_requests_total[5m]))

# The rule: rate() FIRST, then sum(). Always.
# Same for increase(), irate(), delta(), and deriv().

Ratio of sums, not average of ratios

# WRONG - averages ratios, weighting a 3am hour equally with noon
avg(
  rate(http_requests_total{code=~"5.."}[5m])
  / rate(http_requests_total[5m])
)

# CORRECT - ratio of sums, traffic-weighted
sum(rate(http_requests_total{code=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

# The rule: sum the numerator, sum the denominator, then divide.

Label matching in binary operations

When you combine two metrics, Prometheus matches series by their full label sets. Any label present on one side and not the other causes the match to fail silently and return no data, which is the second most common source of empty panels.

Vector matching

# Fails: http_requests_total has code, method, route;
# the denominator after sum() has none of them.
rate(http_requests_total{code=~"5.."}[5m]) / sum(rate(http_requests_total[5m]))

# Works: both sides are aggregated to the same label set
sum by (route) (rate(http_requests_total{code=~"5.."}[5m]))
/
sum by (route) (rate(http_requests_total[5m]))

# When the label sets genuinely differ, be explicit:
#   on(...)     - match only on these labels
#   ignoring(...) - match on everything except these
#   group_left  - many-to-one: extra labels come from the LEFT side
#   group_right - one-to-many: extra labels come from the RIGHT side

# Classic use: join a metric with a metadata / "info" metric
sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))
  * on(pod) group_left(app, team)
    kube_pod_labels

Recording rules: precompute what you query often

Any expression that appears on a dashboard many people load, or inside an alert evaluated every fifteen seconds, should be a recording rule. This moves the cost from query time to ingest time, gives every consumer identical arithmetic, and keeps dashboards fast when the range is a month.

prometheus/rules/red.yml

groups:
  - name: service-red
    interval: 30s
    rules:
      # Naming convention: level:metric:operations
      #   level      - the aggregation level (job, route, cluster)
      #   metric     - the underlying metric name
      #   operations - what was applied, most recent first

      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_requests_errors:rate5m
        expr: sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))

      - record: job:http_error_ratio:rate5m
        expr: |
          job:http_requests_errors:rate5m / job:http_requests:rate5m

      - record: job_le:http_request_duration_seconds_bucket:rate5m
        expr: |
          sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))

      - record: job:http_request_duration_seconds:p99
        expr: |
          histogram_quantile(0.99, job_le:http_request_duration_seconds_bucket:rate5m)

• Rules within a group evaluate sequentially, so a later rule can safely depend on an earlier one in the same group.
• Groups evaluate in parallel with each other, so a rule must never depend on a different group.
• Set interval no shorter than your scrape interval; there is nothing to gain from evaluating faster than data arrives.
• Keep the naming convention rigid. level:metric:operations makes it obvious at a glance whether a series is raw or derived.

Choosing a good metric name

Naming that survives contact with a second team

# Convention:  <namespace>_<subsystem>_<name>_<unit>[_total]

http_requests_total                      # counter, no unit needed
http_request_duration_seconds            # histogram, base unit seconds
http_response_size_bytes                 # histogram, base unit bytes
process_resident_memory_bytes            # gauge
queue_depth                              # gauge
kafka_consumer_lag_messages              # gauge with an explicit unit

# Anti-patterns seen in the wild:
api_latency_ms          # use base units: seconds, not milliseconds
requestCount            # use snake_case
http_requests           # counters should end in _total
errors                  # namespace it: myapp_http_errors_total
cpu_usage_percent       # use a ratio 0-1, or expose the raw counter

Always use base units — seconds and bytes — and let the dashboard handle presentation. Mixed units across services turn every cross-service query into a unit-conversion puzzle, and someone will eventually get it wrong during an incident.

Queries that will bite you

Quick answers