Loki's central bet

Elasticsearch indexes the full content of every log line, which makes arbitrary search fast and makes storage and ingest expensive. Loki takes the opposite bet: index only a small set of labels, store the log content itself as compressed chunks in object storage, and brute-force scan those chunks at query time.

  • Storage cost drops dramatically, because object storage is cheap and there is no large inverted index to maintain.
  • Ingest is cheap and predictable, because there is almost no indexing work per line.
  • Queries that narrow well by label are fast, because Loki only fetches the relevant chunks.
  • Queries that do not narrow by label are slow and expensive, because Loki must decompress and scan everything in the time range.

Every practical decision about Loki follows from that trade-off. Labels are the only index you get, so label design in Loki is exactly as consequential as it is in Prometheus — and for exactly the same reason.

Streams: the same cardinality trap, wearing a different hat

A Loki stream is a unique combination of label values, and it is the unit of chunking and indexing. Every distinct combination gets its own chunk stream in object storage.

Stream design

# GOOD - bounded, small, matches how you actually search
{cluster="prod-eu", namespace="checkout", app="checkout-api", level="error"}

# CATASTROPHIC - one stream per request
{cluster="prod-eu", app="checkout-api", request_id="7f3a-91c2-..."}

# BAD - pod name churns on every rollout, creating dead streams
{app="checkout-api", pod="checkout-api-7d9f8b-x2n4l"}

# BAD - free text as a label
{app="checkout-api", error="connection refused to db-01 after 3 retries"}

High stream cardinality in Loki produces many small chunks. Small chunks compress badly, multiply object-storage requests, and make queries slower rather than faster — the opposite of the intuition that more indexing helps. The guidance is to keep total active streams in the low tens of thousands per tenant, and to treat anything unbounded as content rather than label.

Everything you did not put in a label is still fully searchable. It just gets filtered at query time by scanning, which for a well-narrowed time range and label set is fast enough. That is the whole point.

Structured logging is the prerequisite

Loki works on unstructured text, but everything downstream of ingestion is easier if your applications emit JSON or logfmt with consistent field names. Structured logs also make the label-versus-content decision obvious, because the structure tells you which fields are bounded.

Structured logging in Go

// Go, log/slog - structured by default since 1.21
logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
    Level: slog.LevelInfo,
}))

logger.Error("payment authorisation failed",
    "trace_id",   span.SpanContext().TraceID().String(),  // correlation
    "span_id",    span.SpanContext().SpanID().String(),
    "order_id",   order.ID,           // high cardinality: content, not label
    "user_id",    user.ID,            // high cardinality: content, not label
    "provider",   "stripe",           // bounded: could be a label
    "error_code", "card_declined",    // bounded: could be a label
    "attempt",    attempt,
    "err",        err,
)

The resulting line

{"time":"2026-07-27T09:14:22.104Z","level":"ERROR","msg":"payment authorisation failed","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7","order_id":"ord_8f3a91c2","user_id":"usr_44821","provider":"stripe","error_code":"card_declined","attempt":2,"err":"stripe: card was declined"}
  • Always include the trace ID. It is the single field that makes logs, metrics and traces one system instead of three.
  • Use consistent field names across services. trace_id everywhere, not traceId in one service and trace in another.
  • Log the error, not a prose sentence about the error. Structured fields are queryable; sentences are not.
  • Never log secrets, tokens, card numbers, or full request bodies. Logs are widely readable and hard to redact retroactively.

Shipping logs with Grafana Alloy

Alloy is Grafana's current collector and replaces both Promtail and the older Grafana Agent. It discovers targets, extracts labels, and ships to Loki, using a component-based configuration language.

Alloy pipeline for Kubernetes pod logs

// alloy/config.alloy

discovery.kubernetes "pods" {
  role = "pod"
}

discovery.relabel "pod_logs" {
  targets = discovery.kubernetes.pods.targets

  // Promote a small, bounded set of Kubernetes metadata to labels
  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
    target_label  = "app"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label  = "container"
  }

  // Deliberately NOT promoted to a label: pod name.
  // Keep it as structured metadata instead - queryable, not indexed.
  rule {
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label  = "__pod"
  }
}

loki.source.kubernetes "pods" {
  targets    = discovery.relabel.pod_logs.output
  forward_to = [loki.process.parse.receiver]
}

loki.process "parse" {
  forward_to = [loki.write.default.receiver]

  // Parse JSON and promote ONLY the bounded fields
  stage.json {
    expressions = {
      level      = "level",
      trace_id   = "trace_id",
      error_code = "error_code",
    }
  }

  // level is bounded (5 values) - safe as a label
  stage.labels {
    values = { level = "" }
  }

  // trace_id is unbounded - structured metadata, not a label.
  // Indexed enough to filter on, without creating a stream per trace.
  stage.structured_metadata {
    values = { trace_id = "", error_code = "" }
  }

  stage.timestamp {
    source = "time"
    format = "RFC3339Nano"
  }
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Structured metadata, available from Loki 3.0, is the right home for medium-cardinality fields like trace IDs. It is attached to log lines and filterable without creating a new stream, which closes the old gap between 'label it and explode' and 'bury it in text'.

LogQL: filter, parse, then aggregate

A LogQL query has two halves. The stream selector, in braces, is mandatory and is the only part that uses the index. Everything after it is a pipeline applied to the matching lines.

LogQL query anatomy

# 1. Stream selector - always narrow here first
{namespace="checkout", app="checkout-api"}

# 2. Line filters - fastest pipeline stage, applied before parsing.
#    Put the most selective one first.
{namespace="checkout", app="checkout-api"} |= "error" != "health"

#    |=  contains          !=  does not contain
#    |~  matches regex     !~  does not match regex

# 3. Parsers - turn the line into labels you can filter on
{namespace="checkout", app="checkout-api"} | json
{namespace="checkout", app="checkout-api"} | logfmt
{namespace="checkout", app="checkout-api"} | pattern `<_> <method> <path> <status> <duration>`
{namespace="checkout", app="checkout-api"} | regexp `duration=(?P<duration>[0-9.]+)`

# 4. Label filters - after parsing
{namespace="checkout", app="checkout-api"}
  |= "payment"
  | json
  | error_code = "card_declined"
  | attempt > 1

# 5. Formatting
{namespace="checkout", app="checkout-api"}
  | json
  | line_format "{{.trace_id}} {{.error_code}} order={{.order_id}}"

Metrics from logs

LogQL can turn a log stream into a range vector, which is how you get a graph or an alert out of something you never instrumented as a metric. This is the escape hatch for third-party software you cannot modify.

LogQL metric queries

# Error lines per second by app
sum by (app) (
  rate({namespace="checkout"} |= "level=error" [5m])
)

# Extract a numeric field from the line and compute a quantile.
# Useful for software that logs latency but exposes no metrics.
quantile_over_time(0.99,
  {app="legacy-billing"}
    | logfmt
    | unwrap duration_ms [5m]
) by (route)

# Count distinct users hitting an error - a genuinely high-cardinality
# question that would be impossible as a metric, and is cheap here.
sum by (error_code) (
  count_over_time({app="checkout-api"} | json | level="ERROR" [1h])
)

# Alert on a log pattern with no corresponding metric
sum(rate({app="checkout-api"} |= "panic:" [5m])) > 0

Metric queries over logs are expensive relative to Prometheus queries because they scan raw data. Use them for exploration and for gaps you cannot instrument, but if you need a number continuously on a dashboard or in an alert, add a real metric or a Loki recording rule.

Retention, tiering, and cost

Retention configuration

# loki/config.yaml
limits_config:
  retention_period: 720h            # 30 days default for every tenant
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 32
  max_label_names_per_series: 15
  max_streams_per_user: 15000       # the cardinality circuit breaker
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  allow_structured_metadata: true

# Per-stream overrides: keep audit logs long, debug logs briefly
overrides:
  audit:
    retention_period: 8760h         # 1 year
  dev:
    retention_period: 72h           # 3 days

compactor:
  working_directory: /loki/compactor
  retention_enabled: true
  delete_request_store: s3

schema_config:
  configs:
    - from: 2026-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h
  • Sample the noise. Access logs for successful health checks are the largest volume and the least value. Drop them at the collector with a stage.drop.
  • Set log levels per environment. Debug logging in production is usually a bug, not a feature.
  • Use lifecycle policies on the bucket. Move chunks older than 30 days to infrequent-access storage.
  • Cap ingestion per tenant. One team's logging bug should not degrade everyone else's queries.

Dropping and sampling at the collector

loki.process "drop_noise" {
  forward_to = [loki.write.default.receiver]

  // Drop successful health check lines entirely
  stage.drop {
    expression = ".*GET /healthz.* 200 .*"
    drop_counter_reason = "healthcheck"
  }

  // Keep only 10% of debug lines
  stage.sampling {
    rate = 0.1
    stage.match {
      selector = `{level="debug"}`
    }
  }
}

Correlating logs with metrics and traces

The payoff for putting a trace ID in every line is that Grafana can link the three signals together. Configure a derived field on the Loki datasource and every trace ID in a log line becomes a clickable link into Tempo.

grafana/provisioning/datasources/loki-tempo.yaml

apiVersion: 1
datasources:
  - name: Loki
    type: loki
    uid: loki
    url: http://loki:3100
    jsonData:
      derivedFields:
        - name: TraceID
          # Works with structured metadata or a JSON field
          matcherType: label
          matcherRegex: trace_id
          url: "$${__value.raw}"
          datasourceUid: tempo
          urlDisplayLabel: "View trace"

  - name: Tempo
    type: tempo
    uid: tempo
    url: http://tempo:3200
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        spanStartTimeShift: "-5m"
        spanEndTimeShift: "5m"
        filterByTraceID: true
        tags:
          - key: service.name
            value: app

With this in place the incident workflow becomes: burn-rate alert fires, open the SLO dashboard, click an exemplar on the latency spike, land in the trace, click through to the logs for that exact request. No copy-pasting request IDs between four browser tabs at 3 a.m.

Hands-on: labs/08-logs in the companion repo runs Loki with Alloy, a JSON-logging demo app, and the derived-field configuration above, so the metric-to-trace-to-log path works end to end on a laptop.

Key takeaways

  • Loki indexes labels only. That single design choice makes it cheap and makes label design critical.
  • A Loki stream is a unique label combination. Keep them bounded, in the low tens of thousands per tenant.
  • Put unbounded fields - request IDs, user IDs, error text - in the line content or structured metadata, never in labels.
  • Emit structured JSON with consistent field names, and always include trace_id.
  • Always narrow with the stream selector first, then line filters, then parsers. Query cost drops by orders of magnitude.
  • LogQL metric queries are the escape hatch for software you cannot instrument, not a replacement for metrics.
  • Sample and drop noise at the collector, and set per-tenant retention and ingestion caps.
  • Derived fields in Grafana turn a trace ID in a log line into a one-click jump to the trace.

Quick answers

Should I use Loki or Elasticsearch?

Loki suits teams whose queries almost always start from a known service, namespace, and time range, and who care about storage cost - the common case for Kubernetes platform logging. Elasticsearch suits full-text search across large corpora with unpredictable query patterns, and security analytics where you genuinely need to search everything. Many organisations run Loki for application logs and something else for security data.

What belongs in a Loki label versus the log line?

Labels must be bounded and low cardinality: cluster, namespace, app, container, level, environment. Anything unbounded - request IDs, user IDs, order IDs, error messages - goes in the line content or, from Loki 3.0, in structured metadata, which is filterable without creating a new stream.

Why is my LogQL query so slow?

Almost always because the stream selector is too broad, so Loki is decompressing and scanning far more chunks than necessary. Narrow the labels first, add the most selective line filter before any parser, and shorten the time range. Parsing before filtering is the most common expensive mistake.

Can I alert on logs?

Yes, with LogQL metric queries evaluated by the Loki ruler, which speaks the same rule format as Prometheus. It is the right tool for patterns you cannot instrument, such as a panic string from a third-party binary. For anything you control, a real metric is cheaper and more reliable.