Building your own LGTM stack

What LGTM actually is

LGTM is Grafana Labs' observability stack, and the acronym is a backronym that happens to be convenient. All four components share the same architecture: stateless microservices, object storage as the only durable dependency, and horizontal scalability by component.

In front of them sits a collector — Grafana Alloy or the OpenTelemetry Collector — that scrapes, receives, processes, and routes everything. Alloy is Grafana's distribution and speaks all four backends natively; the OTel Collector is the vendor-neutral choice. Both work; pick one and standardise on it.

The whole stack on a laptop

This compose file gives you a working end-to-end pipeline: a demo app emitting metrics, logs and traces, a collector, all four backends, and Grafana pre-provisioned with datasources and the correlation links from Parts 8 and 9.

docker-compose.yaml

services:
  minio:
    image: minio/minio:latest
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: minio123
    ports: ["9000:9000", "9001:9001"]
    volumes: ["minio-data:/data"]
    healthcheck:
      test: ["CMD", "mc", "ready", "local"]
      interval: 5s
      retries: 12

  createbuckets:
    image: minio/mc:latest
    depends_on:
      minio: { condition: service_healthy }
    entrypoint: >
      /bin/sh -c "
      mc alias set local http://minio:9000 minio minio123 &&
      mc mb -p local/mimir local/loki local/tempo &&
      exit 0"

  mimir:
    image: grafana/mimir:latest
    command: ["-config.file=/etc/mimir/mimir.yaml", "-target=all"]
    volumes:
      - ./mimir/mimir.yaml:/etc/mimir/mimir.yaml:ro
      - mimir-data:/data
    ports: ["9009:9009"]
    depends_on: [createbuckets]

  loki:
    image: grafana/loki:latest
    command: ["-config.file=/etc/loki/loki.yaml"]
    volumes:
      - ./loki/loki.yaml:/etc/loki/loki.yaml:ro
      - loki-data:/loki
    ports: ["3100:3100"]
    depends_on: [createbuckets]

  tempo:
    image: grafana/tempo:latest
    command: ["-config.file=/etc/tempo/tempo.yaml"]
    volumes:
      - ./tempo/tempo.yaml:/etc/tempo/tempo.yaml:ro
      - tempo-data:/var/tempo
    ports: ["3200:3200"]
    depends_on: [createbuckets]

  alloy:
    image: grafana/alloy:latest
    command:
      - run
      - --server.http.listen-addr=0.0.0.0:12345
      - --storage.path=/var/lib/alloy/data
      - /etc/alloy/config.alloy
    volumes:
      - ./alloy/config.alloy:/etc/alloy/config.alloy:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - alloy-data:/var/lib/alloy/data
    ports:
      - "12345:12345"   # Alloy UI - genuinely useful for debugging pipelines
      - "4317:4317"     # OTLP gRPC
      - "4318:4318"     # OTLP HTTP
    depends_on: [mimir, loki, tempo]

  grafana:
    image: grafana/grafana:latest
    environment:
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
      GF_FEATURE_TOGGLES_ENABLE: traceqlEditor,tracesEmbeddedFlameGraph
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
    ports: ["3000:3000"]
    depends_on: [mimir, loki, tempo]

volumes:
  minio-data:
  mimir-data:
  loki-data:
  tempo-data:
  alloy-data:

One collector configuration for all three signals

alloy/config.alloy

// alloy/config.alloy

// ---------- METRICS ----------
prometheus.exporter.unix "node" { }

discovery.docker "containers" {
  host = "unix:///var/run/docker.sock"
}

prometheus.scrape "everything" {
  targets = concat(
    prometheus.exporter.unix.node.targets,
    discovery.docker.containers.targets,
  )
  scrape_interval = "15s"
  forward_to = [prometheus.relabel.cardinality_guard.receiver]
}

// Cardinality control at the collector - Part 7 in practice
prometheus.relabel "cardinality_guard" {
  forward_to = [prometheus.remote_write.mimir.receiver]

  rule {
    source_labels = ["__name__"]
    regex  = "go_gc_duration_seconds.*|promhttp_.*"
    action = "drop"
  }
  rule {
    regex  = "container_label_.*|annotation_.*"
    action = "labeldrop"
  }
}

prometheus.remote_write "mimir" {
  endpoint {
    url = "http://mimir:9009/api/v1/push"
    headers = { "X-Scope-OrgID" = "demo" }
  }
}

// ---------- LOGS ----------
loki.source.docker "containers" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.docker.containers.targets
  forward_to = [loki.process.parse.receiver]
}

loki.process "parse" {
  forward_to = [loki.write.loki.receiver]

  stage.json {
    expressions = { level = "level", trace_id = "trace_id" }
  }
  stage.labels {
    values = { level = "" }          // bounded -> label
  }
  stage.structured_metadata {
    values = { trace_id = "" }       // unbounded -> structured metadata
  }
}

loki.write "loki" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
    headers = { "X-Scope-OrgID" = "demo" }
  }
}

// ---------- TRACES ----------
otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }

  output {
    traces  = [otelcol.processor.batch.default.input]
    metrics = [otelcol.exporter.prometheus.to_mimir.input]
    logs    = [otelcol.exporter.loki.to_loki.input]
  }
}

otelcol.processor.batch "default" {
  send_batch_size = 1000
  timeout         = "5s"
  output {
    traces = [otelcol.exporter.otlp.tempo.input]
  }
}

otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo:4317"
    tls { insecure = true }
  }
}

otelcol.exporter.prometheus "to_mimir" {
  forward_to = [prometheus.remote_write.mimir.receiver]
}

otelcol.exporter.loki "to_loki" {
  forward_to = [loki.write.loki.receiver]
}

Wiring the correlations

The stack is only worth the effort if the signals link to each other. This provisioning file connects all three directions: metric exemplar to trace, trace to logs, and log to trace.

grafana/provisioning/datasources/datasources.yaml

apiVersion: 1
datasources:
  - name: Mimir
    type: prometheus
    uid: mimir
    url: http://mimir:9009/prometheus
    isDefault: true
    jsonData:
      httpHeaderName1: X-Scope-OrgID
      exemplarTraceIdDestinations:
        - name: trace_id
          datasourceUid: tempo
    secureJsonData:
      httpHeaderValue1: demo

  - name: Loki
    type: loki
    uid: loki
    url: http://loki:3100
    jsonData:
      httpHeaderName1: X-Scope-OrgID
      derivedFields:
        - name: TraceID
          matcherType: label
          matcherRegex: trace_id
          url: "$${__value.raw}"
          datasourceUid: tempo
    secureJsonData:
      httpHeaderValue1: demo

  - name: Tempo
    type: tempo
    uid: tempo
    url: http://tempo:3200
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        spanStartTimeShift: "-5m"
        spanEndTimeShift: "5m"
        filterByTraceID: true
      tracesToMetrics:
        datasourceUid: mimir
        queries:
          - name: "Request rate"
            query: 'sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))'
      serviceMap:
        datasourceUid: mimir
      nodeGraph:
        enabled: true

Moving it to Kubernetes

The compose stack teaches you the moving parts. For anything beyond a laptop, use the official Helm charts and run each backend in a scalable mode rather than monolithic.

Helm installation

# Add the repo once
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Mimir - distributed mode
helm upgrade --install mimir grafana/mimir-distributed \
  --namespace observability --create-namespace \
  --values values/mimir.yaml

# Loki - simple scalable mode: read, write and backend targets.
# Choose this over microservices mode until you have a reason not to.
helm upgrade --install loki grafana/loki \
  --namespace observability \
  --set deploymentMode=SimpleScalable \
  --values values/loki.yaml

# Tempo - distributed
helm upgrade --install tempo grafana/tempo-distributed \
  --namespace observability \
  --values values/tempo.yaml

# Alloy as a DaemonSet for node and pod telemetry
helm upgrade --install alloy grafana/alloy \
  --namespace observability \
  --values values/alloy.yaml

helm upgrade --install grafana grafana/grafana \
  --namespace observability \
  --values values/grafana.yaml

values/mimir.yaml

# values/mimir.yaml - the settings that matter most
mimir:
  structuredConfig:
    limits:
      # The cardinality circuit breakers. Set them before you need them.
      max_global_series_per_user: 5000000
      max_global_series_per_metric: 200000
      ingestion_rate: 250000
      ingestion_burst_size: 500000
      max_label_names_per_series: 30
      compactor_blocks_retention_period: 8760h   # 1 year

    common:
      storage:
        backend: s3
        s3:
          endpoint: s3.eu-west-1.amazonaws.com
          bucket_name: mimir-prod
          region: eu-west-1

# Multi-tenancy: each team gets its own X-Scope-OrgID and its own
# limits, so one team's cardinality incident cannot take out the others.
multitenancy_enabled: true

ingester:
  replicas: 3
  zoneAwareReplication:
    enabled: true
  persistentVolume:
    size: 100Gi

querier:
  replicas: 4

store_gateway:
  replicas: 3
  zoneAwareReplication:
    enabled: true

Sizing and cost

Rough figures for planning. Measure your own workload before committing to hardware, but these get you within the right order of magnitude.

• Object storage is the cheap part. Compute for ingesters and queriers dominates the bill; size those carefully and let storage grow.
• Set retention per signal. Metrics 12 months, logs 30 days, traces 7–14 days is a common and defensible split.
• Downsample old metrics. Recording rules that aggregate away instance-level labels cut long-term series count by an order of magnitude.
• Enable the query result cache. Dashboard queries repeat constantly and caching them is close to free.
• Set per-tenant limits from day one. They are the only thing standing between one team's bad deploy and a platform-wide outage.

Quick answers