Distributed tracing that answers 'where did the time go'

The question metrics cannot answer

Your p99 checkout latency went from 300ms to 2.4s. Metrics tell you that, and they tell you which service's own latency rose. What they cannot tell you is the shape of a single slow request: that it spent 40ms in the API gateway, 30ms in auth, then 2.1 seconds waiting on a connection to the inventory service because a connection pool of 10 was serving 400 concurrent requests.

That is what tracing is for. A trace is the complete causal record of one request as it moves through your system, assembled from spans emitted independently by each service.

OpenTelemetry is the part worth committing to

Vendor-specific tracing agents were a lock-in trap for a decade. OpenTelemetry is the CNCF standard that ends it: one set of APIs and SDKs, one wire protocol (OTLP), and a collector that can fan out to any backend. Instrument once, change backends by editing a config file.

Tracer provider setup

// Go: minimal but complete OTel setup
package main

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

func initTracing(ctx context.Context) (func(context.Context) error, error) {
    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("alloy:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }

    // Resource attributes identify the SERVICE and are attached to
    // every span. service.name is mandatory - without it your spans
    // land in a bucket called "unknown_service".
    res, err := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceName("checkout-api"),
            semconv.ServiceVersion(buildVersion),
            semconv.DeploymentEnvironment("production"),
        ),
        resource.WithHost(),
        resource.WithProcessRuntimeDescription(),
    )
    if err != nil {
        return nil, err
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(res),
        // Head sampling decision - see the sampling section below
        sdktrace.WithSampler(
            sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1)),
        ),
    )
    otel.SetTracerProvider(tp)

    // Without this, context does not cross process boundaries and
    // you get thousands of one-span traces instead of one real trace.
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    ))

    return tp.Shutdown, nil
}

Instrumenting: auto first, manual where it matters

Auto-instrumentation libraries cover HTTP servers and clients, gRPC, SQL drivers, and most message brokers. Start there — it gets you the service map and the call-graph latency breakdown for almost no effort. Add manual spans only around business logic that auto-instrumentation cannot see.

Auto plus manual

// Auto-instrumentation covers the transport layer
import (
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

handler := otelhttp.NewHandler(mux, "checkout-api")
client := &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}

// Manual spans for the interesting internal work
func (s *Service) ApplyPromotion(ctx context.Context, order *Order) error {
    ctx, span := s.tracer.Start(ctx, "ApplyPromotion",
        trace.WithAttributes(
            // High cardinality is FINE here - this is the whole point
            attribute.String("order.id", order.ID),
            attribute.String("user.id", order.UserID),
            attribute.String("promo.code", order.PromoCode),
            attribute.Int("order.item_count", len(order.Items)),
            attribute.Float64("order.total", order.Total),
        ))
    defer span.End()

    rules, err := s.rules.Load(ctx, order.PromoCode)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "failed to load promotion rules")
        return err
    }

    span.AddEvent("rules loaded", trace.WithAttributes(
        attribute.Int("rules.count", len(rules)),
    ))

    // ctx carries the span - passing it downstream is what builds the tree
    return s.apply(ctx, order, rules)
}

Follow the semantic conventions

OpenTelemetry defines standard attribute names: http.request.method, server.address, db.system, messaging.destination.name. Using them rather than inventing your own is what makes generic dashboards, automatic service maps, and vendor tooling work without per-service configuration.

Sampling: the decision that determines your bill

Tracing every request at scale is usually unaffordable and rarely useful, because 99% of traces describe requests that worked fine. Sampling is unavoidable; the question is when the decision is made.

The configuration most teams settle on is a low head-sampling rate combined with tail sampling in the collector that keeps 100% of errors and slow traces plus a small random sample of successes.

otel-collector-config.yaml

# OpenTelemetry Collector - tail sampling
processors:
  tail_sampling:
    decision_wait: 10s        # how long to buffer a trace before deciding
    num_traces: 100000        # in-memory trace buffer
    expected_new_traces_per_sec: 2000
    policies:
      # Always keep anything that failed
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      # Always keep anything slow
      - name: slow
        type: latency
        latency:
          threshold_ms: 1000

      # Always keep traces for a service under investigation
      - name: under-investigation
        type: string_attribute
        string_attribute:
          key: service.name
          values: [checkout-api, payments-api]

      # Keep 1% of everything else, for baseline comparison
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 1

Span metrics: percentiles for free from your traces

The collector's span metrics connector generates RED metrics from span data. You get consistent request rate, error rate, and latency histograms for every instrumented service without touching application code — especially valuable for services whose owners never got around to adding Prometheus metrics.

Span metrics connector

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [5ms, 10ms, 25ms, 50ms, 100ms, 200ms, 400ms, 800ms, 1500ms, 3s, 5s, 10s]
    dimensions:
      # Keep these BOUNDED - these become Prometheus labels,
      # and Part 7 applies in full force here.
      - name: http.request.method
      - name: http.response.status_code
      - name: server.address
    exemplars:
      enabled: true          # links the histogram back to real traces

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp/tempo, spanmetrics]
    metrics/spanmetrics:
      receivers: [spanmetrics]
      exporters: [prometheusremotewrite]

Storing traces in Tempo

Tempo makes the same bet as Loki: no index beyond trace ID and a small set of searchable fields, everything in object storage. Retrieval by trace ID is very cheap, which pairs perfectly with exemplars and derived log fields — the two ways you will actually arrive at a trace in practice.

tempo.yaml

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

ingester:
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 336h        # 14 days

metrics_generator:
  registry:
    external_labels:
      source: tempo
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://mimir:9009/api/v1/push
        send_exemplars: true
  traces_storage:
    path: /var/tempo/generator/traces
  processor:
    service_graphs:
      dimensions: [http.method]
    span_metrics:
      dimensions: [http.method, http.status_code]

storage:
  trace:
    backend: s3
    s3:
      endpoint: minio:9000
      bucket: tempo
      access_key: ${S3_ACCESS_KEY}
      secret_key: ${S3_SECRET_KEY}
      insecure: true
    wal:
      path: /var/tempo/wal

overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics, local-blocks]

Tempo's metrics generator produces service graphs directly from trace data, giving you an automatic, always-current dependency map. That map is worth the setup cost on its own — it is invariably more accurate than the architecture diagram in the wiki.

TraceQL: querying traces

TraceQL

# All traces touching a service
{ resource.service.name = "checkout-api" }

# Slow traces only
{ resource.service.name = "checkout-api" && duration > 2s }

# Errors on a specific route
{ resource.service.name = "checkout-api"
  && span.http.route = "/api/v1/checkout"
  && status = error }

# Structural: traces where checkout called inventory AND that call was slow
{ resource.service.name = "checkout-api" }
  >> { resource.service.name = "inventory-api" && duration > 500ms }

# Aggregate across matching spans
{ resource.service.name = "checkout-api" } | avg(duration) > 1s

# Find a specific customer's failed orders - the high-cardinality
# question that would be impossible as a metric
{ span.order.id = "ord_8f3a91c2" }

Quick answers