Tracing

Metrics told you that the system is slow and roughly how badly. Logs told you what happened at one point. Neither answers the question a distributed system makes hardest: where, across a dozen services, did the time actually go? That is what distributed tracing is for — the pillar that follows a single request through every hop it takes and shows you the path.

The problem is the one the overview opened with. A user clicks “checkout.” That request fans out through the gateway, orders, payments, inventory, and a database — five hops, possibly on five machines. Checkout is slow. Which hop? A metric shows aggregate p99 latency but not which service owns it. Logs show events in each service but you have to manually correlate timestamps across five log streams. Tracing solves this directly: it stitches the whole request into one object you can read top to bottom.

Spans and the trace tree

A trace is one request’s complete journey. It is made of spans — each span is a single unit of work (one service handling the request, one database call, one function) with a start time, a duration, and a parent.

   trace_id = 4bf92f...   (the whole request)
   ───────────────────────────────────────────────
   span: gateway          parent: none      ◄── the root span
     └─ span: orders      parent: gateway
          ├─ span: payments    parent: orders
          │    └─ span: db.query  parent: payments
          └─ span: inventory     parent: orders

Each span carries:

• a trace ID (shared by every span in the request — this is the same ID you put in correlation-ID logging),
• a span ID (unique to this span),
• a parent span ID (which span called it — this is what builds the tree),
• a start timestamp and duration,
• attributes (key–value tags: http.method, db.statement, user.id) — and because traces are high-cardinality by design, this is the right place for per-request detail.

The parent/child links are the whole trick. They turn a flat list of spans into a tree that mirrors the actual call graph — so you can see not just that payments was slow, but that it was slow because of the db.query it made.

Context propagation: how the IDs travel

For the tree to assemble, every service must know the trace ID and its parent span ID — and those have to travel with the request across process and network boundaries. That mechanism is context propagation: the calling service injects the trace context into the outgoing request’s headers, and the called service extracts it.

The standard is W3C Trace Context, an HTTP header called traceparent:

   traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
                │  └──────── trace id ───────────┘ └── parent span ─┘ │
                version                                              flags

   gateway ──HTTP──► orders ──HTTP──► payments ──HTTP──► db
      inject traceparent   extract + inject     extract + inject

Every hop extracts the incoming context, starts its own span as a child, and injects the updated context into anything it calls downstream. Break this chain anywhere — one service that doesn’t propagate the header — and the trace splits in two: you get an orphan sub-tree and lose the connection across that hop. Complete propagation is the single hardest operational requirement of tracing.

Why a service mesh makes tracing easier

A service mesh puts a sidecar proxy beside every service, and the proxy can propagate trace headers and even emit spans for the network hops without the application doing anything. It can’t see inside your code (it doesn’t know your db.query took 1.8s), but it gives you the service-to-service skeleton of every trace for free.

OpenTelemetry: SDK and Collector

As the three-pillars page noted, OpenTelemetry (OTel) is the vendor-neutral standard, and tracing is where it’s most mature. Two pieces:

• The SDK / auto-instrumentation runs in your service. It creates spans, manages context propagation, and exports spans over OTLP. For common frameworks (HTTP servers, database clients) much of this is automatic — you get spans for every inbound request and outbound call without writing span code.
• The Collector runs as separate infrastructure. It receives spans from every service, batches and processes them (including sampling, below), and exports to a tracing backend like Jaeger, Tempo, Zipkin, or a vendor.

   service A ─OTLP─┐
   service B ─OTLP─┼──► OTel Collector ──► (sample) ──► Jaeger / Tempo / vendor
   service C ─OTLP─┘     receive · batch · sample · export

The decoupling is the same win as everywhere else in this book: instrument once against the OTel API, and swap the backend without touching application code.

Sampling: you cannot keep every trace

A high-traffic service generates millions of traces an hour, each with many spans. Storing all of them is prohibitively expensive and almost entirely redundant — 99.9% of traces are boring, successful, fast requests. Sampling keeps a representative or interesting subset. There are two fundamentally different strategies, and the difference matters:

	Head-based sampling	Tail-based sampling
When the decision is made	At the start of the request, before you know the outcome	After the whole trace finishes, when you know the outcome
Basis	Random (e.g. keep 1%)	The trace’s properties — keep it if it errored or was slow
Cost	Cheap; decision is local and immediate	Expensive; the Collector must buffer all spans until the trace completes
Risk	You randomly drop the slow/failed traces you most wanted	More infrastructure, more memory in the Collector

   HEAD-BASED                          TAIL-BASED
   ──────────                          ──────────
   request starts                      request starts
   flip a coin → keep 1%               keep ALL spans buffered
   (decide before outcome known)       request finishes
   ✗ may drop the one slow request     was it slow or errored? → KEEP
                                       was it fast and fine?   → DROP
                                       ✓ keeps the interesting ones

Head-based is simple and cheap but blind — it might throw away the exact failing trace you needed. Tail-based deliberately keeps the errors and the slow tail (the ones worth debugging) at the cost of buffering every span until the request ends. Many real setups combine them: keep all errors (tail) plus a small random baseline (head) for healthy traffic.

Sampling is why “I can’t find my trace” happens

If you sample at 1% head-based, the specific slow request a user complained about probably wasn’t kept — its trace simply doesn’t exist. This is the most common tracing frustration. Tail-based sampling on “errored OR slow” is the usual cure, because the traces you go looking for are exactly the ones it keeps.

Reading a span waterfall

The payoff. A tracing UI renders a trace as a waterfall: spans laid out on a time axis, indented by parent/child, each bar’s width its duration. The slow hop is visually obvious.

   trace_id 4bf92f...                 0ms        500ms       1000ms      1500ms   2000ms
   ─────────────────────────────────────────────────────────────────────────────────────
   gateway              ████████████████████████████████████████████████████████████  1980ms
     orders             ░░██████████████████████████████████████████████████████████  1920ms
       inventory        ░░░░████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░    60ms
       payments         ░░░░░░░░██████████████████████████████████████████████████████  1850ms
         db.query       ░░░░░░░░░░░░██████████████████████████████████████████████████  1800ms ◄──

You don’t have to read a word. gateway took ~2s, almost all of it inside orders → payments → db.query, and db.query alone is 1800ms. The slow hop is the database call. inventory ran in 60ms and is irrelevant. A trace turns “checkout is slow somewhere” into “the database query in payments is the 1.8-second culprit” — in one glance, no cross-referencing five log streams by timestamp.

Under the hood — Dapper, the paper that started this

Distributed tracing as you’ve learned it traces to one source: Dapper, Google’s internal tracing system, described in a 2010 paper (“Dapper, a Large-Scale Distributed Systems Tracing Infrastructure”). Almost every modern tracer — Zipkin (from Twitter), Jaeger (from Uber), and the tracing half of OpenTelemetry — descends from its ideas.

Two of Dapper’s design decisions are exactly the ones this page spent its time on, because they were the hard parts then too. First, context propagation: Dapper threaded the trace context through Google’s internal RPC framework so spans from different machines could be stitched into one tree without application code carrying it by hand. Second, sampling: Google found that recording every request was both too expensive and unnecessary, since a small sampled fraction already captures the latency picture of a high-traffic service. The head-versus-tail trade-off you weighed above is the same one Dapper identified back in 2010 — keep overhead low without losing the traces that actually matter.

The thread: from cross-referencing timestamps to reading one picture

The manual, error-prone step tracing removes is manually correlating clocks across services — pulling up five log streams, lining up timestamps by hand, guessing which entries belong to the same request, and hoping the clocks were even in sync. With propagated trace context, that correlation is built in: every span already knows the request and its parent, and the backend assembles the waterfall for you. Production gets safer because time to restore collapses — you locate the failing hop in seconds with evidence, instead of forming a theory (“it’s probably the database”) and SSHing in to test it, which is exactly the guessing the overview promised to kill.

You now have all three signals. The remaining two pages turn signals into a discipline: how to decide how reliable a service should be — SLOs & Error Budgets — and how to respond when it isn’t.

The architect’s lens

Five questions for distributed tracing:

• Why does it exist? Because metrics say that a request was slow and logs say what happened at one point, but neither answers where, across a dozen hops, the time actually went.
• What problem does it solve? It stitches one request into a span tree via propagated traceparent context, so a waterfall shows the slow hop at a glance — “the db.query in payments is the 1.8s culprit” — instead of correlating five log streams by timestamp.
• What are the trade-offs? Complete context propagation is the single hardest requirement — one un-propagating service splits the trace — and you can’t keep every trace, so you sample: head-based is cheap but may drop the exact failing trace, tail-based keeps errored/slow ones but buffers every span.
• When should I avoid it? A single-process monolith with no fan-out gets little from distributed tracing; the payoff scales with the number of service hops.
• What breaks if I remove it? Finding where reverts to manually lining up timestamps across services (and trusting their clocks), so time-to-restore climbs and you debug by theory-and-SSH.

Check your understanding

1. What is a span, and which field turns a flat list of spans into a tree that mirrors the call graph? What does the tree let you see that a list cannot?
2. Explain context propagation. What travels, in what (e.g. traceparent), and what breaks if a single service in the chain fails to propagate it?
3. Why is sampling necessary at all? Contrast head-based and tail-based sampling on when the keep decision is made and which traces each tends to keep or lose.
4. A user reports a slow request but you “can’t find its trace.” Give the most likely reason and the sampling change that fixes it.
5. Reading the waterfall above, which hop is the culprit and how do you know? Why is this faster than correlating five log streams by timestamp?

Show answers

1. A span is a single timed unit of work (one service, one call) with a start, a duration, and a parent. The parent span ID links each span to its caller, turning the flat list into a tree that mirrors the actual call graph — so you can see not just that payments was slow but that it was slow because of the db.query it made.
2. The trace context (trace ID + current span ID) travels with the request, typically in the W3C traceparent HTTP header; each service extracts the incoming context, creates a child span, and injects the updated context downstream. If one service drops it, the trace splits — you get an orphan sub-tree and lose the connection across that hop.
3. A busy service produces millions of mostly-boring traces an hour; storing all is prohibitively expensive. Head-based decides at request start (random, cheap, but may discard the exact slow/failed trace you wanted). Tail-based decides after the request finishes based on outcome (keeps errored/slow traces, but the Collector must buffer all spans and use more memory).
4. Most likely you’re using head-based sampling at a low rate (e.g. 1%), so that specific request simply wasn’t kept and its trace doesn’t exist. Switch to tail-based sampling that keeps “errored OR slow” traces, since those are exactly the ones people go looking for.
5. The db.query in payments — it’s 1800ms of the ~1980ms total, and the bars show gateway → orders → payments → db.query nesting almost all of the time, while inventory ran in 60ms. It’s faster because the waterfall already aligns every hop on one time axis with parent/child structure; you don’t have to line up timestamps across five separate log streams (and trust their clocks) by hand.