The Three Pillars

The overview argued that in a distributed system you can only understand production from the data it emitted while it ran. This page is about what data. There are three kinds of telemetry — logs, metrics, and traces — and they are usually called the “three pillars” of observability. The phrase is a little misleading, because it suggests three interchangeable supports. They are not interchangeable. Each answers a fundamentally different question, and each is useless at the others’ jobs. The skill is knowing which one to reach for.

Start with the questions, not the data formats. That is the only way to remember which is which.

The three questions

   METRICS   ──►  "Is something wrong, and how bad?"        (aggregate, cheap, always on)
   TRACES    ──►  "Where in the request path is it wrong?"  (one request, across services)
   LOGS      ──►  "What exactly happened at that point?"    (the detailed event record)

A healthy debugging session usually moves through all three, in that order:

1. A metric alert fires — error rate jumped, p99 latency doubled. You know something is wrong and roughly how bad, but not where.
2. You pull a trace of a slow or failing request and see it spent 1.8 seconds in the payments service. Now you know where.
3. You read the logs from payments around that timestamp and see connection pool exhausted. Now you know what.

Metrics tell you that and how much; traces tell you where; logs tell you what. Skip a pillar and you’re stuck: metrics alone can’t tell you which of twelve services is the culprit; logs alone bury you in millions of lines with no way to know which matter; traces alone show you the slow hop but not the exception message inside it.

What each one is

Pillar	Shape of the data	Answers	Strength	Where it fails
Metrics	Numbers over time (a time series)	“How much / how many / how fast, in aggregate?”	Cheap, always-on, great for alerting and dashboards	No detail — a number can’t tell you why
Logs	Timestamped event records (text or JSON)	“What exactly happened in this one event?”	Maximum detail per event	Expensive at volume; hard to aggregate; easy to drown in
Traces	A request’s path across services as linked spans	”Where in the call graph did time go?”	Shows the whole distributed request end to end	Sampled (you don’t keep them all); needs every service instrumented

We give each its own page next — Logging, Metrics, Tracing — so here we only need the shape of each and how they fit together.

They are derivable from each other, in theory

A metric is what you get if you count logs over a window. A trace is what you get if you stitch together the logs of one request across services using a shared ID. In principle all three are views of the same event stream. In practice you collect them differently because the cost and query patterns are wildly different — you want metrics cheap and always-on, but you can’t afford to keep a full log line for every event forever.

The cardinality–cost trade-off

Here is the single economic fact that governs all three pillars, and it is worth understanding deeply because it explains nearly every design decision in this Part.

Cardinality is the number of distinct values a dimension can take. region has low cardinality (maybe 5 values). user_id has enormous cardinality (millions). request_id is effectively infinite. The cost of telemetry scales with cardinality — and each pillar pays that cost differently:

                 cardinality you can afford        cost driver
   METRICS   ──► LOW   (tens to thousands)         every label combo = a new time series, stored forever
   TRACES    ──► HIGH  (per-request, but SAMPLED)  cost controlled by keeping only a fraction
   LOGS      ──► HIGH  (per-event, full detail)    cost controlled by volume, retention, and what you index

This is why you cannot just “put everything in metrics.” Adding a high-cardinality label like user_id to a metric creates a separate time series for every user — this is the infamous cardinality explosion, and it can take down your metrics backend (more in Metrics). High-cardinality detail belongs in logs and traces, which are designed to carry it, and which control cost by sampling and retention instead of by limiting dimensions.

The trap of “just log everything” / “just graph everything”

Both failure modes are real. Teams that log everything at full volume get a five-figure monthly bill and a query system too slow to use in an incident. Teams that try to put every dimension in metrics blow up cardinality and crash Prometheus. The discipline is matching the question to the pillar: aggregate questions to metrics, per-request questions to traces, forensic detail to logs.

OpenTelemetry: one standard for all three

For years, each pillar had its own incompatible ecosystem — a logging library, a separate metrics client, a separate tracing SDK, each speaking a different wire format, each locking you into a vendor. Instrumenting a service meant three integrations, and switching backends meant re-instrumenting everything.

OpenTelemetry (OTel) is the industry response: a single, vendor-neutral standard — SDKs plus a wire protocol (OTLP) plus a Collector — that emits all three signals through one pipeline. You instrument your code once against the OTel API, and the Collector routes logs, metrics, and traces to whatever backends you choose (Prometheus, Loki, Jaeger, a vendor — anything that speaks OTLP).

   your service                 OTel Collector              backends
   ────────────                 ──────────────              ────────
   OTel SDK ──► metrics ─┐
            ──► traces  ─┼──► receive → process → export ──► Prometheus / Loki / Jaeger / vendor
            ──► logs    ─┘      (one pipeline, all three)

The value is the same value containers and Terraform bought elsewhere in this book: a standard interface decouples what you instrument from where it’s stored. You can swap the backend without touching application code.

Under the hood — where OpenTelemetry came from

OTel didn’t appear from nowhere; it’s the merger of two earlier, competing standards. OpenTracing (a CNCF project, a vendor-neutral tracing API) and OpenCensus (a Google-originated set of metrics-and-tracing libraries) solved overlapping problems in incompatible ways — so the industry was fragmented by the very projects trying to unify it. In 2019 the two merged into OpenTelemetry under the CNCF, explicitly to end that split: one API, one wire protocol (OTLP), one Collector for all three signals.

The lesson is about standards, not tooling: a standard only delivers the decoupling it promises if the ecosystem actually converges on one of them. Two rival “open standards” are just two lock-ins wearing a nicer label. OTel’s real value is that it won the consolidation — which is why instrumenting against it today is a safe bet in a way that betting on either predecessor in 2018 was not.

The thread: from guessing to asking

The manual, error-prone step the three pillars remove together is forming a theory and then SSHing in to test it by hand. Without telemetry, a slow checkout becomes a guessing game: restart the database, see if it helps; tail a log file and squint; ask in chat whether anyone changed anything. With the three pillars, the same investigation is a query: the metric says error rate is up, the trace says it’s the payments hop, the log says the connection pool is exhausted. You replaced three rounds of guess-and-check with three queries — and production is safer because every one of those queries is faster than a restart and carries no risk of making things worse.

Next we go pillar by pillar, starting with the oldest and most detailed of the three: Logging.

The architect’s lens

Five questions for the three-signal model as a whole:

• Why does it exist? Because no single signal answers every question — metrics, traces, and logs each answer a different one (“how bad?”, “where?”, “what exactly?”), and skipping a pillar strands your investigation.
• What problem does it solve? Guess-and-restart debugging: the three together turn an incident into three queries — the metric flags it, the trace finds the hop, the log says the connection pool is exhausted.
• What are the trade-offs? All three share the cardinality–cost trade, paid differently: metrics must stay low-cardinality (a user_id label explodes the series count), while traces control cost by sampling and logs by volume and retention; and OpenTelemetry only delivers backend-decoupling because it won the OpenTracing/OpenCensus consolidation.
• When should I avoid it? You rarely drop a pillar wholesale, but a tiny single-service tool may not need full distributed tracing — match the pillar to the questions you actually ask.
• What breaks if I collapse them into one? “Log everything” yields a five-figure bill and a query system too slow to use in an incident; “graph everything” blows up Prometheus’s cardinality — each pillar fails at the others’ jobs.

Check your understanding

1. State the one-sentence question each pillar answers. In a typical incident, in what order do you reach for them, and why that order?
2. Why can’t metrics alone debug a distributed slowdown? Why can’t logs alone? What does each lack that the other supplies?
3. Define cardinality. Why does adding a user_id label to a metric cause an explosion, while putting user_id in a log line or trace is fine?
4. Each pillar controls cost a different way. Match each pillar to its cost-control lever (dimension limits, sampling, retention/volume).
5. What does OpenTelemetry actually standardise, and what does that decoupling let you change without touching application code? Name another tool earlier in the book that bought you the same kind of decoupling.

Show answers

1. Metrics: “is something wrong, and how bad?” Traces: “where in the request path is it wrong?” Logs: “what exactly happened there?” You go metrics → traces → logs because you first detect a problem in aggregate, then narrow to the offending hop, then read the detailed event — each step shrinks the search space for the next.
2. Metrics are aggregates with no per-request detail, so they can tell you error rate is up but not which of twelve services caused it. Logs have full detail but are millions of unconnected lines with no aggregate view, so you can’t tell which ones matter. Traces supply the where; logs supply the what.
3. Cardinality is the number of distinct values a dimension can take. A metric stores a separate time series for every label-value combination, so a high-cardinality label like user_id (millions of values) creates millions of series and overwhelms the backend. Logs and traces are per-event records designed to carry arbitrary detail; they control cost by sampling and retention, not by limiting dimensions.
4. Metrics → limit dimensions (keep cardinality low). Traces → sampling (keep a fraction). Logs → volume and retention (and what you index).
5. OpenTelemetry standardises the instrumentation API and the wire protocol (OTLP) for all three signals, plus a Collector to route them. That lets you swap the backend (Prometheus, Loki, Jaeger, a vendor) without re-instrumenting application code — the same decoupling containers or Terraform gave you between a workload/resource and where it runs.