Part 7 · Observability & SRE

You have learned how to build a container image, schedule it on a Kubernetes cluster, and ship it through a CI/CD pipeline using a deployment strategy that limits the blast radius. There is one assumption buried in all of that which we have not yet earned: that you can see what your system is doing. A canary deploy is only safe if you can tell, in seconds, whether the canary is healthy. A rollback is only useful if you know there is something to roll back from. The whole apparatus of safe delivery rests on the ability to observe production — and that ability does not come for free.

This Part is about that ability. It is the other half of the DORA bargain: the metrics measure whether you can recover quickly, and you cannot recover from what you cannot see. Time to restore service is mostly time spent finding out what broke.

Monitoring is not observability

These two words get used interchangeably and they are not the same thing. The distinction is the spine of this entire Part, so get it straight now.

• Monitoring answers questions you thought of in advance. You decide “I care about CPU, error rate, and p99 latency,” you build a dashboard for them, and you set alerts. It is excellent at the failures you predicted. Its blind spot is everything you didn’t.
• Observability is a property of the system: can you ask a new question — one you never pre-built a dashboard for — and get an answer from data the system already emitted? “Show me the slow requests, but only for users on the new checkout flow, in the EU region, since the 14:02 deploy.” If you can answer that without shipping new code, your system is observable.

   MONITORING                          OBSERVABILITY
   "known unknowns"                    "unknown unknowns"
   ──────────────────                  ─────────────────────────────
   predefined dashboards               ask arbitrary questions later
   "is CPU above 80%?"                 "why is THIS request slow?"
   alerts you wrote in advance         slice by any dimension, after the fact
   answers questions you predicted     answers questions you didn't

Monitoring is a subset of observability, not a rival to it. You still want the dashboards and the alerts — but in a system of any real complexity, the failure that pages you at 3am will almost never be one you predicted. The whole game is being able to interrogate production after it surprises you.

A working definition

A system is observable if you can understand its internal state from its external outputs — its logs, metrics, and traces — without deploying new instrumentation to answer a question you only just thought of. The bar is “new question, no new code.”

Why distributed systems force the shift

A single program on one machine barely needs observability. Attach a debugger, read the stack trace, print a variable. But you no longer run single programs on single machines. A user request now fans out across a dozen services, each in its own Pod, possibly on different nodes, talking over the network, behind a load balancer, each scaled to many replicas that come and go.

In that world, “why is checkout slow?” has no single place to look. The slowness might be in any of twelve hops, or in the network between them, or in a database one of them calls, and the Pod that was slow may have been rescheduled and deleted before you logged in. You cannot attach a debugger to a request that crossed twelve machines. The only way to understand the system is from the data it emitted while it ran. Distributed systems don’t make observability nice to have; they make it the only way to debug at all.

The thread for this Part

The recurring question of this book — what manual, error-prone step does this remove, and how does it make production safer? — has a specific answer for observability. The manual step it removes is the SSH-and-guess debugging session: logging into a box, tailing files by hand, restarting things to see what happens, forming theories from intuition under pressure. Observability replaces guessing with asking — turning “I think it’s the database” into “the trace shows the database call took 1.8 seconds.” And it makes production safer in the most direct way the DORA data measures: it collapses time to restore, because you spend that time diagnosing with evidence instead of hunting in the dark.

Roadmap for this Part

Read these in order — each builds on the last.

#	Page	What it answers
2	The Three Pillars	Logs, metrics, traces — what question each answers and where each fails
3	Logging	Structured logs, the aggregation pipeline, correlation IDs, what not to log
4	Metrics	Time series, Prometheus, PromQL, the Four Golden Signals, alerting
5	Tracing	Spans, context propagation, sampling — pinpointing the slow hop
6	SLOs & Error Budgets	SLIs, error budgets, burn-rate alerts, and what SRE actually is
7	Incident Response & Postmortems	On-call, severity, blameless postmortems, MTTR

The shape of the journey: pages 2–5 are the signals — the raw material of seeing your system. Pages 6–7 are the discipline — how Site Reliability Engineering turns those signals into decisions about how fast to move and how to respond when something breaks. By the end you should be able to take any production mystery and say, concretely, which signal you’d reach for and what question you’d ask of it.

We start with the three signals themselves — and why no one of them is enough.

Check your understanding

1. In one sentence each, distinguish monitoring from observability. Why is monitoring best described as a subset of observability rather than its opposite?
2. The brief’s litmus test for observability is “new question, no new code.” Explain what that bar is actually testing for, and why it matters during an incident.
3. Why does a single program on one machine barely need observability, while a distributed system requires it? Name the specific thing about distributed systems that breaks traditional debugging.
4. Using the recurring thread, name the manual debugging step observability removes and the DORA metric it most directly improves.
5. Why is it fair to say that a canary deploy is only as safe as your observability? What can’t you do without it?

Show answers

1. Monitoring answers questions you defined in advance (predefined dashboards and alerts); observability is the ability to ask new questions of a running system after the fact. It’s a subset because dashboards and alerts are one (predefined) way of querying the same underlying signals — you still want them, they just can’t cover the failures you didn’t predict.
2. It tests whether the system already emits enough, richly-labelled data to answer a question you never anticipated — without shipping new instrumentation and waiting for it to deploy. During an incident you can’t afford that round-trip; the answer has to be derivable from data already collected.
3. On one machine you can attach a debugger, read a stack trace, or print a variable. A distributed request fans out across many services, Pods, and network hops, and the Pod that misbehaved may be gone before you log in — you can’t attach a debugger to a request that crossed twelve machines, so the only evidence is the data emitted while it ran.
4. It removes the SSH-and-guess debugging session (log in, tail files, restart things, theorise under pressure) and most directly improves time to restore service, because diagnosis with evidence is far faster than hunting in the dark.
5. A canary only buys safety if you can tell whether the canary cohort is healthier or sicker than the baseline — which requires comparing their signals. Without observability you can’t see the canary failing, so you can’t make the go/rollback decision the strategy depends on.