Revision · Observability & SRE

This part drew the line between monitoring (answers to questions you predicted) and observability (asking new questions of a running system after it surprises you), then built up the three signals and the SRE discipline that turns them into faster time-to-restore.

What this part covered

• The three pillars answer different questions — logs, metrics, and traces each address a distinct question and fail in a distinct way, share a cardinality–cost trade-off, and increasingly unify under OpenTelemetry.
• Structured logging makes a fleet queryable — JSON over printf, sane log levels, an agent-to-store aggregation pipeline, correlation IDs to stitch one request together, and a hard rule about the secrets and PII you must never log.
• Metrics are numbers over time — the time-series model, counters vs gauges vs histograms, Prometheus’s pull model and PromQL, the cardinality explosion to avoid, and the Four Golden Signals (plus RED/USE) that say whether a service is healthy.
• Tracing pinpoints the slow hop — spans in a parent/child tree, context propagation that carries IDs across services, the OpenTelemetry SDK and Collector, head- vs tail-based sampling, and reading a span waterfall to find where the latency actually is.
• SLOs turn signals into decisions — SLI vs SLO vs SLA, choosing good SLIs, the error budget as 1 − SLO, and burn-rate alerting that settles the speed-versus-stability fight with a number instead of an argument.
• Incident response is a system, not heroics — always-on-call ownership, severity levels matched to impact, one incident commander, MTTD/MTTR as the clocks you race, and blameless postmortems whose action items are what actually fix the system.

The takeaway

Pages 2–5 gave you the signals — the raw material of seeing your system — and pages 6–7 gave you the discipline that turns them into decisions about how fast to move and how to respond when something breaks. Together they replace the SSH-and-guess debugging session with evidence, collapsing time-to-restore, which is the DORA metric observability most directly improves. With production now visible and reliability made measurable, the next part hardens the pipeline itself: Security (DevSecOps).