Incident Response & Postmortems

SLOs and error budgets gave you a way to decide how reliable to be and when to slow down. But no SLO, no error budget, and no amount of tracing prevents every failure — the whole premise of designing for recovery is that failures are inevitable and the thing you control is how fast and how well you respond. This final page of the Part is about exactly that: what to do when something is currently broken, and — just as important — what to do after so it breaks less next time.

The problem incident response solves is chaos under pressure. The site is down, customers are angry, five engineers have jumped into a chat, three are independently restarting things, nobody knows who’s in charge, and the same questions are being asked four times. An unstructured response lengthens outages. Incident response is the discipline that imposes structure on that moment so you recover faster — and turns the failure into the most valuable data you’ll get all quarter.

On-call: someone is always responsible

For a system that must be up around the clock, on-call means a named engineer is responsible for responding to alerts at any hour, on a rotation. An alert (the burn-rate page from Metrics and SLOs) fires; the on-call engineer is paged; they acknowledge and begin response.

On-call is also a feedback loop on alert quality — and a humane one only if you protect it:

Severity levels: matching response to impact

Not every problem is an all-hands emergency. Severity levels (SEV) classify an incident so the response is proportional. The exact ladder varies, but the shape is universal:

Severity	Impact	Response
SEV1	Critical — major outage, data loss, most users affected	All-hands, incident commander, wake people up
SEV2	Major — significant degradation, key feature down	Urgent, dedicated response, may page
SEV3	Minor — limited impact, workaround exists	Handle in business hours
SEV4	Low — cosmetic or trivial	Backlog / ticket

Declaring a severity early does two jobs: it pulls in the right amount of help (you don’t wake the VP for a SEV3) and it sets expectations for everyone watching. Under-classifying wastes the recovery clock; over-classifying burns people out. The severity should track user impact, which is exactly what your SLIs measure.

The incident commander: one brain, not five

The single most effective structural fix for incident chaos is the incident commander (IC) role, borrowed from emergency services. During a declared incident, one person is the IC — and crucially, the IC coordinates rather than fixes.

   WITHOUT an IC                       WITH an IC
   ─────────────                       ──────────
   5 engineers, all "fixing"           IC: coordinates, decides, communicates
   3 restart things at once            ├─ Ops lead: actually investigates/fixes
   nobody owns comms                   ├─ Comms lead: updates status page/stakeholders
   theories collide, work is dropped   └─ Scribe: timestamps every action in the log
   leadership asks "what's happening?"
   ...and gets 5 different answers       one source of truth, no duplicated work

The IC’s jobs: maintain a single shared picture of what’s known, decide what to try next (and what not to — stop people from making uncoordinated changes that muddy the signal), delegate the hands-on fixing to an ops lead, and own communication so the engineers fixing the problem aren’t interrupted every two minutes by “any update?” The IC is often not the most senior engineer and often not the one typing commands — separating the coordinator from the fixer is the whole point.

MTTD and MTTR: the clocks you’re racing

Two measurements quantify how well response works, and they decompose the DORA “time to restore” metric into its parts:

   incident begins                                          service healthy
   │                                                              │
   │◄──── MTTD ────►│◄─────────────── MTTR ──────────────────────►│
   │  time to detect │  time to restore (detect→diagnose→fix→verify)
   problem starts    you NOTICE       you understand    you've fixed it

MTTD — Mean Time To Detect: how long from a problem starting to you knowing about it. This is bought directly with the metrics and alerting from earlier in this Part — good alerts shrink MTTD toward zero. A long MTTD means you found out from an angry customer, not your own systems.
MTTR — Mean Time To Restore/Recover: how long from detecting to the service being healthy again. This is where logs and traces pay off — fast diagnosis is most of MTTR, and the whole observability stack exists to make that diagnosis fast.

The link back to the book’s thesis is direct: this entire Part has been an investment in driving MTTD and MTTR down, which is the same thing as driving DORA’s time-to-restore down — the stability metric that, the data shows, unlocks moving fast.

Blameless postmortems: where the real value is

The incident is over. The biggest mistake you can make now is to move on without learning — or worse, to hold a postmortem that hunts for someone to blame. The discipline is the blameless postmortem: a written analysis of what happened, why, and what will change, conducted on the explicit premise that nobody is punished for the failure.

Why blameless? Because blame destroys the one thing the postmortem needs: the truth.

   BLAMEFUL postmortem                 BLAMELESS postmortem
   ───────────────────                 ────────────────────
   "who broke it?"                     "what in the SYSTEM let this happen?"
   engineers hide details              engineers explain candidly
   root cause stays hidden             root cause surfaces
   the SYSTEM doesn't change           the system gets a guardrail
   it happens again                    it can't happen the same way again

If the engineer who ran the bad command knows they’ll be punished, they will minimise, omit, and defend. You lose the candid timeline — and the candid timeline is the entire raw material for fixing the system. The blameless premise is not naïve niceness; it’s an engineering decision to maximise signal.

This is the same culture the foundations page argued for, now applied at the sharpest moment. And the recurring thread of this book is its mechanism: the fix for “a human made a mistake” is almost always to remove the manual, error-prone step that made the mistake possible.

Action items: a postmortem without them is a diary

A postmortem that ends in a description and no changes is a diary — it documents the pain and prevents nothing. The output that matters is action items: concrete, owned, tracked changes that make this class of incident less likely or less severe. Good action items are specific and assigned:

Vague (useless)	Specific (an action item)
“Be more careful with deploys"	"Add a pre-deploy check that blocks rollout if the canary error rate exceeds baseline — owner: A, due: next sprint"
"Improve monitoring"	"Add an SLO burn-rate alert on checkout latency — owner: B"
"Communicate better"	"Auto-create the incident channel and page the IC on SEV1 — owner: C”

Notice every good action item adds automation or a guardrail — it removes a manual step or catches a mistake earlier. That is the postmortem feeding straight back into the shift-left and reconciliation-loop ideas: turn a lesson into a check the machine enforces, so the system, not human memory, holds the line.

In early 2017, a GitLab.com engineer fighting a late-night incident ran a deletion command against what they believed was a spare database replica — but it was the primary, wiping roughly 300 GB of production data. Recovery then exposed the deeper failure: of the several backup and replication methods GitLab believed it had, none worked as expected when reached for. They restored from a staging snapshot several hours old, permanently losing that window of data.

What made it a landmark wasn’t the outage — it was the response. GitLab live-streamed the recovery and published a detailed blameless postmortem naming the systemic causes: backups that were never tested, a destructive command far too easy to point at prod, and no guardrail between a tired engineer and the primary database. Every fix that followed changed the system, exactly as this page argues — “the engineer ran the wrong command” was the symptom, not the root cause. The lasting, uncomfortable lesson: a backup you have never tested to restore is not a backup.

The thread: from heroics to a system that learns

The manual, error-prone step incident discipline removes is ad-hoc, every-person-for-themselves crisis response — uncoordinated firefighting where the recovery depends on whichever hero happens to be online, and the same failure recurs because nothing in the system changed. Structured response (on-call, severity, an incident commander) shortens the outage; the blameless postmortem with owned action items ensures the system improves so the failure can’t recur the same way. Production gets safer in the way DORA measures directly: MTTD and MTTR fall, time-to-restore drops, and — because each incident hardens the system through automation rather than blame — the rate of repeat failures falls too.

That closes Part 7. You can now see your system (logs, metrics, traces), decide how reliable it must be (SLOs and error budgets), and respond and learn when it breaks (incident response and postmortems). The next Part turns to a threat that observability alone can’t catch — making the whole pipeline and platform secure: Part 8 · Security.

Check your understanding

What is alert fatigue, and why is a noisy pager more dangerous than no pager at all? Connect it to the burn-rate alerting from earlier in the Part.
Why does the incident commander coordinate rather than fix? Describe two concrete failures of an IC-less response that the role prevents.
Distinguish MTTD from MTTR and map each to a pillar (or pillars) of observability that shrinks it. Which DORA metric do they decompose?
Explain, mechanically, why a blameful postmortem produces worse outcomes than a blameless one. What does blame destroy?
“The engineer ran the wrong command” — why is this not a root cause? Rewrite it into the kind of question a blameless postmortem asks, and into a good action item.

Show answers

Alert fatigue is what happens when on-call is paged too often for non-actionable or non-urgent things: people burn out and start ignoring pages, so the one real emergency gets missed in the noise — worse than no pager, which at least doesn’t lull you into ignoring it. Burn-rate alerting (page on fast burn, ticket on slow burn) keeps the pager quiet enough to stay trusted.
With five people all “fixing,” changes collide, work is duplicated or dropped, nobody owns communication, and leadership gets contradictory answers. The IC maintains one shared picture, decides what to try (and stops uncoordinated changes that muddy the signal), and shields the fixers from constant status questions — so coordination doesn’t compete with the actual repair.
MTTD (mean time to detect) is problem-start → you-know, shrunk by good metrics/alerting. MTTR (mean time to restore) is detect → healthy, shrunk by logs and traces that make diagnosis fast (diagnosis is most of MTTR). Together they decompose DORA’s time to restore service.
If people expect punishment, they minimise, omit, and defend, so the candid timeline — the raw material for understanding the failure — never surfaces; the real root cause stays hidden and the system isn’t fixed, so it recurs. Blame destroys the truth the postmortem depends on.
It’s a starting point, not a root cause, because it stops at the human and ignores the system that let the error happen. Better question: “Why was that command so easy to run by mistake, and why did nothing catch it before it hit production?” Action item: “Add a guardrail/confirmation (or automated check) that blocks/flags this command against production — owner X, due date Y,” so the next inevitable mistake is caught by the system.