SLOs & Error Budgets

The last three pages gave you signals — logs, metrics, traces. Signals are necessary but not sufficient. They tell you what is happening; they do not tell you whether it’s acceptable, or what to do about it. “p99 latency is 800ms” — is that fine or a crisis? You can’t answer without a target. This page is where observability becomes a discipline: Site Reliability Engineering (SRE), and its central instrument, the error budget.

The deep problem SRE solves is the one Speed vs Stability framed as a false tradeoff. Every team feels the tug between shipping features (speed) and not breaking things (stability), and that tension usually gets resolved by whoever argues loudest — product pushes to ship, ops pushes to freeze. SRE replaces the argument with arithmetic: it sets a numeric reliability target, measures against it continuously, and lets the remaining budget decide who’s right this week.

SLI, SLO, SLA — three letters, three different things

These get conflated constantly. Keep them straight.

Term	What it is	Example	Audience
SLI — Service Level Indicator	A measurement of one aspect of service health	”% of requests served in under 300ms”	Engineering
SLO — Service Level Objective	A target for an SLI over a window	”99.9% of requests under 300ms over 28 days”	Engineering + product
SLA — Service Level Agreement	A contract with consequences if breached	”99.5% uptime or you get a refund”	Legal / customers

The relationship is a hierarchy: the SLI is the number, the SLO is the line you draw on it, the SLA is the promise to a customer with money attached.

   SLI  ──►  "what we measure"     99.94% of requests fast   (a fact)
   SLO  ──►  "what we aim for"     ≥ 99.9% over 28 days       (an internal goal)
   SLA  ──►  "what we promise"     ≥ 99.5% or refund          (an external contract)

Set your SLO stricter than your SLA — deliberately

Your internal SLO should be tighter than any external SLA, so you’re alarmed and acting well before you breach the contract that costs money. If your SLA promises 99.5%, you might run a 99.9% SLO — the gap is your safety margin. You never want the first sign of trouble to be a customer invoking the contract.

Choosing good SLIs

A bad SLI measures something that doesn’t reflect user pain. The discipline is to measure what the user experiences, framed as a ratio of good events to total events:

   SLI = good events / valid events

   availability: successful requests / total requests
   latency:      requests under 300ms / total requests

Good SLIs share traits:

• User-centric — they track what a user would notice (request success, page load), not an internal proxy like CPU. A server can be at 90% CPU and perfectly fast; the user doesn’t care about CPU.
• A ratio of good to total — this makes them naturally a percentage, which makes the SLO and the budget fall out cleanly.
• Measurable from data you already have — usually straight from your metrics, e.g. the 5xx-ratio PromQL from that page is an availability SLI.

The classic mistake is an SLO of “100%.” A 100% target is a trap: it’s effectively unachievable, leaves zero room to ship anything risky, and the cost of the last fraction of a nine is astronomical. Users can’t even tell the difference between 100% and 99.9% — but your ability to ship features can. Pick the loosest SLO your users are happy with, not the tightest you can imagine.

The error budget: 1 − SLO

Here is the idea that makes SRE more than nice dashboards. If your SLO says 99.9% of requests must succeed, then by definition you are allowed to fail 0.1% of them. That allowance is your error budget:

   error budget = 1 − SLO

   SLO 99.9%  →  budget 0.1%  →  over 28 days ≈ 40 minutes of "down"
   SLO 99.0%  →  budget 1.0%  →  over 28 days ≈ 7 hours
   SLO 99.99% →  budget 0.01% →  over 28 days ≈ 4 minutes

The reframe is profound. Failures are no longer a moral failing to be driven to zero — they are a budget to be spent. A 99.9% SLO doesn’t say “never fail”; it says “you have ~40 minutes of failure to spend this month, so spend it wisely.” Unused reliability beyond your SLO is waste — it means you were too cautious and could have shipped more.

This is the cousin of an idea you already met: the DORA insight that you should design for recovery, not zero failures. The error budget operationalises it. You stop asking “how do we never break?” and start asking “have we got budget to spend on this risk?”

Try it: plug in your own SLO and window below to see exactly how much “down” the budget buys — and, if you enter a measured availability, how much of that budget you’ve already spent (and whether you’ve blown it).

How the budget arbitrates speed vs stability

This is the payoff, and it’s the cleanest mechanism in this book for resolving the speed-versus-stability fight without anyone arguing.

   BUDGET REMAINING?
   ─────────────────
   ✅ budget left      →  SHIP. Take risks, deploy the new feature,
                          run the experiment. You can afford a failure.

   ❌ budget exhausted →  FREEZE features. All effort goes to reliability
                          until the budget recovers. Stability wins — automatically.

The fight resolves itself with a number. Plenty of budget left means the team has been too cautious — go faster, ship the risky thing, that’s literally what the budget is for. Budget gone means you’ve been too reckless — stop shipping features and fix reliability until you’re back in the black. Nobody has to win the argument by seniority or volume; the data decides. Product and SRE stop being adversaries because they’re both reading the same gauge.

Why this aligns incentives that used to clash

Before error budgets, “ops” was rewarded for stability and “dev” for features, so they pulled in opposite directions forever. The error budget gives both a shared number: developers are free to spend it on velocity, but they own the consequence when it runs out. It turns a political fight into a shared resource-management problem.

Burn-rate alerting

A naive alert (“we breached the SLO”) fires too late — the budget is already gone. A better question is how fast are we burning the budget right now? That’s the burn rate: the multiple of the “sustainable” spend rate at which you’re consuming budget.

• Burn rate 1× = spending the budget exactly evenly over the window; you’ll end at precisely 0. Fine.
• Burn rate 10× = you’ll exhaust the whole month’s budget in ~3 days. Something is wrong now.

   slow burn  →  small, persistent error rate   →  ticket, fix this week   (low urgency)
   fast burn  →  budget vanishing in hours        →  PAGE someone NOW       (high urgency)

The practice (from Google’s SRE workbook) is multi-window, multi-burn-rate alerts: page on a fast burn over a short window (e.g. 14.4× over 1 hour — an emergency), and open a ticket on a slow burn over a long window (e.g. 3× over 6 hours — needs attention, not a 3am page). This is the rigorous version of the “alert on symptoms, with a for: clause” advice from Metrics: you alert proportionally to how fast you’re destroying your reliability target, so the page actually correlates with urgency.

Toil, and what SRE actually is

SRE (Site Reliability Engineering) is the practice — originating at Google — of running operations as a software problem. Its load-bearing ideas:

• SLOs and error budgets decide how reliable to be and arbitrate risk (this page).
• Toil reduction. Toil is manual, repetitive, automatable operational work that scales linearly with the service — restarting a service by hand, manually rotating logs, clicking through the same runbook every week. SRE practice caps the time spent on toil (a common rule of thumb is ~50%) and spends the rest automating it away. Toil is the exact thing this whole book targets: the recurring thread is “what manual, error-prone step does this remove?”, and toil is the inventory of those steps.
• Blameless culture for incidents — the subject of the next page.

By the numbers — where 14.4× and 6× come from

The “fast burn pages, slow burn tickets” rule has concrete arithmetic behind it. Google’s SRE Workbook publishes a canonical multiwindow, multi-burn-rate table; for a 99.9% SLO over a 30-day window:

   30-day error budget = 0.1% × 30 days         ≈ 43 minutes of "bad"
   at burn rate N×, the whole budget is gone in   30 days ÷ N

   burn 14.4×  → gone in ~50 h  → 2%  burned in 1 hour   → PAGE  (fast)
   burn  6×    → gone in 5 days → 5%  burned in 6 hours  → PAGE
   burn  1×    → exactly on pace → 10% burned in 3 days  → TICKET (slow)

The 14.4 isn’t magic: 14.4 × (1 hour ÷ 720 hours in 30 days) = 2% of the budget. The table is engineered so a page fires only when you’d burn a meaningful slice of the month in a short window — high enough to be a real emergency, which is what keeps the pager trustworthy and fends off the alert fatigue the next page treats.

The thread: from arguing to measuring

The manual, error-prone step SLOs remove is the recurring human judgement call of “is this reliable enough to ship?” — a decision previously made by gut feel, seniority, or whoever was loudest in the room, and re-litigated in every release meeting. The error budget replaces that judgement with a measured quantity that updates continuously, so the speed-vs-stability decision is automatic and consistent instead of political and ad hoc. Production gets safer because the brakes engage by data, not by argument: when reliability is genuinely at risk, feature work stops because the number says so, before customers feel it — not after a heated meeting that ops loses.

When the budget does get blown — when something breaks — you need a disciplined way to respond and to learn. That’s the final page of this Part: Incident Response & Postmortems.

The architect’s lens

Five questions for adopting SLOs and error budgets:

• Why does it exist? Because signals tell you what is happening but not whether it’s acceptable — “p99 is 800ms” is meaningless without a target, so SRE draws a numeric reliability line.
• What problem does it solve? It ends the speed-vs-stability argument with arithmetic: error budget = 1 − SLO, and the budget remaining decides automatically — budget left means ship, budget gone means freeze — instead of whoever argues loudest.
• What are the trade-offs? It demands choosing good user-centric SLIs and resisting the 100%-SLO trap (users can’t tell 100% from 99.9%, but your velocity can), plus the rigor of multi-window burn-rate alerting (page at ~14.4×, ticket on a slow burn).
• When should I avoid it? An internal tool with no real reliability expectations doesn’t need a formal error budget — the machinery earns its keep where the speed-vs-stability tension is genuinely contested.
• What breaks if I remove it? “Is this reliable enough to ship?” goes back to gut feel and seniority, re-litigated every release — and you discover you breached only when a customer invokes the SLA.

Check your understanding

1. Define SLI, SLO, and SLA and give the relationship between them. Why should your internal SLO be stricter than any SLA you’ve signed?
2. Why is a 100% SLO almost always the wrong target? Reason about both user perception and your ability to ship.
3. Derive the error budget from an SLO. Reframe in one sentence what a 99.9% SLO permits rather than forbids, and explain why leftover reliability is “waste.”
4. Walk through how the error budget resolves the speed-vs-stability fight when there is budget left and when there isn’t — and why this removes the need to win an argument.
5. Why is burn-rate alerting better than alerting on “we’ve breached the SLO”? Why page on a fast burn but only open a ticket on a slow burn?

Show answers

1. An SLI is a measurement (e.g. % of fast requests), an SLO is a target for that SLI over a window (e.g. ≥99.9% over 28 days), and an SLA is a customer contract with consequences (e.g. refund below 99.5%). SLI → SLO → SLA goes measurement → internal goal → external promise. The SLO is stricter than the SLA so you start reacting well before you breach the contract that costs money — the gap is your safety margin.
2. 100% is effectively unachievable, infinitely expensive at the margin, and leaves zero room to ship anything risky. Users can’t perceive the difference between 100% and 99.9%, but your feature velocity absolutely can — so the extra nines buy nothing the user notices while costing everything in agility. Pick the loosest SLO users are happy with.
3. Error budget = 1 − SLO. A 99.9% SLO yields a 0.1% budget (~40 min over 28 days). It permits you to fail 0.1% of requests — failures are a budget to spend, not a sin to eliminate. Leftover reliability is waste because it means you were too cautious and could have shipped more risk for the same user experience.
4. Budget left: ship — take risks, deploy the feature, run the experiment; you can afford a failure. Budget exhausted: freeze features and put all effort into reliability until it recovers. The number decides, so nobody has to win by seniority or volume — product and SRE read the same gauge instead of fighting.
5. Alerting on “breached the SLO” fires too late — the budget is already gone. Burn rate measures how fast you’re consuming budget right now, so you can act before exhaustion. A fast burn (e.g. budget gone in hours) is an emergency worth a page; a slow burn (small persistent error rate) needs attention this week but not a 3am wake-up — the alert urgency matches the actual urgency.