Deployment Strategies

You have an immutable artifact that passed every gate. Now it has to replace the version currently serving users. The naive way — stop the old one, start the new one — has a fatal flaw: if the new version is broken, every user hits the breakage at once, and there’s downtime while you swap. Tests catch a lot, but production is where reality lives: real traffic, real data, real load. So the question every deployment strategy answers is the same one: when (not if) a new version misbehaves, how few users are exposed, and how fast can you take it back? That exposure is the blast radius, and shrinking it is the entire game.

Rolling

A rolling deployment replaces instances of the old version with the new one a few at a time, rather than all at once. With ten replicas, you might take down one or two, bring up new ones, wait for them to report healthy, then continue. This is Kubernetes’ default for a Deployment, driven by the reconciliation loop.

   v1 v1 v1 v1 v1   ─►   v2 v1 v1 v1 v1   ─►   v2 v2 v1 v1 v1   ─►   v2 v2 v2 v2 v2
   all old               replace 1 at a time, keep serving        all new

It gives you zero-downtime (old instances serve while new ones come up) and bounds the blast radius to the batch size. The limits: both versions run simultaneously mid-roll (your app and database must tolerate that), and rollback means rolling back, which takes about as long as rolling forward.

Blue-green

Blue-green keeps two complete environments. Blue is live and taking all traffic; green is the new version, fully deployed and warmed up but receiving none. You test green privately, then flip the router so all traffic moves to green at once. Blue stays running, untouched.

            ┌── BLUE  (v1)  ◄── 100% traffic        flip      ┌── BLUE  (v1)  ◄── kept for rollback
   router ──┤                                  ─────────►  ───┤
            └── GREEN (v2)  ◄── 0%, being tested              └── GREEN (v2)  ◄── 100% traffic

The win is instant rollback: if green misbehaves, flip the router back to blue — which is still exactly as it was — in seconds. The cut-over is atomic, so users never see a mix of versions. The cost is running two full environments at once (double the resources during the deploy), and shared state like a database needs care since both sides may touch it.

Canary

A canary release sends a small slice of real traffic — 1%, then 5%, then 25% — to the new version while everyone else stays on the old one. You watch the new version’s metrics (error rate, latency) on that slice; if it stays healthy you widen the slice, and if it degrades you route the slice back. The name comes from the canary in a coal mine: a small, early warning before the whole mine is at risk.

   v1 ◄── 99% ─── traffic ───┐      healthy? widen ─► 95% / 5% ─► 75% / 25% ─► 0% / 100%
   v2 ◄──  1% ───────────────┘      sick?    revert ─► 100% / 0%  (only 1% ever saw it)

This is the tightest blast radius of the three: a bad version is seen by ~1% of users for a few minutes before metrics flag it, and rollback affects only that slice. The cost is sophistication — you need real-time metric analysis and traffic-splitting to do it well, which is why canarying is the heart of progressive delivery.

Feature flags: decouple deploy from release

The strategies above control which version of the artifact serves traffic. Feature flags add a different axis: ship the new code to everyone, but keep the new behavior switched off behind a runtime condition, and turn it on later — for internal users first, then 1% of customers, then all.

if flags.enabled("new-checkout", user):
    return new_checkout(user)   # dark in prod until the flag is flipped
return old_checkout(user)

The profound move here is decoupling deployment from release. Deploying (the bytes are in production) and releasing (users get the new behavior) become two separate decisions. The code can sit dark in production for days; the moment of “going live” becomes a config flip, instantly reversible with no redeploy. This is also what lets trunk-based development merge unfinished features safely — they ship dark.

Choosing, and the thing they all share

Strategy	Blast radius	Rollback speed	Cost
Rolling	Batch of instances	Roll back (~same as forward)	Low
Blue-green	All-or-nothing flip	Instant (flip router)	High (2× env)
Canary	~1% of users	Instant (revert slice)	Metrics + traffic split
Feature flags	Whoever the flag targets	Instant (toggle flag)	Flag system + discipline

Fast rollback beats perfect prevention

No amount of testing makes a deploy risk-free, so the mature posture isn’t “never break prod” — it’s “make breaking prod cheap to undo.” A strategy is good in proportion to how fast and how partially you can take a bad change back. Optimize for the recovery, not just the prevention.

This is the book’s thread applied to the riskiest moment in the whole lifecycle. The manual, error-prone step being removed is the big-bang manual cut-over — stop everything, swap, pray, and scramble to reconstruct the old state if it goes wrong. Every strategy here replaces “all users, all at once, hard to undo” with “few users, gradually, trivially reversible.” The safety isn’t that nothing breaks — it’s that when something breaks, almost no one notices and you can undo it in seconds. The next two pages ask: who does the deploying, and can git itself be the trigger? That’s GitOps.

By the numbers — how small a canary’s blast radius really is

Put numbers on the canary. Say you serve 50,000 requests/minute, a bad version is live on a 1% canary, and your metrics flag the regression in 3 minutes:

canary:      50,000 req/min × 1% × 3 min =   1,500 bad requests
big-bang:    50,000 req/min × 100% × 3 min = 150,000 bad requests

Same 3-minute detection window, a 100× difference in how many users hit the breakage — purely from who you exposed. That ratio is the blast-radius argument: the cost of canarying (traffic-splitting plus real-time metric analysis) buys you two orders of magnitude fewer victims of the next bad deploy.

The canonical cautionary tale

The 2012 Knight Capital disaster — a manual deploy that left one of eight servers on stale code, which a reused flag turned into roughly a $440M loss in 45 minutes — is the textbook example of why this Part exists: deploy the same immutable artifact to every node (never a hand-copy that can miss one), keep rollback instant, and never let “flip a flag” be an untested launch. The feature-flag root cause is dissected in Progressive Delivery & Feature Flags.

The architect’s lens

Whichever strategy you pick, it’s an answer to the same five questions:

• Why does it exist? Because the naive cut-over — stop the old, start the new — exposes every user to a bad version at once, with downtime; strategies exist to shrink that blast radius.
• What problem does it solve? “When (not if) a deploy is bad, how few users notice and how fast can you undo it?” — rolling bounds it to a batch, blue-green gives an instant router flip-back, canary limits it to ~1% (1,500 vs 150,000 bad requests in the worked example).
• What are the trade-offs? Each buys recovery with cost: blue-green doubles resources during the deploy; canary needs real-time metric analysis and traffic-splitting; feature flags add a flag system plus the discipline to retire stale flags.
• When should I avoid it? A simple low-traffic service may not justify canary’s metric tooling or blue-green’s 2× footprint — a plain rolling update is often enough.
• What breaks if I remove it? You’re back to the big-bang manual cut-over that made Knight Capital famous (~$440M in 45 minutes): all users at once, hard to undo, with recovery reconstructed under pressure.

Check your understanding

1. State the one question that every deployment strategy is an answer to. What is the “blast radius”?
2. Compare rolling and blue-green on rollback speed and resource cost. When would you pick each?
3. Why is canary the tightest blast radius of the three version-based strategies, and what capability does it require that the others don’t?
4. Explain “decoupling deployment from release.” How do feature flags make releasing a feature a separate decision from deploying its code?
5. Using the book’s thread, explain why “fast rollback beats perfect prevention” is the mature posture, and what manual step every strategy here removes.

Show answers

1. Every strategy answers: when (not if) a new version misbehaves, how few users are exposed, and how fast can you take it back? The blast radius is that exposure — how many users hit the breakage before you can revert.
2. Rolling has low cost (no extra environment) but rollback takes about as long as rolling forward; blue-green gives instant rollback by flipping the router back to the untouched old environment, at the cost of running two full environments at once. Pick rolling when resources are tight and a slower rollback is acceptable; pick blue-green when you need an atomic cut-over and instant, all-or-nothing rollback.
3. Canary exposes only a small slice of real traffic (~1%) to the new version, so a bad version is seen by far fewer users than a rolling batch or a blue-green flip. It requires real-time metric analysis and traffic-splitting — capabilities rolling and blue-green don’t need.
4. Decoupling deployment from release means deploying (the bytes are in prod) and releasing (users get the new behavior) become two separate decisions. A feature flag ships the new code dark behind a runtime condition; flipping the flag later is the actual release — instantly reversible with no redeploy.
5. No testing makes a deploy risk-free, so the mature posture is to make breaking prod cheap to undo rather than to chase perfect prevention. Every strategy here removes the big-bang manual cut-over — stop everything, swap, pray — replacing “all users at once, hard to undo” with “few users, gradually, trivially reversible.”