Progressive Delivery & Feature Flags

The deployment strategies page introduced the canary — send a small slice of traffic to a new version, watch its metrics, and widen or revert based on what you see. It also introduced feature flags as a way to decouple deployment from release. This page takes both ideas seriously and turns them into a discipline: progressive delivery. The core claim is that a release should not be a moment but a process — a controlled, observed, automatically-governed rollout in which a bad change is caught by software, on a tiny blast radius, before a human is even paged.

The problem it solves is the gap the deployment-strategies page left open. A manual canary still needs a human watching a dashboard, deciding “looks fine, widen it,” at every step — which is slow, subjective, and exactly the kind of toil that gets skipped under pressure. Progressive delivery closes that gap by making the promotion decision itself automatic and tied to the same SLOs and error budgets you’d use to judge the system anyway.

First: deploy is not release

Hold these two as separate events, because everything here depends on the split:

• Deploy — the new bytes are present in production. The artifact is running.
• Release — users get the new behavior. The change is actually exposed.

Conflating them is the old, dangerous default: the instant you deploy, everyone is affected, so a bad deploy is a full outage. Separating them turns “going live” into an independent, gradual, reversible decision. Feature flags are the cleanest way to make the split concrete: ship the code to everyone with the new behavior switched off, then turn it on for a widening audience on your own schedule.

if flags.enabled("new-pricing", user):
    return new_pricing(user)    # deployed everywhere, dark until released
return current_pricing(user)

The code is in production for everyone. Whether any given user experiences it is a runtime decision you control independently — and can reverse in milliseconds without a redeploy.

A flag-gated, percentage rollout

Put the two together and a release becomes a dial you turn slowly while watching the patient. The flag isn’t just on/off — it targets a percentage (and often specific cohorts: internal users, then beta opt-ins, then a geography, then everyone).

   FLAG-GATED PROGRESSIVE ROLLOUT (the new path is dark until you turn the dial)
   ───────────────────────────────────────────────────────────────────────────
   step   audience              flag        watch SLOs       decision
   ────   ────────              ────        ──────────       ────────
    0     nobody (deployed)     off  0%      —               code is live but dark
    1     internal staff        on   ~0%     dogfood         humans sanity-check
    2     1% of users           on   1%      error budget    healthy? → widen
    3     5% → 25% → 50%        on   ↑%      error budget    each step: healthy? → widen
    4     100%                  on   100%    error budget    fully released
          ▲                                                          │
          └──────────── KILL SWITCH: flip to 0% instantly ◄──────────┘
                        (any step, no redeploy, milliseconds)

Two things are doing the work. The percentage bounds the blast radius at every step — at step 2 a catastrophic bug reaches 1% of users, not all of them. The kill switch — flipping the flag back to 0% — is the escape hatch: because release is just a config value, undoing a release is instant and total, with no rebuild, no redeploy, no rollback rollout. You optimize for fast, partial recovery, exactly the posture the deployment-strategies page argued for.

A kill switch is the cheapest insurance in production

Every risky new path should be behind a flag you can flip to off in seconds, from a console, without shipping code. When something goes wrong at 3 a.m., “flip the flag” is a fundamentally calmer incident than “build, test, and deploy a revert.” Wiring the flag is a few minutes of work; not having it is how a small bug becomes a long outage.

Automated canary analysis: let the SLO decide

The leap from a manual canary to progressive delivery is automating the promote/rollback decision. A controller (Argo Rollouts, Flagger, and similar) drives the rollout as a control loop: it shifts a slice of traffic to the new version, waits, queries your metrics, and compares the canary’s behavior against the baseline using your SLOs as the pass/fail criterion — then promotes or aborts on its own.

   AUTOMATED CANARY ANALYSIS (a control loop)
   ──────────────────────────────────────────
   ┌─► shift +X% traffic to canary
   │      │
   │      ▼
   │   wait (bake time) ──► query metrics: canary error rate / latency
   │      │                          vs baseline + SLO threshold
   │      ▼
   │   within budget? ──no──► ABORT: route 100% back to stable, page on-call
   │      │ yes
   │      ▼
   └── promote: widen the slice ... until 100%, then mark stable

This is why the SLOs and error budgets page matters so much here: the SLO is the objective, pre-agreed definition of “healthy.” Instead of a tired human squinting at a dashboard and guessing, the rollout controller has a number — “the canary’s error rate must stay within the error budget” — and enforces it mechanically. The decision is faster (machine speed), consistent (same criterion every time), and trustworthy because it’s the same SLO the team already committed to.

The choice of metric matters. Good canary criteria are symptom-level signals — error rate, latency, saturation — measured on the canary’s own traffic, not aggregate dashboards where 1% of bad requests drowns in 99% of good ones. The whole point is to detect the badness while it’s still confined to the canary slice, which means measuring that slice specifically.

A canary needs enough traffic and time to be meaningful

Two failure modes: too little traffic (1% of a low-volume service may be a handful of requests an hour — no statistical signal, so the analysis can’t distinguish a broken canary from noise), and too little bake time (promote before a memory leak or slow-burning error has shown up). Automated analysis is only as good as the signal it’s analyzing; give it real traffic and real time, or it will confidently promote a bad change.

Where each piece fits

These are different axes that compose, not competitors:

Mechanism	Axis it controls	Reversal
Deployment strategy (rolling/blue-green/canary)	Which version serves traffic	Re-route / roll back
Feature flag	Which users get a new behavior (decouples deploy from release)	Toggle the flag
Automated canary analysis	Whether to promote, judged by SLOs	Abort, route back to stable
Progressive delivery	All of the above, as one governed process	Any of the above, automatically

The service mesh is often the muscle underneath: “send 5% of traffic to v2” is a sentence its control plane enacts across the fleet, which is exactly what the rollout controller needs to drive the slice.

It actually happened — Knight Capital’s reused flag, 2012

On 1 August 2012, the trading firm Knight Capital deployed new code to its order-routing system across eight servers — but one server never received the update. Worse, the deploy reused an old feature flag that, on the un-updated server, switched on long-dead code (an abandoned feature called “Power Peg”) instead of the new behavior. The instant markets opened, that one server began firing millions of erroneous orders; in roughly 45 minutes Knight lost about $440 million and never recovered as an independent company. Two lessons land squarely on this page. First, a partial, inconsistent rollout with no automated health gate is exactly the blast radius progressive delivery bounds — an automated canary watching a steady-state metric would have caught the storm while it was still confined to one server’s traffic. Second, the trigger was a flag that should have been retired long before — precisely why this page warns that flags must be cleaned up so they “don’t rot into permanent dead branches.”

The thread

The manual, error-prone step progressive delivery removes is a human acting as the release valve — watching a dashboard, eyeballing whether a canary “looks okay,” and manually deciding to widen or revert at every step of every rollout. That job is slow, subjective, and the first thing dropped under pressure. Progressive delivery hands it to a controller that judges the canary against pre-committed SLOs and acts at machine speed, while feature flags keep a kill switch one click away. Production gets safer in a precise way: a bad change is exposed to a tiny, bounded slice; it’s caught by an objective criterion rather than a tired guess; and it’s reversed in milliseconds by toggling a flag instead of shipping a fix. You move from “deploy and hope, then scramble” to “release gradually, judged automatically, reversible instantly.”

The cost is honest infrastructure: a flag system to operate (and flags to clean up so they don’t rot into permanent dead branches), a rollout controller, and SLOs good enough to bet a rollback on. But the payoff is that the riskiest moment in the lifecycle stops depending on a human’s attention. Next, we stop trying to avoid failure and start causing it deliberately, to find out whether the system is actually as resilient as we hope: chaos engineering.

The architect’s lens

Five questions for governing a rollout:

• Why does it exist? Because a manual canary still needs a human watching a dashboard deciding “looks fine, widen it” — slow, subjective, and the first thing dropped under pressure.
• What problem does it solve? It makes a release a process, not a moment: deploy decoupled from release via feature flags, a percentage rollout that bounds blast radius, and a controller (Argo Rollouts, Flagger) that promotes or aborts against your SLOs at machine speed — with a millisecond kill switch.
• What are the trade-offs? Honest infrastructure — a flag system to run (and flags to retire before they rot into dead branches, à la Knight Capital), a rollout controller, and SLOs trustworthy enough to bet a rollback on — plus a canary that needs enough traffic and bake time to mean anything.
• When should I avoid it? A low-traffic service can’t give a 1% slice a statistically meaningful signal, and a trivial change may not justify the flag-and-controller machinery.
• What breaks if I remove it? The release valve goes back to a tired human’s guess — a bad change is exposed all at once and reversed by shipping a fix instead of toggling a flag.

Check your understanding

1. State the difference between deploy and release, and explain how feature flags make them two separate decisions.
2. Walk through a flag-gated percentage rollout. What does the percentage bound, and what does the kill switch give you that a code-revert doesn’t?
3. What is automated canary analysis, and why does it depend on having good SLOs? Why is judging the canary’s own traffic better than an aggregate dashboard?
4. Name the two ways an automated canary can be fed too little signal, and what each causes.
5. Using the book’s thread, what manual job does progressive delivery remove, and what are its honest costs?

Show answers

1. Deploy = the new bytes are present in production; release = users actually get the new behavior. A feature flag ships the code to everyone with the new path off, so turning it on (the release) is a runtime decision made independently of the deploy and reversible without a redeploy.
2. The new path is dark at 0%, then exposed to a widening audience (internal → 1% → 5% → 25% → 100%), with SLOs watched at each step. The percentage bounds the blast radius (a bad bug hits only that slice); the kill switch flips the flag to 0% instantly, no rebuild/redeploy — far faster and more total than shipping a code revert.
3. A controller shifts traffic to the canary, waits, queries metrics, and promotes or aborts based on whether the canary stays within the error budget — a control loop. It depends on SLOs because the SLO is the objective, pre-agreed definition of “healthy” that replaces a human’s guess. Judging the canary’s own traffic matters because in an aggregate dashboard 1% of bad requests is drowned by 99% of good ones, hiding the very problem you’re watching for.
4. Too little traffic (1% of a low-volume service is statistically meaningless — can’t tell broken from noise) and too little bake time (promoting before a slow-burning issue like a memory leak appears).
5. It removes a human acting as the release valve — eyeballing canaries and manually deciding to widen or revert. Costs: a flag system to run (and flags to retire), a rollout controller, and SLOs trustworthy enough to bet a rollback on.