Canary Deploy

This is the last step, and the one the whole Part was building toward. We have a signed image, a Deployment and metrics, and Argo CD reconciling the cluster to a git repo. Now we replace the riskiest moment in the lifecycle — the cut-over to a new version — with a canary that ships gradually, judges itself on real metrics, and undoes itself if it’s wrong.

The manual step being removed here is the big-bang cut-over watched by a human: deploy to everyone at once, then stare at a dashboard ready to scramble a rollback by hand. We replace both halves — the all-at-once exposure and the human-in-the-loop judgment — with a controller that exposes the new version to a sliver of traffic, runs the page-5 SLO queries against it, and promotes or rolls back on the answer.

GitOps stays in charge: a deploy is still a commit

Nothing about the GitOps model changes. The flow from the plan still holds: CI builds snip:<sha>, then bumps that tag in the snip-config repo. Argo CD sees the new desired state and applies it. The only difference is what it applies — instead of a plain Deployment that swaps pods all at once, it’s an Argo Rollouts Rollout, which knows how to canary.

   CI pushes snip:<new-sha> ─► commit to snip-config repo ─► Argo CD syncs
                                                                  │
                                                                  ▼
                                            Argo Rollouts takes over the rollout:
                                            10% ─► analyze ─► 50% ─► analyze ─► 100%

So the trigger is a git commit (auditable, reviewable, revertable) and the execution is a gradual, metric-gated rollout. The two ideas compose cleanly.

The Rollout: 10% → 50% → 100%, with analysis at each step

A Rollout replaces a Deployment. Its strategy.canary lists the steps: shift a percentage of traffic, then pause — either for a fixed time, or until an analysis says the new version is healthy.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: snip
  namespace: snip
spec:
  replicas: 3
  selector:
    matchLabels: { app: snip }
  template:        # identical pod spec to the page-5 Deployment
    metadata:
      labels: { app: snip }
    spec:
      containers:
        - name: snip
          image: ghcr.io/acme/snip:a1b2c3d4e5f6   # CI bumps this tag in git
          ports:
            - { containerPort: 8080 }
          readinessProbe:
            httpGet: { path: /healthz, port: 8080 }
  strategy:
    canary:
      steps:
        - setWeight: 10          # send 10% of traffic to the new version
        - analysis:              # judge it on real metrics before going wider
            templates:
              - templateName: snip-success-rate
        - setWeight: 50          # healthy? widen to 50%
        - analysis:
            templates:
              - templateName: snip-success-rate
        - setWeight: 100         # still healthy? full rollout

Read the steps as a sentence: send 10% of traffic to the new version; if analysis passes, go to 50%; if it passes again, go to 100% — and if any analysis fails, abort and roll back. Only ~10% of users ever touch a bad version, for only as long as the analysis takes to notice.

The automated analysis: the SLOs decide

The AnalysisTemplate is where page 5 pays off. It runs the same PromQL we wrote there against the canary’s traffic and declares success or failure. Argo Rollouts uses that verdict to drive the rollout — no human watching a dashboard.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: snip-success-rate
  namespace: snip
spec:
  metrics:
    # 1) Success rate must stay at or above 99% (error budget = 1%).
    - name: success-rate
      interval: 1m
      count: 5                     # take 5 readings, one per minute
      successCondition: result >= 0.99
      failureLimit: 1              # one bad reading aborts the rollout
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc:9090
          query: |
            sum(rate(http_requests_total{app="snip",status!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{app="snip"}[2m]))

    # 2) p99 latency must stay under 250ms.
    - name: p99-latency
      interval: 1m
      count: 5
      successCondition: result <= 0.25
      failureLimit: 1
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc:9090
          query: |
            histogram_quantile(0.99,
              sum(rate(http_request_duration_seconds_bucket{app="snip"}[2m])) by (le))

This is SLOs and error budgets made executable. The 99% success target is an SLO; successCondition: result >= 0.99 is that SLO encoded as a gate. A new version that burns the error budget — pushes success below 99% or p99 above 250ms even once across the readings — fails analysis, and the rollout aborts itself. The error budget stops being a number in a doc and becomes the thing that literally decides whether code reaches all your users.

This is progressive delivery

A canary that promotes or rolls back automatically on metrics — rather than on a human clicking “continue” — is the core of progressive delivery. The release decision moves from human judgment under pressure to a continuous, data-driven control loop: expose a little, measure, decide, expose more. It’s the reconciliation loop shape again, applied to the act of releasing.

The rollback path

When analysis fails, the rollback is automatic and immediate. Argo Rollouts shifts traffic back to the stable version (which never went away) and stops the rollout. No human is paged to perform the rollback; at most they’re notified that one happened.

   step: setWeight 10  ─► analysis ─► FAIL (success 0.97 < 0.99)
                                        │
                                        ▼
                              abort: shift 100% back to stable
                              (only the ~10% slice ever saw the bad version)

There are three independent ways to take a bad change back, in order of how automatic they are:

Mechanism	Who triggers it	When it applies
Analysis abort	Argo Rollouts, automatically	Canary metrics breach an SLO mid-rollout
kubectl argo rollouts undo	A human, immediately	Caught a problem after full promotion
git revert	A human, via the GitOps repo	The proper, attributed, durable rollback

# Watch the canary make its decision in real time.
kubectl argo rollouts get rollout snip --watch

# If something slips past the canary, the GitOps rollback is a revert:
git revert <bad-config-commit>     # Argo CD reconciles the cluster back to good
git push

Two layers of safety, not one

The canary catches problems the metrics can see during the rollout — automatically, in minutes. The git revert path catches everything else, durably, with an audit trail. Defense in depth: the fast automatic layer handles the common case, and the deliberate git layer is always there as the floor. You never rebuild the old state from memory; it’s the previous commit, which git never lost.

The thread, completed

This is the book’s recurring thread at its sharpest, on the riskiest action in the whole lifecycle. The manual, error-prone steps removed are the big-bang cut-over (replaced by a 10%→50%→100% drip) and the human watching a dashboard with a finger on the rollback (replaced by automated analysis on the exact SLOs from page 5). Production gets safer in a way no amount of pre-deploy testing can match: the new version is judged in production, on real traffic, but by so few users and for so short a time that a bad version is a non-event — caught by the canary and rolled back before most people could notice. The cost is the machinery: a metrics stack good enough to trust, well-chosen SLOs, and the traffic-splitting setup. That cost is exactly what turns “deploy and pray” into “deploy and measure.”

We have now taken one commit all the way to production — built, tested, scanned, signed, deployed, observed, and canaried, with no human touching prod. The last page steps back: what we deliberately left out, and where to go from here. Where to Go Next.

Check your understanding

1. The trigger and the execution of a canary deploy are two different things. What triggers it, and what executes it — and why is keeping the trigger in git valuable?
2. Walk through the 10% → 50% → 100% rollout. At each pause, what decides whether it proceeds, and what happens to traffic if the decision is “no”?
3. The AnalysisTemplate reuses the page-5 PromQL. How does this turn an SLO and its error budget from a number in a document into something that controls a deploy?
4. There are three rollback mechanisms. Order them by how automatic they are, and say when each is the right one.
5. Why is a metric-gated canary safer than even a very thorough test suite? What can it catch that tests structurally cannot?

Show answers

1. The trigger is a git commit bumping the image tag in the snip-config repo (Argo CD syncs it); the execution is Argo Rollouts performing the gradual, metric-gated canary. Keeping the trigger in git means the deploy is auditable, reviewable, and revertable — a normal git operation, not an imperative action against the cluster.
2. At each step, traffic shifts to the new weight, then pauses for an analysis that runs the SLO PromQL against the canary. If success rate stays ≥ 99% and p99 ≤ 250ms, it proceeds to the next weight; if any reading breaches an SLO, the rollout aborts and shifts 100% of traffic back to the stable version — so only the ~10% slice ever saw the bad version.
3. The SLO (≥99% success) becomes successCondition: result >= 0.99 querying live traffic; a version that burns the error budget fails analysis and is rolled back. The budget stops being documentation and becomes the literal gate deciding whether code reaches all users.
4. Most automatic: analysis abort (Argo Rollouts, automatic, during a canary). Then kubectl argo rollouts undo (a human, immediately, after full promotion). Then git revert (a human, via the GitOps repo — the proper, attributed, durable rollback). Use the canary for in-rollout SLO breaches, the undo for a fast manual catch, and the revert for the auditable record.
5. Tests run on synthetic inputs in a controlled environment; the canary judges the new version on real production traffic, data, and load — the things tests can’t fully reproduce. It catches behavior that only emerges under real conditions, while exposing only a tiny slice of users for a short time, so the discovery is cheap.