GitOps with Argo CD

GitOps described the pattern: git is the desired state, and an in-cluster agent continuously reconciles reality to match. Argo CD is the most widely used implementation of that agent on Kubernetes — and the satisfying part is that it isn’t a new idea at all. Argo CD is a Kubernetes controller: it runs the exact same reconciliation loop the cluster already runs for built-in resources, except its “desired state” lives in a git repo and its “actual state” is the live cluster.

Argo CD is the control loop, pointed at git

Recall the loop from Part 4: a controller observes desired state, observes actual state, computes the difference, and acts to close it — forever. Argo CD slots straight into that template:

   observe DESIRED          observe ACTUAL         diff & act
   ───────────────          ──────────────         ──────────
   git repo (manifests) ──► live cluster objects ──► if different:
        ▲                          ▲                   apply git → cluster
        │  pull every few          │  watch K8s API     (this is a "sync")
        └─ minutes ────────────────┴────────────────►  loop forever

Because it’s “just a controller,” everything you know about the reconciliation loop transfers directly. There is nothing exotic here — same loop you’ve already seen, new source of truth.

The Application resource

Argo CD is configured the Kubernetes way: through a custom resource called an Application. An Application is a small declarative object that says “keep this place in git reconciled into this place in the cluster.” It has three essential parts: source (which repo, path, and revision holds the manifests), destination (which cluster and namespace they belong in), and a sync policy (how aggressively to reconcile).

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: snip
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/acme/snip-config.git   # the desired state
    targetRevision: main                                # track this branch
    path: k8s/overlays/prod                             # manifests live here
  destination:
    server: https://kubernetes.default.svc              # which cluster
    namespace: snip                                     # which namespace
  syncPolicy:
    automated:
      prune: true        # delete cluster objects removed from git
      selfHeal: true     # revert manual cluster changes back to git
    syncOptions:
      - CreateNamespace=true

Read it as a sentence: track main in the snip-config repo at k8s/overlays/prod, and keep the snip namespace continuously matched to it — pruning what git removes and self-healing what humans change. The Application is itself a manifest, so it too can live in git: GitOps managing GitOps.

Sync and health: the two questions Argo CD answers

Argo CD continuously reports two independent statuses, and keeping them separate is key to reading it.

• Sync status — does the cluster match git? Synced means actual equals desired. OutOfSync means they differ: git has changes the cluster hasn’t picked up, or someone changed the cluster out of band.
• Health status — are the running objects actually working? Argo CD checks each resource’s real health (a Deployment’s pods Ready, a rollout progressing) and reports Healthy, Progressing, Degraded, etc.

   Sync:   Synced ────────► cluster == git
           OutOfSync ─────► drift, or new commit not yet applied
   Health: Healthy ───────► the workload is actually running fine
           Degraded ──────► it's deployed but broken

They’re orthogonal, and that’s the point: a deploy can be Synced but Degraded (you successfully applied a broken config — exactly what you want to see, fast), or OutOfSync but Healthy (running fine, but someone hand-edited it and it no longer matches git). You need both to know the real state.

Automated sync, drift detection, and rollback

With syncPolicy.automated set (as above), Argo CD closes the loop without a human:

• Automated sync — merge a commit to main and Argo CD detects the new desired state and applies it. The deploy is the merge; nobody runs kubectl. This is GitOps’s “a deploy is a git commit” made literal.
• Drift detection + self-heal — selfHeal: true means a manual kubectl edit makes the app go OutOfSync, and Argo CD reverts the cluster back to git. The cluster cannot durably drift from the declared state; the only lasting way to change it is to change git.
• Prune — prune: true means deleting a manifest from git deletes the corresponding object from the cluster, so git stays a complete description, with no orphaned leftovers accumulating.

Rollback is a git revert

A bad change is in production. You don’t open the Argo CD UI and click around the cluster — you git revert the offending commit. The desired state returns to the previous, known-good revision; Argo CD sees the cluster is now OutOfSync and reconciles it back. Recovery is a normal, reviewed, attributed git operation against a state git never lost — not a frantic manual rebuild.

# Roll production back to the last good state — by reverting in git.
git revert <bad-commit-sha>
git push                     # Argo CD detects OutOfSync and reconciles
# (or force an immediate sync instead of waiting for the poll)
argocd app sync snip

Why this is the safe end-state

This is the book’s thread at full strength, and a fitting close to the Part. Argo CD removes the most dangerous manual steps that survived even into early automation: a human (or a CI server holding prod credentials) reaching into the cluster to apply changes; hand edits that silently drift the cluster from what was reviewed; and rollbacks reconstructed under incident pressure. In their place is one durable property — the cluster is continuously and automatically pulled toward a reviewed, version-controlled declaration of what it should be. Drift self-heals, the audit trail is git log, rollback is git revert, and CI never needs cluster credentials because the controller pulls from inside. Production stops being something people do things to and becomes something that converges, on its own, to what git says.

By the numbers — the drift window

“Continuously reconciled” has a clock on it. By default Argo CD re-checks each Application on a polling interval of about 3 minutes (180 s):

git push  → (up to ~3 min poll) → detect OutOfSync → sync
git webhook → near-instant       → detect OutOfSync → sync

So an out-of-band kubectl edit can persist for up to a few minutes before self-heal reverts it, and a merged commit can wait that long before rolling out — unless you wire a webhook from your git host, which collapses the delay to seconds. The number matters operationally: it’s the worst-case age of any drift, and the difference between “my deploy didn’t happen” and “my deploy is 90 seconds into the poll cycle.”

Under the hood — when self-heal fights the autoscaler

selfHeal: true reverts anything that doesn’t match git — which collides head-on with controllers that are supposed to mutate the cluster. The classic case: a HorizontalPodAutoscaler scales a Deployment to 9 replicas under load, but git says replicas: 3. Argo CD sees drift and reverts to 3; the autoscaler scales back up; they fight, flapping the workload. The fix isn’t to disable self-heal — it’s to tell Argo CD which fields are legitimately owned by something else, via ignoreDifferences (here, the Deployment’s spec.replicas). The deeper lesson: GitOps reconciliation has to share the cluster with the autoscalers and other controllers running their own loops, so “the desired state” sometimes means “everything except the fields I delegate.”

The architect’s lens

Five questions to place Argo CD rather than just operate it:

• Why does it exist? Because GitOps needs an agent inside the cluster to run the reconciliation loop — and Argo CD is just a Kubernetes controller whose desired state happens to live in a git repo instead of etcd.
• What problem does it solve? Invisible drift and out-of-band deploys: it continuously pulls manifests from git, reports Synced/Healthy as orthogonal signals, and with selfHeal reverts a manual kubectl edit back to the declared state.
• What are the trade-offs? The default ~3-minute poll means a merge — or a drift — can persist for minutes unless you wire a git webhook; and selfHeal: true fights other controllers (the HPA-vs-replicas flap) until you carve out the delegated fields with ignoreDifferences.
• When should I avoid it? When you’re not on Kubernetes, or for genuinely imperative one-off operations where a standing reconciliation loop is machinery you don’t need.
• What breaks if I remove it? Deploys go back to a human (or CI) running kubectl with prod credentials, drift stops self-healing, and rollback becomes a hand-rebuild instead of git revert.

Check your understanding

1. Why is it accurate to say Argo CD “is just a Kubernetes controller”? Map its behavior onto observe-desired / observe-actual / diff-and-act.
2. What are the three essential parts of an Argo CD Application, and what does each specify?
3. Sync status and health status are independent. Give a realistic example of Synced + Degraded and of OutOfSync + Healthy, and explain why you need both signals.
4. What do selfHeal: true and prune: true each do, and what would silently go wrong if each were off?
5. Walk through rolling back a bad production change with Argo CD. Why is git revert safer than manually re-applying the old config by hand?

Show answers

1. It runs the same control loop as any Kubernetes controller: observe desired = pull manifests from the git repo; observe actual = watch the live cluster objects via the K8s API; diff and act = if they differ, apply git to the cluster (a “sync”), forever. The only twist is that desired state lives in git rather than in etcd.
2. source (which repo, path, and revision hold the manifests), destination (which cluster and namespace they belong in), and sync policy (how aggressively to reconcile — e.g. automated, prune, selfHeal).
3. Synced + Degraded: you successfully applied a manifest, but the workload is broken (pods crash-looping) — exactly the failure you want surfaced fast. OutOfSync + Healthy: the app runs fine, but someone hand-edited the cluster so it no longer matches git. You need both because sync answers “does it match git?” and health answers “is it actually working?” — orthogonal questions.
4. selfHeal: true reverts manual cluster changes back to git; without it, a hand edit would silently persist and the cluster would drift from the declaration. prune: true deletes cluster objects removed from git; without it, deleting a manifest leaves an orphaned object running that git no longer describes.
5. You git revert <bad-commit-sha> and push; Argo CD sees the cluster is now OutOfSync and reconciles it back to the previous, known-good revision (or you force argocd app sync). It’s safer than manual re-applying because it’s a normal, reviewed, attributed git operation against a state git never lost — not a frantic hand-rebuild from memory.