The Reconciliation Loop

If you remember one idea from this entire Part, make it this one. Every page so far has hinted at it — “declare desired state and controllers make it real.” Now we name the mechanism and look straight at it, because it is the thing that makes Kubernetes Kubernetes. Master the reconciliation loop and the forty other concepts in this Part collapse into “oh, that’s just another controller.”

The recurring thread is unusually literal here: the reconciliation loop is the very thing that removes the manual step. The manual step is you, watching and fixing. The loop is a tireless program that watches and fixes in your place — and, crucially, never stops.

Imperative vs declarative — the whole shift in one contrast

Most tools you’ve used are imperative: you give an ordered list of commands and they run, once, top to bottom. docker run, a bash script, clicking through a console — all imperative. The problem is they describe a path, not a destination. If reality drifts after the script finishes — a container dies, someone changes a setting — the script has no idea and does nothing. You have to notice and re-run it.

Kubernetes is declarative: you describe the destination (the desired state) and hand it over. A controller’s job is to make the world match that description and keep it matching, no matter what happens later. You stop writing “do A, then B, then C” and start writing “here is what should be true.”

   IMPERATIVE                         DECLARATIVE (reconciliation)
   "do these steps, once"            "this should be true, always"
   ┌───────────────────┐            ┌─────────────────────────────────────┐
   │ step 1 → step 2 →  │            │   ┌─────────── loop forever ─────┐    │
   │ step 3 → done.     │            │   │ observe actual state          │   │
   │                    │            │   │ compare to desired state      │   │
   │ drift later? it    │            │   │ act to close the gap          │   │
   │ has no idea.       │            │   └───────────────┬───────────────┘    │
   └───────────────────┘            │                   └── repeats forever  │
                                     └─────────────────────────────────────┘

What a controller actually does

A controller is a small program running in the control plane (most live in the controller-manager). Strip away the detail and every one of them runs the same three-step loop:

Observe — read the actual state of the world from the API server (e.g. “how many web Pods are running right now?”).
Compare — diff it against the desired state you declared (e.g. replicas: 4).
Act — take the smallest action that moves actual toward desired (create a Pod, delete a Pod, update a route), then loop.

The loop never exits. It runs again and again, forever. This is why Kubernetes is self-healing and not merely self-installing: it isn’t a one-time setup that drifts, it is a standing intention that is continuously re-asserted.

   You declared:  replicas: 4
   ┌────────── reconcile ──────────┐
   │ observe: 3 Pods running        │  ← a node just died
   │ compare: desired 4, actual 3   │
   │ act:     create 1 Pod          │  ← gap closed
   └────────────────────────────────┘
   ...moments later, the loop runs again:
   ┌────────── reconcile ──────────┐
   │ observe: 4 Pods running        │
   │ compare: desired 4, actual 4   │
   │ act:     nothing to do          │  ← steady state, but still watching
   └────────────────────────────────┘

See it for yourself

The loop is not a metaphor; you can provoke it. Create a Deployment, then fight it:

kubectl create deployment web --image=myapp:1.4.0 --replicas=4
kubectl get pods -l app=web --watch      # watch in one terminal

# In another terminal, delete a Pod — sabotage the actual state:
kubectl delete pod <one-of-the-web-pods>

Watch the first terminal: a replacement Pod appears within seconds. You never asked for a replacement. You declared 4, sabotaged it to 3, and the ReplicaSet controller — observe, compare, act — restored 4. Try to “fix” things by manually deleting Pods and the loop keeps undoing you, because it considers your declared state authoritative, not your manual pokes.

This is the master key to the rest of the Part

Almost everything else you’ll meet is a reconciliation loop with a different “desired” and “actual”:

A Deployment reconciles ReplicaSets toward a target image and replica count (Pods & Workloads).
A Service reconciles routing rules toward “send traffic to the currently-healthy Pods” (Services & Networking).
The Horizontal Pod Autoscaler reconciles replica count toward a target CPU/metric (Scaling & Scheduling).
Operators extend the pattern to your own objects with custom controllers (Kubernetes Operators & CRDs).

And the idea isn’t even confined to Kubernetes. Thermostats, GitOps tools that reconcile a cluster against a Git repo, cruise control — all the same shape. We devote a whole later page to this: The Control Loop Is Everywhere. Once you have the loop, you have a lens you’ll use for the rest of your career.

Under the hood — level-triggered, not edge-triggered

A subtle but load-bearing property: Kubernetes controllers are level-triggered, not edge-triggered. An edge-triggered system reacts to events — “a Pod was deleted” — and acts on that specific transition; miss the event and it is permanently wrong. A level-triggered system reacts to state — “I observe 3 Pods; I want 4” — and acts on the gap, no matter how the gap arose.

This is why the loop is so robust. Controllers watch the API server for efficiency (so they react in milliseconds, not on a slow poll), but the decision is always made by reading the current actual state and comparing it to desired — and they periodically resync the whole state from scratch. So a dropped watch event, a duplicated one, or a controller that crashed and restarted changes nothing: the next reconcile simply observes reality as it is and closes whatever gap remains.

   EDGE-TRIGGERED              LEVEL-TRIGGERED (Kubernetes)
   "a Pod died" → act          "4 wanted, 3 exist" → act
   miss the event → broken     miss an event → next resync still sees 3, fixes it

Edge-triggered automation is a light switch you must flip on exactly the right transition; level-triggered is a thermostat that just keeps reading the room. The deeper machinery — informers, work queues, the shared cache — is in Controllers & Informers.

The architect’s lens

The reconciliation loop is the master key — so judge it as the central design bet it is:

Why does it exist? Because imperative scripts describe a path and run once, so they’re blind to drift the moment they finish. The loop describes the destination and stays responsible for it, re-asserting your desired state forever.
What problem does it solve? It removes you, watching and fixing — a tireless observe→compare→act program restores replicas: 4 within seconds of a node death, no human in the loop and no event you had to catch in time.
What are the trade-offs? The loop treats your declared state as authoritative, so manual pokes get undone — kubectl delete pod just triggers a replacement. You give up imperative control in exchange for self-healing, and you must express intent as desired state rather than steps.
When should I avoid it? For genuinely one-shot, ordered work where “do A then B then C, once” is the actual goal — a migration or batch import — an imperative run is the right shape; a standing loop re-asserting it would be wrong.
What breaks if I remove it? Kubernetes stops being self-healing and becomes merely self-installing: a setup that drifts and stays drifted, because being level-triggered (acting on the gap “4 wanted, 3 exist”) is exactly what survives a dropped watch event, a crash, or a missed edge that an event-driven system would never recover from.

Check your understanding

State the three steps every controller repeats, and explain why the loop never exits.
What is the core difference between an imperative script and a declarative reconciliation loop? Why does drift expose that difference?
In the kubectl delete pod experiment, why does a replacement appear even though you never asked for one — and why does manually deleting Pods feel like fighting the system?
“Automated” and “declarative” sound similar. Using the 3am example, explain why a reconciliation loop is safer than a script that merely automated the same steps.
Name two other Kubernetes features that are “just another reconciliation loop,” and say what plays the role of desired and actual in each.

Show answers

Observe actual state, compare it to declared desired state, act to close the gap — then repeat. It never exits because it’s not a one-time setup but a standing intention that must be continuously re-asserted; the moment it stopped, drift would go uncorrected.
An imperative script describes a path (do A, then B, then C) and runs once; a declarative loop describes the destination and stays responsible for it. Drift exposes the difference: after a script finishes it has no idea the world changed and does nothing, while the loop is still running and corrects the drift.
Because the ReplicaSet controller considers your declared state (replicas: 4) authoritative, not your manual pokes — deleting a Pod drops actual to 3, so observe→compare→act recreates one. Manual deletes feel like fighting the system because the loop keeps undoing them: you’re contradicting the desired state it exists to enforce.
An automated script still just describes steps and runs them for you — once. A reconciliation loop owns the outcome indefinitely. At 3am a script that finished hours ago can’t help when a node dies; the loop is still running and fixes it before you wake. That continuous responsibility is the safety win.
Examples: a Deployment (desired = target image + replica count, actual = the ReplicaSets/Pods that exist); a Service (desired = “route to currently-healthy Pods,” actual = the live membership list); the HPA (desired = target CPU/metric, actual = current utilization and replica count).