The Control Loop Is Everywhere

The reconciliation loop showed you the single idea that makes Kubernetes Kubernetes: a controller observes actual state, compares it to declared desired state, and acts to close the gap — forever. That page ended with a promise: the idea isn’t confined to Kubernetes. This page makes good on it. The control loop is not a Kubernetes feature; it is the master pattern of operations, and once you have the lens you’ll find the same four steps hiding inside systems that look completely unrelated.

The reason to spend a whole page on a pattern you already know is leverage. If five different tools are secretly the same tool, you don’t have to learn five mental models — you learn one, and then ask each new system the same four questions. That compression is what separates an engineer who memorizes tools from one who understands them.

The shape

Strip every detail and the loop is four steps, running forever:

        ┌───────────────── loop forever ─────────────────┐
        │                                                 │
   ┌──► observe ──► compare ──► (gap?) ──► act ───────────┘
   │    actual       to        no  └──► do nothing
   │    state      desired
   │               state
   └── steady state: keep watching, never exit

Four questions define any instance of it:

1. What is desired? — the declared intent. Replicas, a git commit, a target CPU level, an SLO.
2. What is actual? — the observed reality. Running Pods, the live cluster, current load, current error rate.
3. What is the gap? — the diff between them.
4. What act closes the gap? — the smallest corrective action.

A system that only does steps 1–3 is a monitor. A system that also does step 4 is a controller. The whole point of automation is to push as much as possible from “monitor, then a human acts” to “controller acts on its own.”

Monitor vs controller is the key distinction

The difference between paging a human and self-healing is exactly whether step 4 is automated. An alert is a control loop with the act step delegated to a person. Autoscaling is the same loop with the act step delegated to the platform. Deciding which steps a human stays in is one of the most important design choices in operations.

The same loop, five disguises

Here is the master pattern wearing five costumes. The columns are the four questions.

System	Desired	Actual	Act to close the gap
Kubernetes reconciliation	replicas: 4	Pods running now	Create/delete Pods
GitOps	The git repo’s manifests	Live cluster state	Apply the diff to the cluster
Autoscaling	Target CPU / RPS per instance	Current load	Add/remove instances
Alerting	An SLO / threshold	A live metric	Fire (or resolve) an alert → page a human
DNS health checks	”Only healthy endpoints in rotation”	Per-endpoint probe results	Add/remove records from the answer

Look down the table and the costumes fall away. GitOps is the reconciliation loop with git as the desired state and the cluster as the actual — that’s why the GitOps page kept calling it one. Autoscaling is the loop with load as the input and instance count as the lever. Alerting is the loop where the corrective action is “wake someone up.” And a DNS health check is a loop that keeps the set of answers it hands out equal to the set of backends that are actually healthy.

Why the costumes matter less than the shape

Notice what you can now do. Faced with a flaky autoscaler, you don’t reach for autoscaler trivia — you ask the four questions. Is the desired target sane? Is the actual metric being measured correctly and promptly? Is the gap calculation right? Is the act (scaling) too aggressive or too slow? That diagnostic checklist is identical whether the loop is scaling instances, reconciling a cluster, or rotating DNS records, because it is the same loop.

Where loops go wrong — and they all go wrong the same ways

Because every control loop has the same shape, it has the same failure modes. Knowing them once means knowing them everywhere.

Stale observation. If observe reads old data, the loop acts on a world that no longer exists. A health check with a 60-second interval keeps a dead backend in DNS rotation for up to a minute. An autoscaler reading a metric that lags by two minutes scales for traffic that’s already gone.

Oscillation (flapping). If act overcorrects, the next observe sees a gap in the other direction, and the loop swings back and forth. An autoscaler that adds ten instances, sees load drop, removes twelve, sees load spike, adds fifteen — thrashing instead of settling. The cure is hysteresis: cool-down periods, dead-bands, and damped responses that make the loop converge instead of ring.

Fighting loops. Two controllers with overlapping act steps and different desired states will undo each other forever. A classic: a HorizontalPodAutoscaler sets replicas based on CPU while a GitOps agent reconciles replicas back to the number hard-coded in git. Each loop is correct; together they flap. (The fix is to make the field one loop’s responsibility and tell the other to ignore it.)

   STALE OBSERVE          OSCILLATION              FIGHTING LOOPS
   ─────────────          ───────────              ──────────────
   act on a world         over-correct →           loop A: "replicas=10"
   that already           gap flips →               loop B: "replicas=3"
   changed                over-correct back →       each "fixes" the other
                          ...ring forever            ...forever
   fix: fresh, fast       fix: hysteresis,          fix: one owner per
        observation            damping, cool-downs       field of desired state

Every loop needs a damper and a clock

The two most common production incidents involving control loops are flapping (no damping) and acting on stale data (no freshness guarantee). When you build or debug any loop, ask explicitly: how fresh is the observation, and what stops the action from overcorrecting? A loop without answers to both is an outage waiting for a trigger.

Under the hood — this is literally control theory

The vocabulary on this page — hysteresis, damping, dead-bands, oscillation — isn’t borrowed by analogy; it’s lifted straight from control theory, the formal discipline behind thermostats, cruise control, and autopilots. A thermostat is the canonical control loop: desired = the setpoint, actual = the room temperature, act = run the heater. The reason it doesn’t switch on and off every second is the same fix this page prescribes — a dead-band, so it won’t act until the gap exceeds a threshold. The industrial workhorse is the PID controller (proportional-integral-derivative), which damps a loop by responding not only to the current gap (P) but to its accumulation over time (I) and its rate of change (D), so the system settles smoothly instead of ringing. When you tune an autoscaler’s cool-downs you’re doing PID tuning by another name — which is exactly why the failure modes here are generic: they are the failure modes of feedback control itself, studied for over a century.

The thread

The manual, error-prone step the control loop removes is a human standing watch — eyes on a dashboard, ready to react when reality drifts from intent. That job is tedious, slow, and unreliable precisely because it’s human: people blink, sleep, and misjudge. A control loop is a tireless stand-in that never looks away. The safety it buys is continuous correction: the gap between desired and actual is closed in seconds, around the clock, without anyone being awake — which is exactly why the loop, not the human, is what makes modern systems self-healing.

And the cost, stated honestly: a loop that acts on its own can be confidently wrong at machine speed. A misconfigured autoscaler can scale you into a huge bill; a fighting pair of loops can flap a service into the ground faster than any human could. The lesson of this page is that those failures are generic — so you can guard against all of them with one set of habits.

The rest of this Part is, quietly, this loop in new costumes. The next page puts a loop between your services: a service mesh, where sidecars continuously reconcile traffic policy, retries, and encryption that used to live in your application code.

The architect’s lens

Five questions for the master pattern itself:

• Why does it exist? Because a human standing watch — eyes on a dashboard, ready to react when reality drifts from intent — is tedious, slow, and unreliable, since people blink, sleep, and misjudge.
• What problem does it solve? It’s the one pattern (observe → diff desired vs actual → act → repeat) behind Kubernetes, GitOps, autoscaling, alerting, and DNS health checks — automating step 4 turns a monitor into a self-healing controller that closes the gap in seconds, around the clock.
• What are the trade-offs? An automated act can be confidently wrong at machine speed, and every loop shares the same failure modes — stale observation, oscillation/flapping, and fighting loops — so each needs a damper (hysteresis) and a clock (fresh observation).
• When should I avoid automating the act? When the corrective action is high-risk or irreversible — keep a human in the act step (an alert, not auto-remediation) where a machine-speed mistake would be catastrophic.
• What breaks if I remove it? You’re back to a human on watch who must never look away — the gap between desired and actual closes only as fast, and as reliably, as a person can react.

Check your understanding

1. State the four questions that define any control loop. Which one separates a monitor from a controller?
2. GitOps, autoscaling, and alerting are “the same loop in different costumes.” For each, name what plays the role of desired and actual.
3. What is oscillation (flapping), what causes it, and what general technique cures it?
4. Describe the “fighting loops” failure mode with a concrete Kubernetes example, and state the fix in terms of ownership of a field.
5. Using the book’s thread, what manual job does the control loop remove, and what is the honest cost of automating the act step?

Show answers

1. What is desired? What is actual? What is the gap? What act closes it? Automating the act step is what turns a monitor (observe/compare only) into a controller (also acts).
2. GitOps: desired = the git repo’s manifests, actual = the live cluster. Autoscaling: desired = a target CPU/RPS, actual = current load. Alerting: desired = an SLO/threshold, actual = a live metric (and the act is paging a human).
3. Oscillation is the loop swinging back and forth, caused by the act step overcorrecting so the next observation sees a gap in the opposite direction. The cure is hysteresis — cool-downs, dead-bands, damped responses — so the loop converges instead of rings.
4. An HPA sets replicas from CPU while a GitOps agent reconciles replicas back to the value in git; each loop is individually correct but they undo each other forever. The fix is to give one loop ownership of the field and tell the other to ignore it.
5. It removes a human standing watch over a dashboard. The cost is that an automated act step can be confidently wrong at machine speed — a bad loop can flap a service down or scale you into a huge bill faster than a human ever could.