Controllers & Informers

The reconciliation loop gave you the soul of Kubernetes: observe, compare, act, forever. The API machinery page showed how state is written and how watches stream changes. This page joins them — it is the engineering of a controller. “Observe the state of the world” sounds simple until you have ten thousand Pods and a controller that must not melt the apiserver by polling it in a hot loop. The informer machinery is the answer to how do you watch everything efficiently, react reliably, and never lose an update?

The manual step a controller removes is you, the human operator, watching dashboards and fixing drift by hand. But a naïve automation of that — a while true loop that lists every object every second — would hammer the apiserver and fall over at scale. The real cleverness, and this page’s subject, is the cache-and-queue plumbing that makes “watch everything, react to everything” cheap, correct, and crash-safe.

A controller is a reconciliation loop — with real plumbing

Strip a controller to its essence and it’s the three steps you already know. But a production controller doesn’t call the apiserver to observe; that would be ruinously slow and would not survive ten thousand objects times a hundred controllers. Instead it observes from a local, in-memory cache that is kept current by a single watch. The whole pipeline exists to feed and drain that cache safely.

   apiserver ──watch──► ┌─────────── INFORMER ───────────┐
   (LIST + WATCH)       │ reflector → Delta FIFO → cache  │ ──► lister (read-only,
                        │              │  (thread-safe    │      in-memory view)
                        │              │   store)         │
                        │              ▼                  │
                        │         event handlers ─────────┼──► workqueue ──► reconcile()
                        └─────────────────────────────────┘   (rate-limited)   │
                                                                 ▲             │
                                                                 └── requeue ──┘
                                                                   (on error / not-done)

Let’s walk each stage, because the names are exactly what you’ll see in logs and source.

The informer pipeline, stage by stage

Reflector — the component that holds the watch. It does a LIST to get a full baseline plus a resourceVersion, then opens a WATCH from that version and streams every ADDED/MODIFIED/ DELETED event. If the watch expires (the 410 Gone / “too old” case from the API machinery page), the reflector transparently re-LISTs and resumes. One reflector per resource type does all the API traffic — not per object, not per controller.

Delta FIFO — a queue of deltas (the changes the reflector produced), keyed by object, that preserves order and coalesces. It is the buffer between “events arriving from the watch” and “the cache being updated.” Crucially it deduplicates: several updates to the same object that haven’t been processed yet collapse appropriately, so a busy object doesn’t flood the pipeline.

The cache (thread-safe store) — as deltas are popped from the FIFO, they’re applied to a local indexed store: a full, in-memory replica of every object of that type. This is the controller’s view of “actual state,” and it’s read without ever touching the apiserver.

Lister — the read-only, indexed accessor over the cache. When reconcile() needs to ask “what Pods match this label selector right now?”, it asks the lister, which answers from memory in microseconds.

Event handlers → workqueue — the informer lets you register OnAdd/OnUpdate/OnDelete handlers. These handlers do almost nothing: they compute the key (namespace/name) of the affected object and add it to a workqueue. That’s the whole job. The actual work happens later, when a worker pops the key and calls reconcile().

Why handlers only enqueue a key — never do the work

This is the design decision that makes controllers robust. Handlers run synchronously on the event stream; if they did heavy work they’d block the stream and serialize everything. By enqueuing only a key, the controller (a) decouples event rate from work rate, (b) deduplicates — ten events for the same object become one queued key — and (c) can retry that key independently. The workqueue is where all the reliability lives.

Level-triggered, not edge-triggered

Here is the single most important design property, and it follows directly from “enqueue a key, not an event.” An edge-triggered system reacts to the change (“a Pod was deleted!”). A level-triggered system reacts to the current state (“for key web-7d4, what should be true, and is it?”). Kubernetes controllers are level-triggered.

Why does this matter so much? Because edges get lost. If a controller is restarting when an event fires, an edge-triggered design misses it forever. A level-triggered controller doesn’t care what changed or how many events it missed — when it processes key web-7d4, it reads the current desired and actual state from the lister and computes the correct action from scratch.

   EDGE-TRIGGERED (fragile)            LEVEL-TRIGGERED (robust — Kubernetes)
   "react to the event"                "react to current state for this key"
   pod deleted → handle deletion       reconcile(web): read desired=4, actual=3,
   (miss the event = stuck forever)    act to make actual match. Missed events
                                       are irrelevant; you recompute every time.

This is why a controller can crash, restart, re-LIST the whole world, and converge correctly — it never relied on having seen every individual edge. It’s also why reconcile() must be idempotent: it may run for the same key many times with nothing to do, and that must be harmless.

Resync: a safety net, not the main mechanism

The informer can be configured with a resync period (say, 10 minutes). On resync, it re-delivers every object in its cache to the handlers as a synthetic update — re-enqueuing every key. This is not how the controller normally learns of changes (the watch does that in real time). Resync exists as a backstop: if a bug ever dropped a key, or a reconcile silently no-op’d when it shouldn’t have, the next resync forces a fresh reconcile of everything. It’s a periodic “check the whole world again,” made cheap because it reads from the cache, not the apiserver. Set it too low and you waste CPU re-reconciling steady state; too high and you widen the window before a missed key self-heals.

Leader election: exactly one active replica

You run controllers in multiple replicas for availability. But two replicas of the same controller both creating Pods for the same Deployment would be chaos. The fix is leader election: the replicas contend for a lock — a Lease object in the API (coordination.k8s.io/v1) — and only the holder is active. The others stand by, watching the lease.

   replica A ──┐
   replica B ──┼─► race to acquire Lease "web-controller"
   replica C ──┘
   A wins → renews the lease every few seconds (renewDeadline)
   A crashes → stops renewing → lease expires (leaseDuration)
   B/C see it expired → race again → B becomes leader, resumes reconciling

The lease carries a holderIdentity, a leaseDurationSeconds, and a renewTime. The leader renews continuously; if it dies, the lease lapses and a standby takes over within leaseDuration. This buys HA without split-brain: exactly one writer at a time, automatic failover, no human paging required. The cost is a few seconds of reconcile pause during a failover — almost always worth it.

Rate limiting and backoff: the workqueue’s other job

A workqueue isn’t a plain queue. It’s a rate-limited queue, and this is what keeps a controller from turning a transient error into a self-inflicted denial of service. When reconcile() fails for a key, the controller calls AddRateLimited(key), which re-enqueues it after an exponentially increasing delay — typical defaults start around a few milliseconds and back off to a cap (often ~1000 seconds). A second limiter caps the overall enqueue rate (a token bucket) so no single hot object can starve the rest.

   reconcile(web) fails → AddRateLimited(web)
     attempt 1: requeue after   5ms
     attempt 2: requeue after  10ms
     attempt 3: requeue after  20ms ... doubling ... up to a cap (~1000s)
   reconcile(web) succeeds → Forget(web)   (resets the per-key backoff counter)

Forget(key) is the matching call on success: it clears the backoff so the next failure starts from the bottom again. Forget on success, backoff on failure — that pair is the contract every well-written controller follows.

A worked controller loop

Here is the whole shape, in Go-flavored pseudo-code, with no detail that isn’t real. Read it against the diagram at the top.

// Setup: one informer feeds a cache (lister) and a workqueue.
informer.AddEventHandler(ResourceEventHandlerFuncs{
    AddFunc:    func(o any) { queue.Add(keyOf(o)) },       // enqueue a KEY,
    UpdateFunc: func(_, n any) { queue.Add(keyOf(n)) },    // never do work here
    DeleteFunc: func(o any) { queue.Add(keyOf(o)) },
})

// Each worker goroutine runs this forever:
func worker() {
    for {
        key, shutdown := queue.Get()      // blocks until a key is available
        if shutdown { return }
        err := reconcile(key)             // the observe → compare → act step
        if err != nil {
            queue.AddRateLimited(key)      // failed: requeue with backoff
        } else {
            queue.Forget(key)              // succeeded: clear the backoff
        }
        queue.Done(key)                    // mark this key as no longer in-flight
    }
}

// The actual logic — LEVEL-TRIGGERED and IDEMPOTENT:
func reconcile(key string) error {
    ns, name := split(key)
    desired, err := lister.Deployments(ns).Get(name)  // read from CACHE, not apiserver
    if errors.IsNotFound(err) {
        return nil                          // object gone; nothing to do — harmless
    }
    actual := countRunningPods(lister, desired.Selector)
    switch {
    case actual < desired.Replicas:
        return createPods(desired.Replicas - actual) // act: close the gap up
    case actual > desired.Replicas:
        return deletePods(actual - desired.Replicas) // act: close the gap down
    default:
        return nil                          // already converged — do nothing
    }
}

Notice three things. reconcile reads from the lister (the cache), so the hot path never hits the apiserver. It recomputes the correct action from current state every call — level-triggered — so a missed event or a duplicate call is irrelevant. And it returns nil when there’s nothing to do, which is the common case and must be cheap.

The thread, named precisely

The manual step removed is a human watching for drift and hand-correcting it. A controller automates that — but the informer/workqueue machinery is what makes the automation safe at scale: it watches via one efficient connection instead of polling, survives restarts because it’s level-triggered, never double-acts because of leader election, and never amplifies an outage because of rate-limited backoff. The cost is real complexity (caches can be briefly stale; a bug in reconcile runs everywhere) — which is exactly why you reach for a framework rather than writing this plumbing by hand.

You almost never write this from scratch

Everything above is provided by controller-runtime and client-go — informers, listers, workqueues, leader election, and rate limiters are libraries, not things you reimplement. When you build a controller for your own resource type, you define a Custom Resource Definition and write only the reconcile function; the framework supplies the rest of the pipeline. That is the operator pattern, and it’s the payoff this page has been building toward: Kubernetes Operators & CRDs takes the machinery you just learned and turns it into a way to teach Kubernetes about your application’s desired state. The same shape — observe, compare, act — scales from a built-in ReplicaSet controller to an operator that manages a whole database cluster, a pattern so general the book devotes a page to it: The Control Loop Is Everywhere.

Next, we follow the work a controller creates downstream: a Pod the ReplicaSet controller just made is unscheduled, and the scheduler — itself a controller built on exactly this machinery — must decide where it runs.

By the numbers — how far the workqueue backs a wedged key off

The default rate limiter starts a failing key at 5 ms and doubles on each retry, capped at 1000 s. Solve for how many failures it takes to reach that cap:

   delay(n) = 5 ms × 2^(n−1),  capped at 1000 s
   5 ms × 2^(n−1) ≥ 1,000,000 ms  →  2^(n−1) ≥ 200,000  →  n ≈ 18
   so by ~the 18th straight failure the key is retried only every ~1000 s (≈16.7 min)

A permanently broken object therefore settles to roughly three or four reconcile attempts per hour — not a hot loop. That is the entire point: one wedged key cannot turn a transient error into a self-inflicted flood against the apiserver, while a key that recovers calls Forget and snaps straight back to 5 ms.

The architect’s lens

Step back from the pipeline and answer the five questions about the pattern every Kubernetes controller is built on:

• Why does it exist? Because the manual step is a human watching for drift and hand-correcting it — but a naïve automation (a while true listing every object every second) would melt the apiserver at ten thousand Pods. The informer’s cache-and-queue plumbing makes “watch everything, react to everything” cheap and crash-safe.
• What problem does it solve? One reflector does a single LIST+WATCH that feeds a local in-memory cache (so reconcile() reads from the lister in microseconds, never the apiserver), and a rate-limited workqueue of keys decouples event rate from work rate, deduplicates, and retries each key independently.
• What are the trade-offs? The cache can be briefly stale; a bug in reconcile runs everywhere; a permanently failing key backs off to roughly once per ~1000 s (≈18 doublings from 5 ms) — robust, but slow to recover; and leader election costs a few seconds’ reconcile pause on failover.
• When should I avoid it? Avoid hand-rolling it — almost always. controller-runtime/client-go ship informers, listers, workqueues, leader election, and limiters as libraries; you write only an idempotent, level-triggered reconcile (react to current state for a key, never to the edge), the payoff being the operator pattern.
• What breaks if I remove it? Drift correction reverts to a human; an edge-triggered design loses an event across a restart and stays stuck forever; without leader election two replicas double-act on the same Deployment; and without rate-limited backoff a transient error becomes a self-inflicted DoS on the apiserver.

Check your understanding

1. Why does a real controller read from a local cache (the lister) instead of calling the apiserver inside reconcile(), and what component keeps that cache current?
2. Explain level-triggered vs edge-triggered. Why does the level-triggered choice let a controller crash, restart, and still converge correctly — and what property must reconcile() therefore have?
3. The informer’s event handlers do almost nothing — they just enqueue a key. List two concrete benefits that come from “enqueue a key, do the work later in a workqueue.”
4. What problem does leader election solve, what object backs the lock, and what’s the cost when the leader fails?
5. A reconcile keeps failing for one key. Trace what the rate-limited workqueue does on each failure and on eventual success, and explain why this protects the apiserver.

Show answers

1. Calling the apiserver in the hot path would be far too slow and would overload it at scale (thousands of objects times many controllers). The lister reads from an in-memory cache instead. The informer (specifically its reflector doing one LIST+WATCH, feeding a Delta FIFO into the cache) keeps that cache current.
2. Edge-triggered reacts to the change event; level-triggered reacts to the current state for a key. Level-triggered survives crashes because, on restart, it re-LISTs the world and recomputes the correct action from current desired vs actual — it never depended on having seen every individual edge (which could be lost). Therefore reconcile() must be idempotent: running it repeatedly with nothing to do must be harmless.
3. Any two of: it decouples event rate from work rate (heavy work doesn’t block the watch stream); it deduplicates (many events for one object collapse to a single queued key); and it enables independent retry with backoff per key. (The handler also stays fast, keeping the event stream flowing.)
4. It ensures exactly one active replica so multiple replicas don’t both act on the same objects (split-brain). The lock is a Lease object (coordination.k8s.io/v1); the holder renews it continuously and standbys take over if it lapses. The cost is a brief reconcile pause (up to leaseDuration) during failover.
5. On each failure it calls AddRateLimited(key), re-enqueuing after an exponentially increasing delay (doubling up to a cap), and a token-bucket limiter caps overall enqueue rate. On success it calls Forget(key), resetting the per-key backoff. This protects the apiserver because a persistently failing object backs off instead of being retried in a tight loop — preventing a transient error from becoming a self-inflicted DoS.