The API Machinery

The architecture page gave you the one-line summary: the API server is the only door, etcd is the only truth. That sentence is doing an enormous amount of work, and this page unpacks all of it. Everything else in Kubernetes — every controller, the scheduler, every kubelet, your kubectl — is a client of one HTTP API backed by one consistent store. If you understand the request path through that API and how etcd records the result, you understand the spine of the whole system. Everything downstream is just reacting to what gets written here.

The manual step this machinery removes is the oldest one in operations: scattered, unsynchronized configuration that no two machines agree on. Instead there is exactly one store, one writer, and one audited path to change it. The cost is that this path is the cluster’s hard floor on consistency and throughput — so it pays to know precisely how it behaves under load and contention.

The apiserver is the only thing that touches etcd

This is the single most important architectural fact on the page, so state it plainly: no controller, no kubelet, no scheduler ever connects to etcd. They all go through kube-apiserver, which alone holds etcd’s client credentials. The apiserver is otherwise stateless — it caches some things, but its memory is disposable; restart it and it rebuilds from etcd. You can run three of them behind a load balancer and they behave identically, because none of them owns any truth.

   kubectl ─┐
   scheduler─┤      ┌──────────────── kube-apiserver ───────────────┐     ┌────── etcd ──────┐
   kubelet  ─┼────► │ authn → authz → mutating adm → validate →      │ ───►│  Raft-replicated │
   ctrl-mgr ─┤      │   schema check → validating adm → persist      │     │  MVCC key-value  │
   you (curl)┘      └───────────────────────────────────────────────┘     └──────────────────┘
                    (stateless; restartable; horizontally scalable)        (THE only source of truth)

Why funnel everything through one process? Because the door is where you enforce the invariants: authentication, authorization, admission policy, and schema validation all live in one place. If controllers wrote etcd directly, every one of them would have to re-implement security and validation, and you’d have no single audit log. One door is what makes “who changed this object, when, and were they allowed to?” an answerable question.

etcd: a consistent log, not a database you query

etcd is a distributed key-value store built on the Raft consensus algorithm. Raft elects one member as leader; all writes go to the leader, which appends them to a replicated log and waits for a majority (quorum) of members to acknowledge before committing. This is why production clusters run an odd number of etcd members — 3 or 5 — so a majority survives the loss of one (or two) members.

   write "pods/web-xyz" ─► etcd LEADER ─► append to Raft log entry #4012
                                        ├─► replicate to follower A ─► ack
                                        ├─► replicate to follower B ─► ack   (2 of 3 = quorum)
                                        └─► COMMIT #4012, apply to state, reply OK
   loss of one follower: still 2 of 3 → cluster keeps writing.
   loss of two: no quorum → writes BLOCK (consistency over availability).

Two consequences fall out of this design, and both matter daily:

• Writes are linearizable but not free. Every write is a consensus round trip to a quorum. etcd trades availability for consistency (it’s a CP system in CAP terms): if it can’t reach a majority, it stops accepting writes rather than risk a split brain. A slow disk on the etcd leader slows the whole cluster’s control plane.
• etcd holds the entire cluster. Every object — Deployments, Pods, Secrets, even the observed status — is a key in etcd. Lose etcd without a backup and the cluster is gone, which is why backup and restore is non-negotiable.

resourceVersion, MVCC, and watches

etcd is multi-version: it keeps a global, monotonically increasing revision counter, bumped on every write, and stores historical versions of keys keyed by revision. Kubernetes surfaces this as the metadata.resourceVersion on every object — an opaque string that is really an etcd revision. You must treat it as opaque (don’t do arithmetic on it), but two facts about it drive everything:

1. It lets clients ask “give me everything that changed since revision X” — the basis of watches.
2. It is the token for optimistic concurrency (below).

A watch is a long-lived streaming connection: a client says “watch all Pods from resourceVersion: 184320” and the apiserver streams every ADDED/MODIFIED/DELETED event after that point. This is how the entire system stays in sync without polling. etcd’s MVCC makes it possible — because old revisions are retained, a watcher that reconnects can resume from where it left off instead of re-listing the world.

   client: GET /api/v1/pods?watch=true&resourceVersion=184320
   apiserver streams:
     {type: ADDED,    object: pod web-a  (rv 184321)}
     {type: MODIFIED, object: pod web-a  (rv 184355)}   ← status updated
     {type: DELETED,  object: pod web-b  (rv 184402)}
   ...connection held open, events pushed as they happen.

Compaction and “resourceVersion too old”

etcd’s history is not infinite. Compaction periodically discards revisions older than a threshold to reclaim space. If a watcher is disconnected long enough that its last-seen resourceVersion has been compacted away, the apiserver returns 410 Gone (“resourceVersion too old”). A correct client must then do a fresh LIST to re-establish a baseline and resume watching from the list’s resourceVersion. This LIST-then-watch dance is exactly what an informer automates — and why “expired watch → full relist” is a normal, expected event, not an error.

The request path: authn → authz → admission → validation → etcd

When any request hits the apiserver, it runs a fixed gauntlet. Each stage can reject; only requests that survive all of them reach etcd. Knowing this order is the difference between guessing and diagnosing a rejected apply.

   REQUEST  ──►  ① Authentication   "who are you?"        (certs, tokens, OIDC) → 401 if unknown
            ──►  ② Authorization    "may you do this?"    (RBAC, etc.)          → 403 if denied
            ──►  ③ Mutating admission   webhooks/controllers may CHANGE the object
            ──►  ④ Schema validation    does it match the OpenAPI/CRD schema?   → 422 if malformed
            ──►  ⑤ Validating admission  webhooks/policies may REJECT (no edits)
            ──►  ⑥ Persist to etcd  (compare-and-swap on resourceVersion)
   RESPONSE ◄── the stored object, with a fresh resourceVersion

The split into mutating then validating admission is intentional and load-bearing: defaults and sidecar injection happen first (mutating), then the final object — after all mutations — is checked (validating), so a validating policy never sees a half-mutated object. We give the admission chain its own page, Admission Control & Policy; here the point is just where it sits in the path. Authentication and authorization get the same treatment in Security in Depth.

Optimistic concurrency: why your update sometimes fails with 409

Two controllers can want to write the same object at the same moment. Kubernetes does not lock objects (locks across a distributed fleet would be a nightmare). Instead it uses optimistic concurrency built on resourceVersion. An update carries the resourceVersion the client read; the apiserver performs a compare-and-swap in etcd: write only if the stored version still matches. If someone else wrote first, the versions differ, the swap fails, and you get 409 Conflict.

   ctrl A reads pod (rv=500) ─┐
   ctrl B reads pod (rv=500) ─┤
   ctrl A writes (if rv==500) → OK, now rv=501
   ctrl B writes (if rv==500) → 409 Conflict!  (stored is 501, not 500)
   ctrl B must RE-READ (rv=501), re-apply its change, retry.

This is not a bug to suppress — it’s the system telling you your view was stale. The correct response is re-read and retry, which is exactly why controllers are written as idempotent reconciliation loops: a retry just recomputes the desired action from fresh state. A 409 is the price of lock-free concurrency, and it’s a good trade.

Declarative apply and server-side apply

kubectl apply is supposed to be declarative: you send the whole object you want, and the system reconciles. The old client-side apply computed a three-way merge on your laptop using a last-applied-configuration annotation — fragile, invisible to other clients, and prone to silently clobbering fields it didn’t know about.

Server-side apply (SSA) moves the merge into the apiserver and adds field ownership. Each applier (a fieldManager — your kubectl, the HPA, a controller) is recorded in metadata.managedFields as the owner of the specific fields it sets. The apiserver merges based on this ownership map.

   metadata.managedFields:
     - manager: kubectl    operation: Apply  → owns spec.replicas, spec.template...
     - manager: hpa        operation: Apply  → owns spec.replicas   (CONFLICT!)

Now a real scenario: you set replicas: 3, and the HPA also wants to own replicas. With SSA, a conflicting apply fails loudly unless you pass --force-conflicts to take ownership — instead of the silent last-writer-wins of the old world. Field ownership turns “two things fighting over one field” from an invisible flapping bug into an explicit, diagnosable conflict.

The thread, made precise

The manual, error-prone step here is hand-merging configuration and hoping nobody else touched the same field. Server-side apply removes it: the merge is computed server-side from a recorded ownership map, and contested fields surface as a refusal you can see, not a silent overwrite you discover in an incident. The cost is a more intricate object (managedFields is verbose) — a fair price for losing the “who clobbered my setting?” class of outage.

Finalizers: hooks that delay deletion

When you delete an object, it doesn’t always vanish immediately. If it has a non-empty metadata.finalizers list, the apiserver does not remove it from etcd. Instead it sets metadata.deletionTimestamp (the object enters a “terminating” state) and waits. Each finalizer names a controller that must do cleanup — drain a load balancer, delete a cloud disk, deregister DNS — and then remove its own finalizer string. Only when the list is empty does the apiserver actually delete the key.

   kubectl delete svc web
   → apiserver sets deletionTimestamp, object STAYS (finalizers: [service.k8s.io/lb])
   → cloud controller tears down the load balancer
   → cloud controller removes "service.k8s.io/lb" from finalizers
   → finalizers now [] → apiserver deletes the key from etcd → object gone

This is why an object sometimes appears “stuck Terminating”: a finalizer’s controller is down or wedged and never removes its entry. Force-deleting by editing out the finalizer skips the cleanup — which is exactly the leaked cloud resource the finalizer existed to prevent. Finalizers are the safety interlock between “delete the record” and “delete the real-world thing the record represents.”

ownerReferences and garbage collection

Kubernetes objects form a tree. A Deployment owns a ReplicaSet; the ReplicaSet owns Pods. This is recorded in each child’s metadata.ownerReferences, pointing up at its parent. The garbage collector (a controller in the controller-manager) watches this graph and deletes orphans.

   Deployment "web"  ───owns───►  ReplicaSet "web-7d4"  ───owns───►  Pod web-7d4-aa
                                                                     Pod web-7d4-bb
   delete the Deployment → cascade: GC deletes the RS → GC deletes the Pods

Deletion has three propagation policies, and the difference is operationally real:

Policy	What it deletes	When you’d use it
Foreground	Owner waits in “terminating” until all children are gone first	You need cleanup ordered child-first
Background (default for most)	Owner deleted immediately; GC sweeps children after	The common case — fast, eventually consistent
Orphan	Owner deleted; children keep running, ownerReference removed	Detach Pods from a controller deliberately

ownerReferences are also how Foreground deletion uses a finalizer (foregroundDeletion) to hold the parent until children clear. Two mechanisms you just learned — finalizers and owner references — compose to give cascade delete its ordering guarantees. This is the cleanup you would otherwise do by hand (and forget), encoded as a controller that can’t forget.

Putting it together

Trace one kubectl apply -f deploy.yaml through everything above: it authenticates you, authorizes you via RBAC, runs mutating then validating admission, validates against the schema, and writes to etcd via a compare-and-swap on resourceVersion, which bumps the global revision. That write becomes a watch event streamed to every interested client — and the first of those clients is a controller, which is exactly where the next page begins. The apiserver wrote the desired state; now something has to make it real.

By the numbers — what actually fits in etcd

etcd is sized for configuration, not bulk data, and two limits make that concrete. Each value is bounded by etcd’s request size, and the whole store has a finite budget:

   max object size  ≈ 1.5 MiB   (etcd --max-request-bytes default)
     → a single ConfigMap or Secret is limited to ~1 MiB of data
   default DB quota = 2 GiB      (etcd --quota-backend-bytes default)
     recommended ceiling ≈ 8 GiB (beyond it, latency and recovery time degrade)
   hit the quota → etcd raises a NOSPACE alarm → the cluster goes READ-ONLY
     until you compact history, defragment, and clear the alarm

So a cluster is not a database. Stuff a few thousand oversized objects — verbose CRDs, Secrets abused as blob storage, a runaway flood of Events — into etcd and you approach the quota, at which point all writes stop: a self-inflicted outage that blindsides teams who assumed etcd was infinitely roomy.

It actually happened — CVE-2018-1002105, the first critical Kubernetes CVE (2018)

The “one door” is only as safe as the door’s own code. In late 2018, CVE-2018-1002105 (CVSS 9.8) showed what a flaw in the request path costs. The apiserver proxies certain requests to backends — aggregated API servers, and the kubelet for exec/attach/port-forward. A crafted upgrade request could leave that backend connection established and authenticated as the apiserver itself; the attacker could then send arbitrary requests down it, reaching the backend with the apiserver’s own credentials. Via the aggregated-API path it was reachable even by unauthenticated callers; via the kubelet path a user with pod-exec rights gained full control of every node — effectively cluster-admin. The fix shipped in patch releases 1.10.11, 1.11.5, 1.12.3, and 1.13. The lesson the rest of this Part keeps returning to: when everything funnels through one audited door, that door’s correctness is the cluster’s security — which is why Security in Depth treats authn and authz as load-bearing, not boilerplate.

The architect’s lens

Step back from the request path and answer the five questions behind the one door, the one truth:

• Why does it exist? Because the oldest failure in operations is scattered, unsynchronized configuration that no two machines agree on. Funneling every client through one stateless apiserver backed by one Raft-replicated etcd gives exactly one store, one writer, and one audited path to change it.
• What problem does it solve? It makes “who changed this object, when, and were they allowed?” an answerable question — authn, authz, admission, and schema validation all live at the single door, so controllers never re-implement security, and resourceVersion watches keep the whole system in sync without polling.
• What are the trade-offs? etcd is CP — every write is a consensus round-trip to a quorum, so a slow leader disk slows the entire control plane; lock-free concurrency means a contended write gets 409 and must re-read and retry; and etcd is sized for config (≈1.5 MiB per object, a ~2 GiB default / ~8 GiB practical store), not bulk data.
• When should I avoid leaning on it? Never bypass the door — but don’t abuse etcd as a database: stuffing oversized Secrets or a flood of Events toward the quota raises the NOSPACE alarm and turns the cluster read-only, and editing out a finalizer to force-delete a stuck object skips the very cleanup it existed to do.
• What breaks if I remove it? You lose the single audit log and the consistency floor; controllers writing etcd directly would each need their own security and validation; watches and optimistic concurrency vanish; and the LIST-then-watch resync that survives compaction has nothing to resume from — and as CVE-2018-1002105 showed, the door’s correctness is the cluster’s security.

Check your understanding

1. Why is it architecturally significant that only the apiserver talks to etcd, and what property would you lose if controllers wrote etcd directly?
2. A watcher reconnects after a long pause and gets 410 Gone / “resourceVersion too old.” Explain what happened in etcd and what a correct client does next.
3. Two controllers read the same object at resourceVersion: 500 and both try to write. Walk through what etcd does and why one gets a 409 Conflict — and why that conflict is a feature, not a failure.
4. With server-side apply, you set replicas: 3 and the HPA also manages replicas. What does field ownership do that client-side apply did not, and what’s the trade-off?
5. An object is stuck Terminating and won’t delete. What mechanism is holding it, what is that mechanism for, and what’s the danger of forcibly removing it?

Show answers

1. It means there is exactly one place to enforce authentication, authorization, admission, schema validation, and auditing — and one consistent writer of the store. If controllers wrote etcd directly, each would have to re-implement security and validation, and you’d lose the single audit log and the guarantee that every change passed the same gauntlet.
2. etcd compacted away the revision the client last saw, so the apiserver can no longer stream events from that point. The client must do a fresh LIST to get a new baseline and resume watching from that list’s resourceVersion. This LIST-then-watch is normal behavior an informer automates.
3. etcd does a compare-and-swap: write only if the stored resourceVersion still equals the one the client read. The first writer succeeds and bumps it to 501; the second’s CAS fails because the stored version no longer matches 500, so it gets 409 Conflict. It’s a feature because it means lock-free concurrency caught a stale write — the client just re-reads and retries, which is trivial for an idempotent reconcile loop.
4. Field ownership records which fieldManager owns each field in managedFields, so a conflicting apply fails loudly (unless you pass --force-conflicts) instead of silently overwriting — the old client-side three-way merge would let last-writer-win invisibly. The trade-off is a more verbose object (managedFields metadata) in exchange for losing the silent-clobber class of bug.
5. A finalizer in metadata.finalizers is holding it: the apiserver set deletionTimestamp but won’t delete the key until the list is empty. The finalizer’s controller is supposed to do real-world cleanup (e.g. delete a cloud load balancer/disk) and then remove its own entry. Forcibly removing the finalizer deletes the record but skips the cleanup, leaking the real resource the finalizer existed to reclaim.