Kubernetes Architecture

The last page said Kubernetes lets you declare desired state and have controllers make it real. This page shows the machinery that does it. There are only a handful of components, and once you know what each one wants, the cluster stops being a black box.

Keep the recurring thread in view: what manual, error-prone step does this architecture remove — and how does it make production safer? The short answer is that there is exactly one source of truth (the cluster’s stored desired state) and exactly one door to change it (the API server). No more configuration scattered across machines, no more “is box 3 in sync with box 1?” The architecture itself is a safety property.

Two halves: the control plane and the nodes

A Kubernetes cluster splits into two roles. The control plane is the brain — it decides what should be true. The worker nodes are the muscle — they run your actual containers. On a production cluster the control plane runs on dedicated machines (and is replicated for high availability); on a laptop tool like kind or minikube everything is squeezed onto one node.

                         ┌──────────── CONTROL PLANE (the brain) ────────────┐
                         │                                                    │
   kubectl apply  ─────► │   kube-apiserver  ◄──── the ONLY door in/out       │
   (you declare          │        │                                          │
    desired state)       │        ▼                                          │
                         │      etcd   ◄── the single source of truth        │
                         │        ▲          (all desired + actual state)    │
                         │        │                                          │
                         │   ┌────┴─────────────┐                            │
                         │   │ scheduler         │ ← assigns Pods to nodes    │
                         │   │ controller-manager│ ← runs the reconcile loops │
                         │   └───────────────────┘                            │
                         └────────────────────┬───────────────────────────────┘
                                              │ (everyone talks via the API server)
              ┌───────────────────────────────┼───────────────────────────────┐
              ▼                                ▼                                ▼
        ┌───────── NODE 1 ─────────┐    ┌───────── NODE 2 ─────────┐    ┌──── NODE 3 ────┐
        │ kubelet (runs/watches Pods)│   │ kubelet                  │    │ kubelet        │
        │ kube-proxy (Service rules) │   │ kube-proxy               │    │ kube-proxy     │
        │ container runtime (containerd)│ │ container runtime        │    │ runtime        │
        │   └─ your Pods/containers   │   │   └─ your Pods           │    │   └─ your Pods │
        └────────────────────────────┘   └──────────────────────────┘    └────────────────┘

The control plane, component by component

kube-apiserver is the front door and the only component anything talks to directly. kubectl, the controllers, the kubelets — all of them go through the API server. It validates every request, applies authentication and authorization (RBAC), and is the only thing that reads and writes etcd. This single-door design is what makes the cluster auditable and consistent: there is one place to secure, one place to log changes, one definition of the truth.

etcd is a distributed, consistent key-value store. It holds all cluster state — every object you’ve declared (your desired state) and the recorded status of those objects (the observed actual state). If etcd is the cluster’s memory, the API server is the only one allowed to touch it.

etcd is your cluster

Lose etcd and you’ve lost the cluster’s entire definition. This is why production clusters run etcd replicated (an odd number of members for quorum) and back it up. The manual step Kubernetes removes — config scattered and unbackupped across many boxes — is replaced by one store you can actually protect.

kube-scheduler watches for Pods that have been created but not yet assigned to a node. For each, it filters out nodes that can’t fit it (not enough CPU/memory, wrong constraints) and scores the rest, then writes its choice back through the API server. It only decides; it doesn’t start anything. That bin-packing judgment you’d otherwise make by hand (the first wall from the previous page) is now a component.

kube-controller-manager runs the reconciliation loops — the Deployment controller, ReplicaSet controller, Node controller, and many more — each watching its objects and driving actual state toward desired. This is the engine behind self-healing, and the whole of the next page.

(There’s also cloud-controller-manager, which talks to your cloud provider for load balancers and the like — useful to know exists, not essential right now.)

The worker nodes, component by component

kubelet is the agent on every node. It watches the API server for Pods assigned to its node, tells the container runtime to start them, and continuously reports their health back up. It is the hands that turn the scheduler’s decision into running containers — and the eyes that report whether they’re actually alive.

kube-proxy programs the node’s network rules so that traffic to a Service’s stable virtual IP lands on a healthy Pod. We unpack this in Services & Networking.

container runtime is the software that actually runs containers — usually containerd. Kubernetes talks to it through the Container Runtime Interface (CRI). This is the same container-as-a-process machinery from Part 3; Kubernetes sits on top, it doesn’t replace it.

The flow of a single kubectl apply

Watch how the pieces cooperate. This is the architecture in motion:

kubectl apply -f web.yaml         # 1. you POST desired state to the API server

   1. kubectl → kube-apiserver: "I want a Deployment with 4 replicas"
   2. apiserver validates it, writes the object to etcd            (desired state stored)
   3. controller-manager notices: 4 wanted, 0 exist → creates 4 Pod objects in etcd
   4. scheduler notices 4 unassigned Pods → picks nodes, writes the assignments
   5. each kubelet notices "a Pod is mine" → tells containerd to run it
   6. kubelet reports status back to the apiserver → etcd            (actual state recorded)

Nobody was commanded step by step. You stated a wish; each component reacted to the stored state and did its one job. That decoupling is the genius of the design — and the reason a crashed component can restart and simply pick up from etcd, because the truth lives in the store, not in any one process’s memory.

kubectl get nodes                 # the worker nodes
kubectl get pods -A               # every pod, including control-plane ones (kube-system)
kubectl get --raw='/readyz?verbose'   # control-plane health checks (the old `componentstatuses` is deprecated)

By the numbers — why etcd clusters are odd-sized

etcd keeps its replicas consistent with the Raft consensus algorithm, which commits a write only once a majority of members hold it. That majority — quorum — is floor(n/2) + 1, so the failures a cluster can survive are:

   members (n)   quorum   can lose
   ───────────────────────────────
       3           2          1
       5           3          2
       7           4          3

   add a 4th member → quorum 3 → still only tolerates 1 failure

That last line is why production etcd runs 3, 5, or 7 members, never 4: an even size needs a larger majority for no extra fault tolerance, while adding one more thing that can fail. Lose quorum and the cluster goes read-only — it stops accepting changes rather than risk splitting the one source of truth.

The architect’s lens

Step back from the component diagram and judge the architecture as a design choice:

• Why does it exist? Because the alternative — desired state scattered across many boxes with no single way to change it — drifts and can’t be audited. The control-plane/node split gives you exactly one source of truth (etcd) and one door (the API server) to change it.
• What problem does it solve? It decouples deciding from doing: the scheduler decides placement, the kubelet acts, controllers reconcile — each reacting to stored state, so a crashed component restarts and resumes from etcd instead of losing its place.
• What are the trade-offs? The single door is also a single dependency — etcd is your cluster, so you must replicate it (3, 5, or 7 members for Raft quorum) and back it up, and lose-quorum means the cluster goes read-only rather than risk splitting the truth.
• When should I avoid it? You don’t avoid the architecture if you run Kubernetes — but for a single app on one box, standing up a control plane, etcd, and an API server is overhead the why-orchestration walls don’t yet justify.
• What breaks if I remove it? Drop the one-door rule and config silently scatters and drifts across machines, with no single place to authenticate, authorize (RBAC), validate, or log changes; lose etcd entirely and you’ve lost the cluster’s whole definition, not just one process.

Check your understanding

1. What are the two halves of a cluster, and which one actually runs your containers?
2. Why is “the API server is the only door, etcd is the only truth” a safety property and not just an implementation detail?
3. The scheduler picks a node for a Pod but doesn’t start anything. Which component actually starts the container, and where does it run?
4. Walk through what happens, component by component, between kubectl apply -f web.yaml and a container running on a node.
5. Why is losing etcd so much worse than losing any other single component, and what do operators do about it?

Show answers

1. The control plane (the brain — API server, etcd, scheduler, controller-manager) decides what should be true; the worker nodes (the muscle) actually run your containers.
2. Because it gives you exactly one source of truth (etcd) and exactly one door (the API server) to change it. That means one place to authenticate, authorize (RBAC), validate, and log every change — config can’t silently scatter and drift across machines, so consistency and auditability are structural properties, not things you hope for.
3. The kubelet on the chosen node actually starts the container — it watches the API server for Pods assigned to its node, tells the container runtime (containerd) to run them, and reports their health back. The scheduler only decides; the kubelet acts.
4. kubectl POSTs the Deployment to the API server, which validates it and writes it to etcd; the controller-manager sees 4 wanted / 0 existing and creates 4 Pod objects; the scheduler sees unassigned Pods and binds each to a node; each kubelet sees a Pod is now its own and tells containerd to run it; the kubelet then reports status back through the API server into etcd.
5. Because etcd holds the cluster’s entire definition (all desired and observed state) — lose it and you’ve lost the cluster itself, whereas any other component can crash and restart by reading state back from etcd. Operators run etcd replicated with an odd number of members for quorum and back it up (and turn on encryption at rest).