Part 4½ · Kubernetes in Depth

Part 4 taught you to use Kubernetes: what a Pod, Deployment, Service, and PersistentVolume are, and how to declare them. It ended on the single idea that ties the whole system together — the reconciliation loop. This Part picks up exactly there and goes down. We assume you have finished Part 4 and never re-explain the basics; instead we open each component, look at the data structures and the wire protocol, and answer the question a senior engineer actually asks at 3am: what is this thing doing, and why is it doing it that way?

That framing is deliberate. Part 4 is the map; Part 4½ is the territory. When a kubectl apply hangs, when a Pod is Pending forever, when a rolling update stalls, when two controllers fight over the same field — you cannot debug what you cannot see inside. The recurring thread of the whole book carries through here too, just one layer deeper: what manual, error-prone step does this mechanism remove, and how does it make production safer? Only now the “mechanism” is etcd’s Raft log, the admission chain, the informer cache, the CSI gRPC calls — the machinery itself.

How deep is “in depth”?

These are the most technically detailed pages in the book. Expect real YAML, real command output in text fences, real field names (resourceVersion, ownerReferences, oom_score_adj, NodeStageVolume), and real internal flow. We still teach why before how — every mechanism is motivated by the problem it solves before we trace its wiring — and we stay balanced on trade-offs, because every design choice here bought something and cost something.

   PART 4  (the intro)                 PART 4½  (this part — the internals)
   ──────────────────                  ───────────────────────────────────
   "a Deployment keeps N Pods"   ───►  apiserver → etcd (Raft, MVCC, watch),
   "controllers reconcile"             informer→reflector→workqueue→reconcile,
   "the scheduler picks a node"        the scheduling framework's plugin chain,
   "kubelet runs the container"        kubelet sync loop → CRI → OCI runtime,
   "Services have a stable IP"         kube-proxy: iptables vs IPVS vs eBPF
        what & how-to-use                      how it actually works

What you need before this Part

This Part leans on three foundations. If any feels shaky, skim it first — these pages won’t repeat it:

• The Kubernetes architecture — the control plane / node split, and “the API server is the one door.”
• The reconciliation loop — observe, compare, act, forever. Every controller page below is a variation on it.
• Namespaces & cgroups — the kernel primitives the kubelet, the CRI, and node-pressure eviction all stand on.

Roadmap for this Part

Read in order — but each page also stands alone as a reference once you know the system.

#	Page	What it opens up
2	The API Machinery	apiserver as sole etcd writer; Raft, MVCC, watch; the request path; server-side apply; finalizers & garbage collection
3	Controllers & Informers	informer → reflector → delta FIFO → lister → workqueue → reconcile; level-triggered design; leader election; backoff
4	The Scheduler in Depth	the scheduling framework’s plugins; filter/score/reserve/bind; affinity, taints, spread; preemption
5	The Kubelet & Container Runtime	the sync loop; PodSpec → CRI → OCI runc; the pause container; probes; node leases
6	Cluster Networking Internals	the flat-IP model; CNI; kube-proxy (iptables/IPVS/eBPF); CoreDNS; NetworkPolicy
7	Admission Control & Policy	mutating → validating chain; webhooks; ValidatingAdmissionPolicy (CEL); Gatekeeper/Kyverno; Pod Security
8	Security in Depth	the 4 C’s; authn vs authz; RBAC; bound ServiceAccount tokens; securityContext; secrets encryption & KMS
9	Storage Internals (CSI)	controller vs node plugin; CreateVolume/Publish/Stage; PV/PVC binding; dynamic provisioning; snapshots
10	Autoscaling Internals	the metrics pipeline; the HPA algorithm; VPA; Cluster Autoscaler; KEDA; how they compose
11	Resource Management & Node Pressure	requests→cgroup shares, limits→quotas; QoS; the OOM killer; eviction signals; PDBs; ResourceQuota
12	Cluster Lifecycle & Operations	kubeadm bootstrap; HA control plane; cert rotation; etcd backup/restore; version skew & rolling upgrades
13	Debugging & Troubleshooting Kubernetes	a systematic playbook — events first; the Pending/CrashLoop/ImagePull/OOM decision tree; kubectl debug

The shape of the journey mirrors a request’s own life. We start where every change enters the system — the API machinery and etcd — then follow that change outward through the controllers that react to it, the scheduler that places the work, and the kubelet that finally runs it. From there we widen to the cross-cutting systems — networking, security, storage, autoscaling, resources — and end on the two pages you’ll reach for under fire: operations and debugging.

By the end you should be able to hold the whole machine in your head: trace a kubectl apply from your terminal to a running container and back, name the component responsible at every hop, and — when one of them misbehaves — know exactly where to look and why.

Check your understanding

1. In one sentence, what is the difference in intent between Part 4 and Part 4½?
2. The roadmap is ordered to follow “a change’s own life through the system.” What enters first, and what acts last — and why is that a useful mental model?
3. This Part promises to stay first-principles even at maximum depth. What does “why before how” buy a reader who is debugging a production incident?
4. Name the three foundation pages this Part deliberately does not repeat, and one thing each contributes that the deep pages assume you already know.

Show answers

1. Part 4 teaches what objects exist and how to use them; Part 4½ teaches how the machinery underneath actually works — the same system, one layer down.
2. A change enters through the API server into etcd, then controllers react, the scheduler places the work, and the kubelet runs it last. It’s useful because debugging is mostly “where in this pipeline did the change get stuck?” — and knowing the order tells you which component to inspect next.
3. Knowing why a mechanism exists tells you what it’s supposed to do, so you can spot the gap between intended and observed behavior — the heart of debugging. Memorized “how” alone leaves you guessing when reality deviates from the happy path.
4. Kubernetes architecture (the control-plane/node split and “one door, one truth”); the reconciliation loop (observe→compare→act, the shape every controller takes); and namespaces & cgroups (the kernel primitives the kubelet and resource management stand on).