Revision · Kubernetes in Depth

This part took Part 4’s map and walked the territory: it followed a change through the real machinery — etcd, controllers, the scheduler, the kubelet, and the cross-cutting systems around them — always asking “what is this component doing, and why that way?” so you can debug what you can’t normally see.

What this part covered

• The apiserver is the one door — every change is authn’d, authz’d, admitted, validated, and written to etcd through a single writer, so understanding that request path tells you where a kubectl apply can hang.
• etcd is a consistent log, not a queryable DB — Raft, MVCC, and resourceVersion power watches and optimistic concurrency, which is why an update sometimes fails with a 409 and how finalizers and ownerReferences drive ordered deletion and garbage collection.
• Controllers are reconciliation loops with real plumbing — the informer → reflector → delta FIFO → lister → workqueue → reconcile pipeline is level-triggered, backs off with rate limiting, and leader-elects so exactly one replica acts.
• Scheduling is constraint satisfaction, one Pod at a time — the framework’s PreFilter/Filter/Score/Reserve/Permit/Bind extension points, plus affinity, taints, topology spread, and preemption, decide where work lands and why a Pod stays Pending.
• The kubelet turns a PodSpec into processes — its per-node sync loop drives CRI → containerd/CRI-O → runc, sets up the pause-container sandbox, invokes CNI and CSI, runs probes, and heartbeats via node leases.
• The network is flat, then filtered — routable Pod IPs with no pod-to-pod NAT, CNI to assign them, kube-proxy (iptables/IPVS/eBPF) to implement Services, CoreDNS for names, and NetworkPolicy enforced by the CNI dataplane.
• Admission is the last gate before etcd — mutating then validating webhooks, CEL-based ValidatingAdmissionPolicy, Gatekeeper/Kyverno, and Pod Security Admission let the cluster add defaults and reject bad objects automatically.
• Security is least privilege, layer by layer — the 4 C’s, authn vs RBAC authz, short-lived bound ServiceAccount tokens, securityContext kernel confinement, and encrypting Secrets at rest with a KMS close the gaps from identity down to etcd.
• Storage and autoscaling are driven loops too — CSI’s controller/node split runs CreateVolume → Publish → Stage → Publish to mount a PVC (with WaitForFirstConsumer for topology), while the metrics pipeline feeds the HPA formula, VPA, Cluster Autoscaler, and KEDA.
• Resources, operations, and debugging keep it alive — requests and limits become cgroup shares and quotas across three QoS classes with OOM scoring and eviction signals; kubeadm, HA etcd, cert rotation, and version skew keep the cluster standing; and a describe-and-events-first playbook triages Pending/CrashLoop/ImagePull/OOM.

The takeaway

Held together, these pages let you carry the whole machine in your head: trace a kubectl apply from your terminal to a running container, name the component responsible at every hop, and know exactly where to look when one misbehaves. The two pages you’ll reach for most under pressure — operations and debugging — are only useful because you now understand the internals they operate on. Next up: Observability & SRE, which gives you the signals to see all of this machinery from the outside once it’s running in production.