The Kubelet & Container Runtime

The scheduler ends its job by writing one field — spec.nodeName — onto a Pod. Nothing is running yet. A Pod with a node name is still just a row in etcd. Turning that declaration into real Linux processes, with the right namespaces, cgroups, network, and storage, is the work of the kubelet: the agent running on every node, and the only Kubernetes component that actually touches containers.

This page goes underneath the container chapter — where a container was revealed to be just a process with namespaces and cgroups — and shows how the kubelet drives the runtime stack that assembles exactly that, from a YAML PodSpec down to runc calling clone(). The thread: the kubelet removes the manual job of SSHing into a box to docker run the right images with the right flags, restart them when they crash, and notice when they go unhealthy. It turns “run these containers and keep them running, here” into a control loop bound to one node.

The kubelet is a per-node reconciliation loop

Everything in the reconciliation loop applies here, scoped to a single machine. The kubelet’s desired state is the set of Pods assigned to its node; its actual state is the containers currently running; its act step drives the runtime to close the gap. It learns desired state from several sources, merged into one stream:

   DESIRED Pods for this node come from:
     • the API server   (watch on spec.nodeName == myNode)   ← the normal path
     • static pod files (/etc/kubernetes/manifests/*.yaml)   ← no API server needed
     • an HTTP endpoint (rarely used)
        │
        ▼  merged → the kubelet's "pod manager"
   ┌──────────────── SyncLoop (event-driven) ────────────────┐
   │  on add/update/delete Pod, or periodic resync, or        │
   │  PLEG event (a container changed state):                 │
   │      computePodActions(desired, actual)                  │
   │      → create sandbox? (re)start containers? kill some?  │
   │      → run probes, update status                         │
   └──────────────────────────┬───────────────────────────────┘
                              ▼
                  write Pod status back to the API server

The loop is level-triggered, not edge-triggered: on every iteration it recomputes the full desired vs. actual diff for each Pod rather than reacting to a single event, so a missed event self-corrects on the next pass. The PLEG (Pod Lifecycle Event Generator) periodically relists running containers from the runtime and emits events when something changed underneath — that’s how the kubelet notices a container the runtime restarted or killed, independent of any API-server message.

PodSpec → CRI → OCI runtime → a process

The kubelet does not know how to run a container. It speaks CRI — the Container Runtime Interface, a gRPC API — to a runtime that does. This is the boundary that let Kubernetes drop the built-in Docker shim: anything implementing CRI (containerd, CRI-O) plugs in unchanged.

   kubelet  ──CRI (gRPC over a local socket)──►  containerd / CRI-O
                                                        │  creates an OCI
                                                        │  runtime bundle
                                                        ▼
                                            OCI runtime: runc (or crun, kata, gVisor)
                                                        │  clone() + namespaces
                                                        │  + cgroups + pivot_root
                                                        ▼
                                            a real Linux process (your container)

Two interfaces, two layers:

• CRI (Kubernetes ↔ runtime) splits into two gRPC services: RuntimeService (sandboxes, containers, exec/attach/logs) and ImageService (pull/list/remove images). The socket is typically unix:///run/containerd/containerd.sock or .../crio/crio.sock.
• OCI (runtime ↔ kernel) is the lower contract: an image spec (how layers and config are packaged) and a runtime spec (a config.json “bundle” that runc reads to set up namespaces, cgroups, mounts, and the entrypoint). containerd/CRI-O generate the OCI bundle and shell out to runc to actually create the process.

CRI and OCI are different boundaries — don’t conflate them

CRI is the Kubernetes contract: “how the kubelet asks for Pods.” OCI is the industry contract: “how an image is packaged and how a runtime starts a container.” containerd implements CRI on its top edge and uses an OCI runtime (runc) on its bottom edge. Swapping runc for gVisor or Kata Containers (stronger isolation via a userspace kernel or a microVM) is an OCI-runtime change, configured per-Pod through a RuntimeClass — the CRI layer above is untouched.

The pod sandbox and the pause container

A Pod is not one container — it’s a group that shares a network namespace, IPC, and a lifecycle. So before running any of your containers, the runtime creates a pod sandbox: it sets up the shared namespaces and holds them open. That holder is the pause container — a tiny process whose entire job is to pause() and do nothing, owning the network namespace so the IP and interfaces persist even as your app containers crash and restart.

   RunPodSandbox  → create pause container, allocate the shared net namespace
        │            (kubelet then calls the CNI to wire that namespace → Pod IP)
        ▼
   CreateContainer/StartContainer  → each app container joins the sandbox's namespaces
        │
   your containers come and go; the sandbox (pause) stays → the Pod IP is stable

This is why kubectl get pods shows a Pod IP that survives container restarts: the address belongs to the sandbox, not to any one container. Kill the pause container and the whole Pod is recreated with a new sandbox — and a new IP.

CNI and CSI: invoked by the kubelet, not built in

The kubelet orchestrates two more plugin interfaces at sandbox-setup time, mirroring CRI’s design — Kubernetes defines the contract, vendors provide the implementation:

• CNI (Container Network Interface): right after RunPodSandbox, the kubelet (via the runtime) calls the configured CNI plugin to give the sandbox’s network namespace an IP and routes. The flat-network result is what Cluster Networking Internals builds on.
• CSI (Container Storage Interface): for each volume, the kubelet drives the NodeStage and NodePublish steps to mount the storage into the Pod before containers start — the node-side half of the flow detailed in Storage Internals (CSI).

Probes: liveness, readiness, startup

The kubelet doesn’t just start containers — it continuously checks their health and acts on the result. Three probe types, each answering a different question, and the distinction is load-bearing:

Probe	Question	Failure action
liveness	Is it alive, or wedged?	Restart the container
readiness	Can it serve traffic right now?	Remove it from Service endpoints (no restart)
startup	Has it finished booting yet?	Hold off liveness/readiness until it passes

spec:
  containers:
    - name: app
      livenessProbe:
        httpGet: { path: /healthz, port: 8080 }
        periodSeconds: 10
        failureThreshold: 3        # 3 consecutive fails → restart
      readinessProbe:
        httpGet: { path: /ready, port: 8080 }
        periodSeconds: 5
      startupProbe:                # slow JVM/migration boot: up to 30 × 10s = 5 min
        httpGet: { path: /healthz, port: 8080 }
        failureThreshold: 30
        periodSeconds: 10

The classic bug is using a liveness probe where you meant readiness: a dependency hiccups, liveness fails, the kubelet restarts a perfectly healthy process, the restart makes things worse, and you’ve built a crash loop out of a transient blip. Readiness is the safe default for “temporarily can’t serve.” The startup probe exists so a slow-booting app doesn’t get liveness-killed before it’s even up — without it you’d have to inflate liveness initialDelaySeconds and lose fast crash detection forever.

A failing readiness probe is invisible until you look

A Pod can be Running and 1/1 in kubectl get pods and still be pulled out of every Service because readiness is failing — there are no restarts and no crash, just silently no traffic. When “the Pod is up but nothing reaches it,” check kubectl describe pod for readiness probe events before anything else (Debugging Kubernetes).

Static pods: how the control plane bootstraps itself

There’s a chicken-and-egg problem: the API server runs as a Pod, but a normal Pod needs the API server to exist first. Static pods break the cycle. The kubelet watches a directory (/etc/kubernetes/manifests/) and runs any PodSpec it finds there directly, with no API server involved. This is how kubeadm brings up the control plane — kube-apiserver, etcd, kube-controller-manager, and kube-scheduler are static pods managed by the kubelet on the control plane nodes (covered in Cluster Lifecycle & Operations). The kubelet creates a read-only mirror Pod in the API server so the static pod is visible with kubectl get pods, but you can’t edit it through the API — the file on disk is the source of truth.

Node registration, status, and the node lease heartbeat

The kubelet also represents the node itself. On startup it registers by creating a Node object (or joining via a bootstrap token), advertising the node’s capacity, labels, and taints. Then it keeps two distinct heartbeats running:

   NodeStatus update  (the heavy one: conditions, capacity, images, addresses)
        → written every few minutes, OR sooner if something material changes
   Node Lease update  (the light one: a tiny Lease object, just "I'm alive")
        → bumped every ~10s in the kube-node-lease namespace

The two were split for scale. Pushing a full NodeStatus every few seconds across thousands of nodes overwhelmed etcd with writes that barely changed. The node lease is a near-empty object the kubelet renews cheaply; the node controller watches lease freshness to decide liveness. Miss the lease renewals past the grace period and the node controller marks the node NotReady and taints it node.kubernetes.io/not-ready:NoExecute — which, via the mechanism from the scheduler page, eventually evicts the Pods so the scheduler can reschedule them elsewhere. The heavy NodeStatus carries the detail (disk pressure, memory pressure, allocatable resources); the cheap lease carries the pulse.

Why “NotReady” and “Pods evicted” are minutes apart

A node dropping off isn’t instant downtime by design. The lease grace period must lapse before NotReady, then the NoExecute toleration’s tolerationSeconds (default 300s) must lapse before Pods are evicted. That deliberate delay rides out brief network blips so a 20-second partition doesn’t needlessly reschedule every Pod on the node — a trade-off of recovery speed for stability.

It actually happened — the dockershim “Docker is deprecated” panic (2020–2022)

CRI being a clean boundary had a famous payoff and a famous misunderstanding. Kubernetes 1.20 (December 2020) deprecated the built-in dockershim — the kubelet’s special-case adapter for talking to Docker Engine — and removed it for good in 1.24 (May 2022). Headlines read “Kubernetes is dropping Docker,” and many teams panicked that their Docker-built images would stop working. They didn’t: an image is an OCI artifact, independent of which runtime runs it. What went away was only the kubelet’s in-tree shim; clusters pointed their CRI socket at containerd (the very runtime Docker itself is built on) or CRI-O and kept running the exact same images. The episode is this page’s CRI-vs-OCI distinction in the wild — confusing the runtime contract with the image contract cost the community a year of needless alarm.

The thread, made concrete

The kubelet erases the entire manual node-operator role. By hand, running a service on a box meant: pull the right image, docker run it with the correct namespaces and resource flags, mount its volumes, wire its network, watch it, restart it when it died, health-check it, and pull it out of the load balancer when it went sick — per container, per node, forever, and remembered correctly at 3 a.m. The kubelet does all of it as a level-triggered loop bound to declared PodSpecs: CRI assembles the containers, the pause container stabilises the Pod IP, probes gate traffic and trigger restarts, and the node lease lets the cluster notice a dead machine and move its work. The safety win is that node-level operations stop depending on a human’s memory and presence. The cost is a deep, plugin-laden stack (kubelet → CRI → containerd → runc, plus CNI and CSI) where a failure can hide in any layer — which is exactly why the debugging page teaches you to read it top-to-bottom.

The kubelet handed the network namespace to the CNI and got back a Pod IP. How that IP becomes routable across the whole cluster, and how a Service virtual IP turns into a real backend, is Cluster Networking Internals, next.

The architect’s lens

Step back from the runtime stack and answer the five questions about the only component that actually touches containers:

• Why does it exist? Because a Pod with spec.nodeName is still just a row in etcd — turning it into real Linux processes with the right namespaces, cgroups, network, and storage, then keeping them running, was a human SSHing in to docker run with the right flags and restarting crashes at 3 a.m. The kubelet makes that a level-triggered loop bound to one node.
• What problem does it solve? It speaks CRI to a runtime (containerd/CRI-O → runc) so any OCI image plugs in unchanged, holds the Pod IP stable via the pause container’s sandbox, gates traffic and triggers restarts with liveness/readiness/startup probes, and lets the cluster notice a dead machine via the ~10 s node lease.
• What are the trade-offs? It’s a deep, plugin-laden stack (kubelet → CRI → containerd → runc, plus CNI and CSI) where a failure can hide in any layer; and the lease grace period plus the NoExecute tolerationSeconds (default 300 s) mean a dead node’s Pods take minutes to move — riding out brief partitions at the cost of recovery speed.
• When should I avoid its defaults? Don’t wire a dependency check as a liveness probe — it restarts a healthy-but-slow app into a crash loop; use readiness for “temporarily can’t serve” and a startup probe for slow boots instead of inflating liveness initialDelaySeconds; and swap runc for gVisor/Kata only via RuntimeClass when you genuinely need stronger isolation.
• What breaks if I remove it? The whole node-operator role returns by hand — pull, run, mount, wire, watch, restart, health-check, and pull-from-LB per container per node, forever — and the control plane can’t even bootstrap, since the apiserver runs as a static pod the kubelet starts before any apiserver exists.

Check your understanding

1. The scheduler set spec.nodeName but nothing is running. What does the kubelet do to turn that declaration into processes, and why is its sync loop described as level-triggered?
2. Distinguish CRI from OCI. Where does containerd sit relative to each, and what does runc actually do? How does swapping in gVisor fit this picture?
3. What is the pause container for, and why does it explain a Pod IP surviving a container crash?
4. A dependency briefly hiccups and a container’s health endpoint returns 503 for 20 seconds. Contrast what happens if that endpoint is wired as a liveness probe vs a readiness probe — and where a startup probe helps.
5. Why does the kubelet maintain two heartbeats (NodeStatus and the node lease), and trace what happens — through NotReady, taints, and eviction — when a node stops renewing its lease.

Show answers

1. The kubelet reconciles its node’s assigned Pods (desired) against the containers actually running (actual), driving the runtime via CRI to create sandboxes, start/kill containers, run probes, and report status. It’s level-triggered because each pass recomputes the full desired-vs-actual diff rather than reacting to a single event, so a missed event self-corrects on the next iteration (the PLEG also relists running containers to catch runtime-side changes).
2. CRI is the Kubernetes↔runtime gRPC contract (how the kubelet asks for Pods); OCI is the industry image/runtime contract (how images are packaged and containers started). containerd implements CRI on top and generates an OCI bundle, then shells out to runc, which actually clone()s the process with namespaces/cgroups/mounts. Swapping in gVisor/Kata changes only the OCI runtime (via RuntimeClass); the CRI layer is untouched.
3. The pause container holds the Pod’s shared network (and IPC) namespace open and does nothing else, so the Pod IP and interfaces belong to the sandbox, not any app container. App containers can crash and restart while the sandbox persists — hence the Pod IP survives container restarts; only destroying the sandbox (pause) gives a new IP.
4. As a liveness probe, three failures make the kubelet restart a healthy process over a transient blip — potentially a self-inflicted crash loop. As a readiness probe, the Pod is just removed from Service endpoints (no restart) and rejoins when it recovers — the safe choice for “temporarily can’t serve.” A startup probe holds off liveness/readiness until a slow-booting app is up, so it isn’t liveness-killed before it finishes booting.
5. They were split for scale: pushing a full NodeStatus (conditions, capacity, images) every few seconds would flood etcd, so the kubelet renews a tiny node lease (~10s) as a cheap pulse and writes the heavy status only periodically/on change. If lease renewals stop past the grace period, the node controller marks the node NotReady and applies node.kubernetes.io/not-ready:NoExecute; after the Pods’ tolerationSeconds (default 300s) the Pods are evicted and rescheduled elsewhere.