Debugging & Troubleshooting Kubernetes

This is the last page of the Part, and the one you’ll reach for most. Everything before it explained how the machinery works; this page is what to do when it visibly isn’t. The whole Part has been one long argument that Kubernetes is a stack of reconciliation loops, and that turns out to be the key to debugging it: a healthy cluster is one where every loop has closed its gap, and a broken one is a loop that can’t. Your job is to find which loop is stuck and why. The events stream and component logs are how the loops tell you.

The thread, one final time: good debugging is the manual step we’re trying to make rare. The aim of everything you’ve learned — declared state, self-healing loops, eviction by policy, autoscaling — is to keep humans out of the 3 a.m. loop. When you do get pulled in, a systematic method beats poking at random; the playbook below is how you stay out of the failure modes that turn a 10-minute fix into a 3-hour outage.

Always start with describe and events

Before logs, before exec, before anything: kubectl describe and the events stream. Kubernetes narrates its own reconciliation through Events — the scheduler, kubelet, and controllers all post them — and describe shows you the object’s current status plus its recent events in one view. Ninety percent of “why won’t this Pod run?” is answered here without ever touching a log file.

kubectl describe pod web-7d9f-abc12        # status, conditions, and recent events
kubectl get events --sort-by=.lastTimestamp   # cluster-wide, newest last
kubectl get pod web-7d9f-abc12 -o wide     # node, IP, restart count at a glance

Read the Events before forming a theory

The single most common debugging mistake is jumping to kubectl logs and then to Stack Overflow before reading the events. The events usually say what’s wrong in plain English — FailedScheduling: 0/5 nodes are available: insufficient memory, Failed to pull image: not found, Back-off restarting failed container. Form your hypothesis from the events; use logs to confirm it. Working the other way around is how you spend an hour fixing the wrong thing.

The Pod-state decision tree

Most Pod problems land in one of four states, and the state tells you which loop is stuck. Read the state, then take the branch — don’t guess.

   kubectl get pods → STATUS?

   Pending ──────────► scheduler can't place it. describe → events:
                        • "insufficient cpu/memory" → no node fits requests
                        • "node(s) had untolerated taint" → needs a toleration
                        • "unbound PersistentVolumeClaim" → storage not ready
                        • stuck forever → no nodes / Cluster Autoscaler not adding

   ImagePullBackOff ─► kubelet can't FETCH the image (never started). events:
                        • wrong tag/name, private registry without imagePullSecret,
                          rate-limited registry, no network from node

   CrashLoopBackOff ─► image ran, process EXITED, kubelet keeps restarting it.
                        → kubectl logs --previous   (the crash is in the PREVIOUS container)
                        • bad config/missing env, failed migration, can't reach a dependency,
                          failing liveness probe killing a healthy-but-slow app

   OOMKilled ────────► container exceeded its memory limit (exit 137). describe shows it.
                        → raise memory limit or fix the leak
                          (see /kubernetes-deep/resource-management-and-node-pressure/)

The distinction people most often miss is ImagePullBackOff vs CrashLoopBackOff. ImagePullBackOff means the container never started — the kubelet couldn’t even fetch the image, so there are no logs to read; fix the image reference or the pull credentials. CrashLoopBackOff means the container did start and then exited — there are logs, and the answer is almost always in them. The “BackOff” in both names is the kubelet’s exponential back-off: it waits longer between each retry (10s, 20s, 40s… capped at 5m) so a broken Pod doesn’t hammer the registry or the node.

CrashLoopBackOff is a symptom, not a cause

CrashLoopBackOff only tells you the container keeps exiting — it says nothing about why. The cause is in the previous container’s logs (it’s already gone when you look) and sometimes in the events (a failing liveness probe restarting an app that was actually fine, just slow to start — fix the probe’s initialDelaySeconds, not the app). Never “fix” a CrashLoopBackOff without reading --previous logs.

Reading logs, including the crash you missed

The container has already restarted by the time you run kubectl logs, so the default shows the new (possibly healthy-looking) container. The crash is in the one that just died:

kubectl logs web-7d9f-abc12 --previous           # the container that CRASHED
kubectl logs web-7d9f-abc12 -c sidecar           # a specific container in a multi-container Pod
kubectl logs -f -l app=web --max-log-requests=10 # follow logs across all matching Pods

--previous is the flag that turns “it just restarts and I can’t see anything” into a root cause. For patterns across many Pods, ship logs to a real aggregator — see Logging — so you’re not racing the next restart.

Getting inside: exec and ephemeral debug containers

When the events and logs aren’t enough, get a shell next to the workload. The classic tool is exec:

kubectl exec -it web-7d9f-abc12 -- sh     # a shell inside the running container

But modern, hardened images are distroless — no shell, no curl, no ps — so exec fails with “executable not found.” The answer is an ephemeral debug container: kubectl debug injects a new container, with your own tooling image, into the running Pod, sharing its network and (optionally) process namespace — without restarting or rebuilding it.

# Attach a debug container with networking tools into a live Pod:
kubectl debug -it web-7d9f-abc12 --image=nicolaka/netshoot --target=web

# Copy a crashing Pod and override its command so it stays up for inspection:
kubectl debug web-7d9f-abc12 -it --copy-to=web-dbg --container=web -- sh

# Debug a NotReady node by getting a privileged shell into its host namespace:
kubectl debug node/node-1 -it --image=busybox

   exec            → run a command in an EXISTING container   (needs a shell in the image)
   debug (ephemeral) → inject a NEW container into a RUNNING Pod (bring your own tools)
   debug --copy-to → clone the Pod with a tweaked spec        (when the original won't even start)
   debug node/…    → a container in the NODE's namespaces     (debug the host itself)

Ephemeral containers exist because production images shouldn’t have a shell

There’s a real tension: a debuggable image (shell, curl, package manager) is also a bigger attack surface — exactly what the security page tells you to strip out. Ephemeral debug containers resolve it: ship a minimal distroless image and keep full debugging power, because you attach the tooling at debug time instead of baking it in. You get safety and introspectability instead of trading one for the other — a textbook example of the book’s thread.

Service and DNS connectivity

“The Pods are running but nothing can reach the service” is a different class of problem — the workload is fine; the networking loop is the gap. Work outward from the Pod, checking one hop at a time. (Cluster Networking Internals and Services & Networking explain the mechanism this debugs.)

   1. Does the Service have endpoints?
        kubectl get endpoints my-svc        ← EMPTY = the selector matches no Ready Pods
   2. Do the Pod labels match the Service selector?       (the #1 cause of empty endpoints)
   3. Does DNS resolve?
        kubectl run tmp --image=nicolaka/netshoot -it --rm -- \
          nslookup my-svc.my-ns.svc.cluster.local
   4. Can you reach the Pod IP directly, bypassing the Service?   (isolates Service vs app)
   5. Is a NetworkPolicy blocking it?      (default-deny with no matching allow rule)

The highest-yield check is step 1: an empty Endpoints list. A Service is just a stable name in front of the set of Pods its selector matches and that are Ready. Empty endpoints means either the label selector doesn’t match any Pod (a typo in app: web vs app: webapp) or the matching Pods aren’t passing their readiness probe — so the Service has nothing to route to. For DNS, remember the fully-qualified form service.namespace.svc.cluster.local; if short names fail but FQDNs work, suspect the Pod’s ndots/search-domain config or CoreDNS itself.

Readiness probes are an endpoint switch, not just a health check

A readiness probe doesn’t restart anything — it controls whether a Pod is in its Service’s Endpoints. A too-strict or wrongly-configured readiness probe silently pulls a perfectly healthy Pod out of rotation, and you get “the app is up but traffic 503s.” When endpoints are unexpectedly empty, check the probe before you suspect the network.

Node NotReady and reading control-plane logs

When a whole node goes NotReady, every Pod on it is in jeopardy and the scheduler stops using it. A node’s readiness is reported by its kubelet via the node lease heartbeat (kubelet page); NotReady means the kubelet stopped heartbeating or is reporting a problem. Work down the stack:

kubectl describe node node-1     # conditions: MemoryPressure, DiskPressure, PIDPressure, Ready
# then, on the node itself:
systemctl status kubelet         # is the kubelet even running?
journalctl -u kubelet -f         # kubelet logs: runtime errors, cert issues, CNI failures
systemctl status containerd      # is the container runtime up?

Common culprits: the kubelet or container runtime crashed, the node ran out of disk (DiskPressure), the CNI plugin failed so Pods can’t get IPs, or — tying back to Cluster Lifecycle — the kubelet’s client certificate expired and it can no longer authenticate to the apiserver.

For control-plane problems (the apiserver is slow, objects won’t create, the scheduler isn’t placing Pods), the components run as static pods, so you read them like any other Pod — or from the host:

kubectl -n kube-system logs kube-apiserver-cp-node-1
kubectl -n kube-system logs kube-scheduler-cp-node-1
kubectl -n kube-system logs etcd-cp-node-1            # "leader changed" / slow disk → etcd trouble

If the apiserver itself is down you can’t use kubectl at all — fall back to journalctl/crictl logs on the control-plane host and check etcd’s health, because an unhealthy etcd quorum takes the whole API with it.

Closing the Part

Step back and notice what made all of this tractable: you never debugged Kubernetes by guessing. You read the state, identified which reconciliation loop was stuck — scheduler (Pending), kubelet (ImagePull/Crash/OOM), endpoints controller (empty Service), node lease (NotReady) — and went to that loop’s events and logs. That is the throughline of the entire Part. The reconciliation loop wasn’t just how Kubernetes runs; it’s how you reason about it when it breaks.

And that is the thread, closed: every mechanism in these thirteen pages exists to remove a manual, error-prone, 3-a.m. step — the scheduler removes hand-placing workloads, autoscalers remove typing scale, eviction and QoS remove hand-arbitrating resources, etcd snapshots remove unrecoverable loss. When the loops are working, no human is in the path. When they aren’t, this page is how you find the one that’s stuck — quickly, systematically, and without making it worse. That is what it means to operate Kubernetes, rather than merely run it.

Check your understanding

1. Before reading any logs, what two commands should you run on a misbehaving Pod, and why are they almost always the fastest route to a hypothesis?
2. A Pod is in ImagePullBackOff; another is in CrashLoopBackOff. For each, can you read application logs, and what does the state tell you about how far the container got?
3. Why is kubectl logs --previous so often the command that cracks a CrashLoopBackOff, when plain kubectl logs shows nothing useful?
4. Your image is distroless, so kubectl exec -- sh fails. How do you get a shell’s worth of tooling next to the workload anyway, and why is this better than baking a shell into the image?
5. A Service returns no traffic though its Pods are Running. What’s the single highest-yield thing to check first, and what two root causes does an empty result point to?

Show answers

1. kubectl describe pod <name> and the events stream (kubectl get events --sort-by=.lastTimestamp). Kubernetes narrates its reconciliation through Events — the scheduler, kubelet, and controllers post them — and describe shows status plus recent events in one view, usually stating the problem in plain English (FailedScheduling: insufficient memory, Failed to pull image). Form the hypothesis from events; use logs to confirm.
2. ImagePullBackOff: the container never started — the kubelet couldn’t fetch the image — so there are no application logs; fix the image name/tag or pull credentials. CrashLoopBackOff: the container started and exited — there are logs (in the previous container), and the cause is almost always in them.
3. Because the container has already restarted by the time you look, so plain kubectl logs shows the new (often still-starting or healthy-looking) container. --previous shows the container that actually crashed, where the error message and stack trace live.
4. Use an ephemeral debug container: kubectl debug -it <pod> --image=nicolaka/netshoot --target=<c> injects a new container with your tooling into the running Pod, sharing its namespaces, without a rebuild or restart. It’s better because the production image stays minimal/distroless — smaller attack surface — while you still get full debugging power, attached only at debug time.
5. Check the Service’s Endpoints (kubectl get endpoints <svc>). An empty list means either the label selector matches no Pod (a label typo), or the matching Pods are not Ready (failing readiness probe), so the Service has nothing to route to — both pull Pods out of rotation regardless of whether the app itself is healthy.