Cluster Lifecycle & Operations

Every page so far assumed a cluster already existed and worked. This one builds and operates it. The architecture page named the control-plane components — apiserver, etcd, scheduler, controller-manager, and the kubelet on each node. Here we answer the operator’s questions: how do those pieces get bootstrapped onto bare machines, how do you run more than one of them so a single failure isn’t an outage, how do the certificates that secure them get issued and rotated, how do you back up the one component whose loss is unrecoverable, and how do you upgrade the whole thing without taking down the workloads it’s running?

The thread is sharp here. The manual step being removed — and the one that bites hardest when it’s not automated — is a human hand-placing certificates, hand-editing static manifests, and hoping the etcd box never dies. The practices below turn cluster operation from heroic, error-prone toil into repeatable, recoverable procedure. The cost is that you must understand the machinery well enough to recover it at 3 a.m.

Bootstrapping with kubeadm

You rarely assemble a control plane by hand anymore; kubeadm is the official bootstrapper. It doesn’t manage your OS or provision machines — it takes already-running Linux hosts with a container runtime and a kubelet, and turns them into a conformant cluster. The control-plane components it starts are static pods: the kubelet on the control-plane node watches a directory (/etc/kubernetes/manifests/) and runs whatever Pod manifests it finds there, with no apiserver involved — which neatly solves the chicken-and-egg problem of “how does the apiserver run before the apiserver exists.” (Static pods are covered on the kubelet page.)

   kubeadm init
     ├─ preflight checks (kernel, ports, swap, runtime)
     ├─ generate CA + certificates  → /etc/kubernetes/pki/
     ├─ write static-pod manifests   → /etc/kubernetes/manifests/  (apiserver, etcd,
     │                                                               controller-manager, scheduler)
     ├─ kubelet sees manifests, starts the control plane
     ├─ wait for apiserver healthy, apply RBAC + bootstrap-token config
     └─ print:  kubeadm join <ip>:6443 --token … --discovery-token-ca-cert-hash sha256:…

A worker then runs that printed kubeadm join: it uses a short-lived bootstrap token to authenticate to the apiserver, requests a client certificate via the CSR API (TLS bootstrapping), and registers itself as a Node. The join token is deliberately short-lived because anyone holding it can add a node to your cluster.

HA control plane: stacked vs external etcd

One control-plane node means one apiserver and one etcd — a single point of failure. High availability means running three (or five) control-plane nodes and putting a load balancer in front of the apiservers. The apiserver itself is stateless and horizontally scalable: all state lives in etcd, so you can run N apiservers behind one VIP and any of them can serve any request. The interesting choice is where etcd lives.

   STACKED etcd                          EXTERNAL etcd
   ┌── cp-node-1 ──┐                      ┌── cp-node-1 ──┐    ┌─ etcd-1 ─┐
   │ apiserver     │   each node runs     │ apiserver     │    │  etcd    │
   │ etcd  ◄──────►│   its own etcd,      │ ──────────────┼───►│  etcd-2  │  separate
   └───────────────┘   members form a     └───────────────┘    │  etcd-3  │  3-node
   (repeat ×3, etcd    raft quorum         (repeat ×3,          └──────────┘  raft cluster
    members peer)                          apiservers share the external etcd)

	Stacked etcd	External etcd
etcd location	co-located on each control-plane node	dedicated etcd hosts
Machines needed	3	3 + 3 (or more)
Blast radius	losing a node loses an apiserver and an etcd member	apiserver and etcd fail independently
Default for	kubeadm simple HA	larger/regulated clusters

Either way, etcd needs an odd number of members for quorum — raft requires a majority to commit a write, so 3 members tolerate 1 failure and 5 tolerate 2. (Why etcd is the only writer and how raft and resourceVersion work is on The API Machinery.)

Quorum, not count, is what keeps you alive

Two etcd members are worse than one: a majority of two is two, so losing either one loses quorum and the cluster goes read-only. Always run an odd number — 3 or 5. And never run an even number “to be safe”; you’ve only widened the window in which a single failure stops all writes.

Certificates and rotation

A Kubernetes cluster is a web of mutual TLS: the apiserver authenticates to etcd, the kubelets to the apiserver, the controller-manager to the apiserver, and clients (you, via kubeconfig) to the apiserver. kubeadm generates a cluster CA and issues all of these from it, storing them under /etc/kubernetes/pki/. Certificates expire — kubeadm’s default is one year — and an expired apiserver or etcd certificate is a hard outage: components can no longer authenticate to each other.

kubeadm certs check-expiration          # table of every cert and its expiry
kubeadm certs renew all                 # renew all control-plane certs from the CA

Two rotation paths matter and they’re easy to conflate:

• Control-plane certs are renewed by kubeadm certs renew (and renewed automatically on every kubeadm upgrade). The CA itself typically lasts ten years; the leaf certs are the ones that lapse.
• Kubelet client certs rotate automatically when the kubelet is configured with rotateCertificates: true — it requests a fresh cert via the CSR API before the current one expires, so worker nodes don’t need hands-on renewal.

An expired cluster cert is a silent time bomb

Nothing warns you that certs are about to expire — the cluster runs perfectly until the day they lapse, then the control plane stops authenticating and kubectl returns x509 errors. Put kubeadm certs check-expiration on a schedule, or simply upgrade often enough that the automatic renewal on kubeadm upgrade keeps them fresh. This is one of the most common “the cluster died and nothing changed” incidents.

etcd backup and restore

etcd holds all cluster state — every object, every Secret, every bit of desired state. Lose etcd with no backup and the cluster is unrecoverable; you can rebuild the control plane, but the contents are gone. So the single most important operational habit is regular etcd snapshots.

# Back up — point etcdctl at one healthy member with its client certs:
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backups/etcd-$(date +%F).db

# Restore — rebuild a data dir from the snapshot, then point etcd at it:
ETCDCTL_API=3 etcdctl snapshot restore /backups/etcd-2026-06-24.db \
  --data-dir=/var/lib/etcd-restored

Restore is not a single command you run on a live cluster — it’s a procedure. You stop the apiserver and etcd static pods (move their manifests out of /etc/kubernetes/manifests/), restore the snapshot into a fresh data directory on each etcd member, point each member’s static-pod manifest at the new data dir with matching peer URLs, then bring etcd and the apiserver back. A restore returns the cluster to the exact state at snapshot time — anything created after the snapshot is gone.

A backup you’ve never restored is a hope, not a backup

The etcd restore procedure is fiddly enough — stopping static pods, matching peer URLs across members — that the first time you should run it is not during an incident. Rehearse a restore on a throwaway cluster so the runbook is muscle memory. This is the exact “manual, error-prone step” the thread warns about: an unrehearsed restore under pressure is where clusters are lost for good.

Version skew and rolling upgrades

You can’t upgrade everything at once, so Kubernetes defines a version-skew policy that says which component-version mismatches are supported during the in-between state. The rules flow from one principle: the apiserver is the hub, and everything else must be no newer than it.

• The kube-apiserver is upgraded first and is the reference version.
• kubelet and kube-proxy may be up to three minor versions older than the apiserver, never newer.
• controller-manager, scheduler must be no newer than the apiserver (same or one older).
• You upgrade one minor version at a time — no skipping from 1.30 to 1.32 directly.

This is why the upgrade order is fixed: control plane first, then nodes. A newer apiserver can serve older kubelets; an older apiserver can’t reliably serve newer kubelets.

   1. Upgrade control plane            kubeadm upgrade apply v1.31.x   (one node at a time in HA)
        apiserver → controller-mgr → scheduler → etcd (certs auto-renew)
   2. Upgrade nodes, one at a time:
        kubectl cordon  node-1         (mark unschedulable — no new Pods land here)
        kubectl drain   node-1         (evict Pods, respecting PodDisruptionBudgets)
        kubeadm upgrade node           (upgrade kubelet/kube-proxy on the node)
        kubectl uncordon node-1        (return it to service)
   3. Repeat for node-2, node-3 …

cordon and drain are the safety pair. Cordon marks the node unschedulable so the upgrade doesn’t race new Pods onto it. Drain evicts the running Pods through the Eviction API — which respects PodDisruptionBudgets — so the workload keeps its minimum availability while its Pods are recreated on other nodes. Drain one node at a time and the PDBs guarantee the service never drops below its budget, even mid-upgrade. This is the same Eviction-API machinery the Cluster Autoscaler uses for safe scale-down.

The thread, restated for operations: kubeadm replaces hand-placing certs and manifests; HA replaces “pray the one box stays up”; cert rotation and etcd snapshots replace silent expiry and unrecoverable loss; and the version-skew policy with cordon/drain replaces a risky all-at-once upgrade with an orderly rolling one that PDBs keep safe. Each removes a manual, error-prone step — and each demands you understand it well enough to recover when it goes wrong.

A running, upgradable cluster will still break in production. The final page is the operator’s field manual for when it does: Debugging & Troubleshooting Kubernetes.

By the numbers — etcd’s brutal disk-latency budget

Every Kubernetes write waits on etcd committing to a quorum, and committing means an fsync to disk on each member. That puts the whole control plane on etcd’s storage latency:

   etcd's hard requirement: 99th-percentile WAL fsync  < ~10 ms
   slow disk → fsync stalls → heartbeats miss → leader election →
        apiserver writes hang → "etcdserver: request timed out"

This is why etcd’s documented guidance is dedicated SSDs (and members within roughly 10 ms network RTT — one region). A noisy-neighbour VM or a spinning disk under etcd doesn’t fail loudly; it produces a control plane that intermittently can’t write — the most baffling kind of outage, because the workloads keep running and only kubectl apply is mysteriously slow. When a control plane feels sluggish, etcd disk latency is the first number to pull, not the last.

Check your understanding

1. How does kubeadm start the control plane before any apiserver exists? Name the mechanism and the directory it watches.
2. Why must etcd run an odd number of members, and why are two members strictly worse than one?
3. A cluster that “nothing changed on” suddenly returns x509 errors from kubectl and the apiserver is down. What’s the most likely cause, and what command would have warned you?
4. Walk the etcd restore at a high level. Why is it a procedure rather than a single live command, and what state do you end up in?
5. Why is the upgrade order always control-plane-first-then-nodes, and what role do cordon, drain, and PodDisruptionBudgets play in keeping a node upgrade safe?

Show answers

1. Via static pods: the kubelet on the control-plane node watches /etc/kubernetes/manifests/ and runs any Pod manifests there without an apiserver. kubeadm writes the apiserver/etcd/controller- manager/scheduler manifests into that directory, and the kubelet brings them up — solving the chicken-and-egg of running the apiserver before the apiserver exists.
2. etcd uses raft, which needs a majority to commit a write. With 3 members the majority is 2 (tolerates 1 failure); with 5 it’s 3 (tolerates 2). Two members are worse than one because a majority of two is two — losing either one loses quorum and the cluster goes read-only, so you’ve added a failure mode without adding fault tolerance.
3. Expired control-plane certificates (kubeadm’s default leaf-cert lifetime is one year). The cluster runs fine until they lapse, then components can’t authenticate and you get x509 errors. kubeadm certs check-expiration lists every cert and its expiry and would have warned you; regular kubeadm upgrade also auto-renews them.
4. Stop the apiserver and etcd static pods (move their manifests aside), etcdctl snapshot restore into a fresh data dir on each member with matching peer URLs, repoint the static-pod manifests at the new dir, then restart etcd and the apiserver. It’s a procedure, not a live command, because you must take etcd down and rebuild its data dir consistently across members. You end up at the exact state as of the snapshot — anything created afterward is lost.
5. Because the apiserver is the hub and nothing may be newer than it: a newer apiserver can serve older kubelets, but not vice versa, so you upgrade the control plane first, then nodes (kubelet may trail by up to three minors). Cordon stops new Pods landing on the node being upgraded; drain evicts its Pods via the Eviction API, which respects PDBs, so the workload keeps its minimum availability while Pods move to other nodes — one node at a time, the service never dips below budget.