Storage & Stateful Workloads

Everything so far leaned on one assumption that makes Kubernetes work: Pods are disposable. The reconciliation loop freely kills a Pod and starts a fresh one, and for a stateless web app that’s wonderful — there’s nothing to lose. But it hides a brutal default. A container’s filesystem is ephemeral: when the Pod dies, its disk dies with it. Write a file inside a Pod, let the Pod restart, and the file is gone.

   Pod v1                  Pod dies            Pod v2 (rescheduled)
   ┌──────────────┐                            ┌──────────────┐
   │ /data/db.sqlite│  ──── crash / evict ───► │ /data/  EMPTY │
   │  10 GB of rows │       (disk discarded)    │  fresh, blank │
   └──────────────┘                            └──────────────┘
                     all data lost ✗

For a database, a queue, anything that remembers, this is a catastrophe. The whole “cattle, not pets” model that makes orchestration scale is exactly wrong for stateful data. Reconciling these two worlds is the hardest part of Kubernetes, and this page is about the machinery that does it.

Decoupling the disk from the Pod

The fix is to make storage a separate object with its own lifecycle, attached to the Pod rather than born and killed with it. Kubernetes splits this into two pieces, mirroring the Service/Endpoints split you’ve seen — what I want vs what fulfills it:

• A PersistentVolume (PV) is a real piece of storage in the cluster — a cloud disk (EBS, Persistent Disk), an NFS share. It’s the supply: infrastructure.
• A PersistentVolumeClaim (PVC) is a Pod’s request for storage: “I need 10Gi, read-write.” It’s the demand: the app’s point of view.

Kubernetes binds a claim to a volume that satisfies it. The Pod references the PVC by name and never knows or cares which physical disk backs it — the same indirection that lets the disk outlive the Pod.

   Pod  ──mounts──►  PVC (claim: "10Gi RWO")  ──bound to──►  PV (real EBS volume)
        the abstraction the app asks for          the actual disk on infrastructure
   Pod can die and be replaced; the PVC and PV persist and re-attach.

StorageClasses: dynamic provisioning

Manually creating a PV for every claim is the old, error-prone way — an ops person clicking “create disk” and hand-writing YAML to match. A StorageClass automates it. It names a kind of storage (fast SSD, cheap HDD) and a provisioner that creates the underlying disk on demand when a PVC asks:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com      # the cloud's CSI driver
parameters: { type: gp3 }
reclaimPolicy: Delete             # delete the disk when the PVC is deleted

A claim just names the class; the disk is provisioned automatically:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: db-data
spec:
  accessModes: [ReadWriteOnce]    # one node mounts it read-write
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 10Gi

kubectl get pvc db-data          # STATUS: Bound  → a PV was auto-created
kubectl get pv                   # the dynamically provisioned volume

Access modes are a hardware truth, not a preference

ReadWriteOnce (RWO) means one node mounts it read-write — the common case for a block disk, because a cloud block volume physically attaches to one VM. ReadWriteMany (RWX) needs a shared filesystem like NFS. You can’t wish a single EBS volume into being shared by ten Pods; the access mode reflects what the storage can actually do.

StatefulSets: when Pods need identity

A Deployment treats its Pods as interchangeable — web-7d9 and web-x2k are the same, in any order. A clustered database can’t live like that. The primary and the replicas have roles; db-0 must keep being db-0 across restarts, with its own disk, its own stable name, started in order. That’s what a StatefulSet provides over a Deployment:

• Stable network identity — Pods are named db-0, db-1, db-2, not random hashes.
• Stable per-Pod storage — each Pod gets its own PVC (data-db-0, data-db-1) that re-attaches to the same Pod when it restarts. db-0 always remounts db-0’s data.
• Ordered, graceful creation, scaling, and deletion (db-0 before db-1), so a cluster can form and dissolve safely.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db                 # the headless Service (below)
  replicas: 3
  selector: { matchLabels: { app: db } }
  template:
    metadata: { labels: { app: db } }
    spec:
      containers:
        - name: postgres
          image: postgres:16
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:           # one PVC PER Pod, auto-created
    - metadata: { name: data }
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: fast-ssd
        resources: { requests: { storage: 10Gi } }

Headless Services give each Pod a DNS name

A normal Service load-balances across Pods behind one virtual IP — useless when you must address db-0 specifically to send a write to the primary. A headless Service (clusterIP: None) skips the virtual IP and instead gives every Pod its own stable DNS record:

apiVersion: v1
kind: Service
metadata: { name: db }
spec:
  clusterIP: None                 # headless: no virtual IP, per-Pod DNS
  selector: { app: db }
  ports: [{ port: 5432 }]

   db-0.db.default.svc.cluster.local  → the primary
   db-1.db.default.svc.cluster.local  → replica 1
   db-2.db.default.svc.cluster.local  → replica 2

Now replicas can find the primary by name, and the cluster can reason about its own members.

Why stateful workloads are genuinely hard on Kubernetes

It’s worth saying plainly: Kubernetes gives you durable disks and stable identity, but it does not understand your database. It won’t promote a replica when the primary dies, handle failover, or run a correct backup — those are application-level concerns the platform can’t infer. This is why production databases on Kubernetes almost always run via an Operator (covered in Kubernetes Operators & CRDs): a controller that encodes the operational knowledge of running that specific database. And it’s why many teams deliberately run their database as a managed service (Managed Services) and keep only stateless workloads in the cluster.

The honest default

Stateless in Kubernetes, stateful outside it — unless you have an Operator and the appetite to operate it. Kubernetes makes ephemeral easy and stateful possible, not stateful easy.

Why this makes production safer

The thread again: what manual, error-prone step does this remove? Without these objects, persisting data means an admin manually creating a disk, attaching it to the right VM, hoping the right Pod lands there, and re-doing it all on failure — every step a chance to lose data or mis-mount it. PVCs and StorageClasses turn that into a declared request fulfilled automatically; StatefulSets guarantee the right disk re-attaches to the right Pod with a stable name, so a database node can die and come back as itself instead of as a blank slate. The error-prone “wire the disk to the box by hand” step disappears — and the data survives the very Pod churn that makes the rest of Kubernetes safe. Next: how the scheduler decides where Pods run and how the cluster scales.

Under the hood — the multi-attach trap on failover

ReadWriteOnce means one node may mount the volume — and that guarantee is what makes StatefulSet failover surprisingly sticky. When a node dies ungracefully (a kernel panic, a yanked power cable), Kubernetes can’t confirm the old node ever released the disk. To avoid the one thing that would corrupt a database — two nodes writing the same block volume at once — it refuses to attach the volume elsewhere, so the rescheduled db-0 sits in ContainerCreating with a Multi-Attach error for volume… until the stale attachment is cleared (a timeout, a force-detach, or removing the dead node). It’s safety working as intended, but it means stateful failover is slower and more manual than the instant Pod replacement you get for stateless work — one more reason a database is harder here than a web tier. The full storage path is in Storage Internals (CSI).

The architect’s lens

Step back from PVs and StatefulSets and judge running state on Kubernetes at all:

• Why does it exist? Because a container’s filesystem is ephemeral — it dies with the Pod — yet databases must remember. PVs/PVCs decouple the disk from the Pod’s lifecycle, StorageClasses provision it on demand, and StatefulSets add stable identity (db-0 always remounts db-0’s data).
• What problem does it solve? It removes the manual “create a disk, attach it to the right VM, hope the right Pod lands there” toil — a declared PVC is fulfilled automatically, and the right disk re-attaches to the right named Pod so a DB node comes back as itself, not a blank slate.
• What are the trade-offs? Kubernetes gives durable disks but does not understand your database — no failover, no promotion, no correct backup. Access modes are a hardware truth (ReadWriteOnce = one node, because a cloud block volume attaches to one VM), and ungraceful node death triggers the Multi-Attach error standoff that makes stateful failover slower and more manual than stateless.
• When should I avoid it? When you can run the database as a managed service — the honest default is stateless in the cluster, stateful outside it, unless you have an Operator and the appetite to operate it.
• What breaks if I remove it? Any data written to a Pod vanishes on its next restart, the reconciliation loop’s freedom to kill and replace Pods turns from a feature into data loss, and persistence reverts to hand-wiring disks to boxes — the very step these objects delete.

Check your understanding

1. Why is a container’s filesystem ephemeral, and why is that a feature for a web app but a catastrophe for a database?
2. Distinguish a PersistentVolume from a PersistentVolumeClaim. Which represents supply and which represents demand, and why is the indirection useful?
3. What does a StorageClass automate, and what changes in the day-to-day workflow when one exists?
4. Name three guarantees a StatefulSet gives that a Deployment does not, and explain why a headless Service is needed alongside it.
5. Using the book’s thread, explain why “stateless in the cluster, stateful outside” is a common default — what does Kubernetes not know how to do for your database?

Show answers

1. A container’s filesystem is born and killed with the Pod — when the Pod dies, its disk is discarded. For a web app that’s a feature (nothing to lose, so Pods are freely replaceable and self-healing works); for a database it’s a catastrophe, because the rows are gone with the Pod.
2. A PersistentVolume (PV) is the real disk — the supply (infrastructure); a PersistentVolumeClaim (PVC) is the Pod’s request — the demand (“I need 10Gi RWO”). The indirection lets the Pod reference storage by claim name without knowing which physical disk backs it, so the disk can outlive and re-attach across Pod replacements.
3. A StorageClass automates dynamic provisioning — its provisioner creates the underlying disk on demand when a PVC asks, instead of an ops person manually clicking “create disk” and hand-writing matching PV YAML. Day to day, you just declare a claim naming the class and a correctly-sized volume appears and binds automatically.
4. Stable network identity (db-0, db-1 instead of random hashes), stable per-Pod storage (each Pod keeps its own PVC that re-attaches to the same Pod), and ordered, graceful creation/scaling/deletion. A headless Service (clusterIP: None) is needed because a normal Service load-balances behind one virtual IP, but you must address db-0 specifically (e.g. to write to the primary) — headless gives each Pod its own DNS record.
5. Because Kubernetes gives you durable disks and stable identity but does not understand your database — it won’t promote a replica on primary failure, handle failover, or run a correct backup. Those need an Operator that encodes the database’s operational knowledge; absent that (and the appetite to run it), teams keep stateless workloads in the cluster and run the database as a managed service.