Storage Internals (CSI)

Storage & Stateful Workloads introduced the user-facing objects: a PersistentVolumeClaim is the demand, a PersistentVolume is the supply, a StorageClass automates the matchmaking, and a StatefulSet keeps the right disk attached to the right Pod. That page deliberately hand-waved the verb in the middle — “the disk is provisioned automatically.” This page is that verb, broken open.

When you create a PVC and seconds later a cloud disk exists, is attached to the right node, and is mounted inside your container, a precise sequence of components and remote calls just ran. Understanding it is the difference between “storage is magic” and “I can debug why my Pod is stuck ContainerCreating with a FailedAttachVolume event.” The machinery is CSI, the Container Storage Interface.

Why CSI exists at all

Originally, every storage backend — EBS, GCE disks, Ceph, NFS — was a driver compiled into Kubernetes itself (“in-tree”). That meant the storage vendors’ code shipped inside the Kubernetes binary, on Kubernetes’ release schedule, with a bug in any driver able to crash the kubelet. It didn’t scale and it coupled two things that should be independent.

CSI is the decoupling: a standard gRPC interface that any storage system implements out-of-tree, as its own pods, on its own release cadence. Kubernetes speaks CSI; the vendor ships a CSI driver that translates CSI calls into their storage system’s API. The same interface is used by Nomad and Mesos too — it’s an industry standard, not a Kubernetes one. This is the reconciliation-loop philosophy applied to storage: Kubernetes declares “this volume should be attached and mounted,” and the driver’s controllers make it so.

The two halves of a CSI driver

A CSI driver isn’t one program — it’s split by where the work has to happen, and this split is the single most important thing to internalize.

   CONTROLLER PLUGIN (one, runs anywhere)        NODE PLUGIN (a DaemonSet, one per node)
   ┌──────────────────────────────────┐          ┌──────────────────────────────────┐
   │ talks to the cloud/storage API     │          │ runs ON the node where the Pod is  │
   │ • CreateVolume   (make the disk)   │          │ • NodeStageVolume  (format+mount   │
   │ • DeleteVolume                     │          │     to a global per-node path)     │
   │ • ControllerPublishVolume          │          │ • NodePublishVolume (bind-mount    │
   │     (attach disk to a node/VM)      │          │     into the Pod's directory)      │
   │ • CreateSnapshot                   │          │ • NodeUnpublish / NodeUnstage      │
   └──────────────────────────────────┘          └──────────────────────────────────┘
        runs as a Deployment with                      runs as a privileged DaemonSet
        sidecars (provisioner, attacher…)              because mounting needs node access

• The controller plugin does the cloud-API work: create a disk, attach it to a VM. This needs cloud credentials, not node access, so it runs as an ordinary Deployment (often a single active replica via leader election).
• The node plugin does the kernel work: format and mount the attached disk into the container. This must run on the specific node where the Pod landed — you can’t mount a disk into a container from across the cluster — so it’s a privileged DaemonSet, one instance per node, reached by the kubelet over a Unix socket.

CSI splits its calls along exactly this line: Controller service calls (CreateVolume, ControllerPublishVolume) go to the controller plugin; Node service calls (NodeStageVolume, NodePublishVolume) go to the node plugin on the right node.

The sidecars do the Kubernetes translation, not the driver

The CSI driver only speaks gRPC; it knows nothing about PVCs or VolumeAttachments. Community sidecar containers — external-provisioner, external-attacher, external-snapshotter, node-driver-registrar — watch Kubernetes objects and translate them into CSI calls. The external-provisioner sees a pending PVC and calls CreateVolume; the external-attacher sees a VolumeAttachment and calls ControllerPublishVolume. This keeps the vendor’s driver tiny and Kubernetes-agnostic.

The full lifecycle: PVC to mounted volume

Here is the end-to-end sequence for a dynamically provisioned volume — the four CSI calls in their actual order, and which component drives each.

  1. You create a PVC (storageClassName: fast-ssd)
        │
  2. external-provisioner (controller) sees pending PVC ──► CSI CreateVolume
        │                                                     → cloud creates a real disk
        │  ◄── a PV object is created and BOUND to the PVC
        │
  3. Pod is scheduled to node-7. kube-controller-manager creates a VolumeAttachment.
        │
  4. external-attacher (controller) sees it ──► CSI ControllerPublishVolume
        │                                          → cloud ATTACHES the disk to node-7's VM
        │
  5. kubelet on node-7 needs the volume for the Pod:
        ├─► CSI NodeStageVolume   → format (if new) + mount to a GLOBAL path:
        │       /var/lib/kubelet/plugins/.../globalmount   (once per node, shared)
        └─► CSI NodePublishVolume → BIND-MOUNT global path into the Pod's dir:
                /var/lib/kubelet/pods/<uid>/volumes/.../mount   (once per Pod)
        │
  6. Container starts with the volume mounted. ✓

The two-step mount in stage 5 looks redundant but isn’t. NodeStageVolume mounts the disk once per node to a global staging path and formats it if it’s blank; NodePublishVolume then bind-mounts that staging path into each Pod that uses the volume. The split lets one ReadWriteMany volume serve several Pods on the same node — stage once, publish many — and cleanly separates “is the disk usable on this node” from “is it visible to this Pod.”

   NodeStageVolume (once/node)     NodePublishVolume (once/Pod)
   raw disk ──format+mount──► /…/globalmount ──bind-mount──► /…/pods/<uidA>/…/mount
                                            └─────bind-mount──► /…/pods/<uidB>/…/mount

Teardown runs the mirror image: NodeUnpublishVolume (un-bind from the Pod), NodeUnstageVolume (unmount the global path), ControllerUnpublishVolume (detach from the VM), and — only if the reclaim policy says so — DeleteVolume.

PV/PVC binding and the provisioning flow

Binding is its own small controller (the PV controller in kube-controller-manager). Its job: marry every Pending PVC to a suitable Available PV, and trigger dynamic provisioning when none exists.

   PVC created ──► PV controller looks for a matching Available PV
       ├─ found a static PV that satisfies size+accessMode+storageClass? ──► bind them
       └─ none, but storageClass has a provisioner? ──► dynamic provisioning:
              external-provisioner calls CreateVolume, makes a PV, binds it
   PVC.status: Bound   PV.status: Bound   (a 1:1, exclusive binding)

A binding is exclusive and bidirectional — one PVC ↔ one PV, each pointing at the other via claimRef / volumeName. Once bound it’s sticky: even if the PVC is deleted and the PV’s reclaim policy is Retain, that PV stays marked for the old claim (status Released) and won’t auto-bind to a new claim, precisely so stale data can’t silently leak to the next tenant.

StorageClass and dynamic provisioning

The storage page showed a StorageClass; here’s what its fields do to the flow:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: { name: fast-ssd }
provisioner: ebs.csi.aws.com           # which CSI driver to call
parameters:                            # passed verbatim into CreateVolume
  type: gp3
  encrypted: "true"
reclaimPolicy: Delete                  # Delete vs Retain — what happens to the disk on PVC delete
allowVolumeExpansion: true             # may a PVC request more space later?
volumeBindingMode: WaitForFirstConsumer

provisioner selects the driver. parameters are an opaque key/value bag handed straight to the driver’s CreateVolume — the driver defines what they mean (here, an EBS volume type and at-rest encryption). reclaimPolicy and volumeBindingMode are important enough to get their own sections.

Reclaim policies: what happens to the data

When a PVC is deleted, what becomes of the underlying disk? The PV’s persistentVolumeReclaimPolicy decides:

Policy	On PVC delete	Use when
Delete	the PV and the real cloud disk are deleted	ephemeral/dev data; the common dynamic default
Retain	the disk and PV survive (status Released); manual cleanup	data you must not lose to an accidental kubectl delete pvc

Delete means a deleted PVC destroys real data

The default reclaim policy for most dynamic StorageClasses is Delete. So kubectl delete pvc db-data — or deleting the namespace it lives in — can silently destroy a production database’s disk. For anything durable, set the StorageClass (or the PV) to Retain, and treat the underlying disk’s lifecycle as a separate, deliberate decision from the Kubernetes object’s.

Topology and WaitForFirstConsumer: bind where the Pod will land

Cloud disks have a brutal constraint: a disk in availability zone us-east-1a cannot be attached to a VM in us-east-1b. Now picture the naive flow: a PVC is created, the provisioner immediately makes a disk in some zone, then the scheduler tries to place the Pod — and discovers the only nodes with spare capacity are in a different zone than the disk. The Pod is unschedulable, permanently. The decision order was backwards.

volumeBindingMode: WaitForFirstConsumer fixes this by delaying provisioning until a Pod actually needs the volume, so the scheduler picks the node first and the disk is then created in that node’s zone.

   Immediate binding (the trap)              WaitForFirstConsumer (correct)
   ─────────────────────────────             ──────────────────────────────
   PVC ──► disk made in zone-a (blind)        PVC ──► (waits, status Pending)
   Pod ──► scheduler: only zone-b free        Pod ──► scheduler picks node in zone-b
        ──► can't attach zone-a disk ✗             ──► NOW provision disk in zone-b ✓

The provisioner learns the chosen zone from the PV’s topology — a CSI driver reports the topology keys it cares about (zone, region, rack) via each node plugin’s NodeGetInfo response, and the scheduler treats them as scheduling constraints. This is why a dynamically provisioned WaitForFirstConsumer PVC sits Pending until you create a Pod that mounts it: that’s correct, not broken. It’s the same constraint-satisfaction problem the scheduler solves everywhere, with the disk’s zone added as one more constraint.

Volume snapshots

CSI also standardizes point-in-time snapshots, exposed as Kubernetes objects that mirror the PV/PVC pair:

• A VolumeSnapshotClass is the snapshot analogue of a StorageClass (which driver, what parameters).
• A VolumeSnapshot is the request for a snapshot (analogue of a PVC).
• A VolumeSnapshotContent is the real snapshot in the backend (analogue of a PV).

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata: { name: db-backup-friday }
spec:
  volumeSnapshotClassName: ebs-snapshots
  source:
    persistentVolumeClaimName: db-data    # snapshot this PVC's current state

The external-snapshotter sidecar sees the VolumeSnapshot and calls CreateSnapshot on the controller plugin. You can later create a new PVC with dataSource pointing at the snapshot to restore it — the clone is provisioned from the snapshot instead of blank. This is infrastructure for backups, but it’s not a backup strategy: a crash-consistent disk snapshot may catch a database mid-write. As the storage page warned, a correct database backup is application-level knowledge an Operator encodes; CSI gives you the primitive, not the policy.

By the numbers — why a dead node strands an RWO volume for ~6 minutes

A ReadWriteOnce volume can be attached to exactly one node at a time. So when a node fails hard, the disk is still attached to the dead machine, and the Pod cannot simply move:

   node-7 dies (NotReady)          RWO disk still attached to node-7's VM
        │  a ~6-minute force-detach timeout must elapse first
        ▼
   attach-detach controller force-detaches from node-7
        │  ControllerPublishVolume attaches it to the new node
        ▼
   Pod finally schedules — until then it shows:  Multi-Attach error for volume

That roughly six-minute default is deliberate caution: force-detaching a volume that is actually still mounted (a node that’s merely unreachable, not dead) risks corruption, so the controller waits to be sure. It’s the single most common “my StatefulSet Pod is stuck ContainerCreating after a node died” surprise — by-design safety, not a bug, and the reason genuinely HA stateful systems replicate at the application layer rather than trusting one disk to re-attach quickly.

The thread: what manual step disappears

The recurring question: what manual, error-prone step does this remove — and how does it make production safer? The manual step is an operator at a cloud console: create a disk in the right zone, attach it to the right VM, SSH in to format and mount it, hope the Pod lands on that exact VM, and reverse every step by hand on teardown — each one a chance to lose data, mount the wrong disk, or strand a Pod in the wrong zone. CSI turns that entire sequence into a declared PVC that the controller and node plugins fulfill through a precise, idempotent gRPC handshake; WaitForFirstConsumer removes the disk-in-the-wrong-zone class of bug structurally; snapshots make point-in-time copies a declarative object instead of a console click. The cost is real operational surface — a driver, its sidecars, and a DaemonSet to run and debug, and reclaim policies that can destroy real data if set carelessly — which is exactly why knowing this internal flow matters when a volume won’t attach. Next, we turn from where data lives to how the cluster grows and shrinks to meet load: Autoscaling Internals.

The architect’s lens

Step back from the gRPC handshake and answer the five questions behind adopting CSI:

• Why does it exist? Because in-tree drivers shipped storage-vendor code inside the Kubernetes binary — a driver bug could crash the kubelet, and drivers were chained to Kubernetes’ release schedule. CSI decouples them as a standard gRPC interface any backend implements out-of-tree, on its own cadence (and it’s a cross-orchestrator standard, not Kubernetes-only).
• What problem does it solve? It turns “create a disk in the right zone, attach it to the right VM, format and mount it, reverse on teardown” into a declared PVC fulfilled by a precise sequence — CreateVolume → ControllerPublishVolume → NodeStageVolume → NodePublishVolume — split across a controller plugin (cloud API) and a per-node DaemonSet (kernel mount).
• What are the trade-offs? Real operational surface: a driver, its sidecars (external-provisioner, -attacher, -snapshotter), and a privileged DaemonSet to run and debug; reclaimPolicy: Delete (the dynamic default) means a stray kubectl delete pvc destroys real data; and a dead node strands an ReadWriteOnce volume for the ~6-minute force-detach timeout.
• When should I avoid the defaults? Set Retain, not Delete, for anything durable; keep WaitForFirstConsumer so the disk is provisioned in the node’s zone rather than blindly (a zone-mismatched disk can never attach); and don’t treat a crash-consistent CSI snapshot as a database backup — that’s application-level knowledge an Operator must encode.
• What breaks if I remove it? Storage reverts to an operator at a cloud console doing every create/attach/format/mount by hand and hoping the Pod lands on that exact VM; the structural fix for wrong-zone disks disappears; and snapshots stop being declarative objects.

Check your understanding

1. Why was the move from in-tree storage drivers to CSI worth making? Name two concrete problems the in-tree model caused.
2. A CSI driver has a controller plugin and a node plugin. What work does each do, where does each run, and why can’t the node work be done from a central controller?
3. Put the four core CSI calls — CreateVolume, ControllerPublishVolume, NodeStageVolume, NodePublishVolume — in order and say what each accomplishes. Why is the node-side mount split into two steps?
4. Your dynamically provisioned PVC sits Pending and never binds until you create a Pod. Is this a bug? Explain what WaitForFirstConsumer is preventing.
5. Using the book’s thread, contrast reclaimPolicy: Delete vs Retain and explain why the default can be dangerous for a production database — then state the manual toil CSI removes overall.

Show answers

1. In-tree drivers shipped storage-vendor code inside the Kubernetes binary, so a driver bug could crash the kubelet and drivers were stuck on Kubernetes’ release schedule. CSI moves drivers out-of-tree as their own pods on their own cadence, decoupling two things that should be independent (and it’s a cross-orchestrator standard, not Kubernetes-only).
2. The controller plugin does cloud-API work — create the disk, attach it to a VM — needs cloud creds, not node access, so it runs as a Deployment. The node plugin does kernel work — format and mount — which must happen on the specific node where the Pod landed, so it’s a privileged DaemonSet (one per node). You can’t mount a disk into a container from across the cluster, hence the node-local requirement.
3. CreateVolume (controller) makes the real disk; ControllerPublishVolume (controller) attaches it to the chosen node’s VM; NodeStageVolume (node) formats if blank and mounts it to a global per-node path; NodePublishVolume (node) bind-mounts that global path into the Pod’s directory. The split exists so the disk is staged once per node (NodeStage) and then published once per Pod (NodePublish), letting one volume serve multiple Pods on the node.
4. Not a bug. With WaitForFirstConsumer, provisioning is delayed until a Pod needs the volume so the scheduler picks the node first and the disk is created in that node’s zone. Immediate binding would create the disk in a blind zone, and if the Pod could only be scheduled in a different zone, the disk (which can’t cross zones) could never attach — a permanently unschedulable Pod.
5. Delete removes the PV and the real cloud disk when the PVC is deleted (the common dynamic default), so kubectl delete pvc — or deleting its namespace — can destroy a production database’s disk. Retain keeps the disk (status Released) for manual handling, which is what durable data needs. Overall, CSI removes the manual, error-prone toil of an operator creating/attaching/formatting/ mounting disks by hand in the right zone and reversing it on teardown — replacing it with a declared PVC fulfilled by an idempotent gRPC handshake.