Kubernetes Operators & CRDs

The reconciliation loop page promised that operators “extend the pattern to your own objects with custom controllers.” This is where that promise is kept. By now you can recite the loop: a controller observes actual state, diffs it against declared desired state, and acts to close the gap, forever — and the control loop is everywhere page showed it’s the master pattern. An operator is the moment you stop just using that pattern and start writing it: a control loop of your own, for a kind of object Kubernetes has never heard of.

The problem operators solve is the limit of the built-in objects. Kubernetes knows how to reconcile a Deployment — keep N stateless replicas alive. But it has no idea how to run your PostgreSQL cluster: how to take a backup, fail over to a replica, run a version upgrade safely, or grow storage. That knowledge lives in a human’s head and a runbook — the manual, woken-at-3-a.m. expert. Operators are how you move that knowledge out of the human and into the cluster.

Two halves: a new noun, and a controller for it

An operator is two pieces working together, and it helps to name them separately.

Custom Resource Definitions (CRDs) extend the Kubernetes API with new object types. Out of the box the API speaks Pod, Service, Deployment. A CRD teaches it a new noun — say, PostgresCluster — so that kubectl get postgrescluster works and the API server will store and serve objects of that kind, with validation, just like a built-in. You’ve added a word to the cluster’s vocabulary.

The operator (also called a custom controller) is the verb for that noun: a control loop you write that watches your custom resources and reconciles the real world to match them. The CRD lets you declare “I want a 3-node Postgres cluster, version X, with daily backups.” The operator is the program that makes that sentence true and keeps it true.

   BUILT-IN                              YOUR OPERATOR
   ────────                              ─────────────
   noun:  Deployment (k8s knows it)      noun:  PostgresCluster (a CRD you defined)
   verb:  built-in controller            verb:  your operator (a controller you wrote)
   loop:  keep N replicas alive          loop:  observe the DB cluster's real state
                                                compare to the spec (nodes, version,
                                                  backup schedule)
                                                act: provision, fail over, back up,
                                                  upgrade — exactly as an expert would

The shape is identical to the reconciliation loop you already know — that’s the whole point. You are not learning a new mechanism; you are pointing the mechanism you’ve understood since Part 4 at a problem only you understand.

The operator pattern is “your runbook, as a control loop”

The classic definition: an operator encodes the operational knowledge of a human expert running an application — the runbook for backups, failover, scaling, and upgrades — into software that runs that runbook continuously and automatically. Where you once paged a database expert to fail over a primary, an operator observes the primary is unhealthy and fails over itself, the same way, every time, in seconds.

A concrete spec

A custom resource looks and feels exactly like a built-in one — same YAML shape, same kubectl, same GitOps flow — which is precisely why it’s powerful. Here’s a (representative) Postgres cluster declared as one object:

apiVersion: db.example.com/v1
kind: PostgresCluster          # the noun a CRD added to the API
metadata:
  name: orders-db
spec:                          # YOUR desired state
  version: "16"
  instances: 3                 # 1 primary + 2 replicas
  storageGB: 100
  backup:
    schedule: "0 2 * * *"      # nightly
    retentionDays: 14
status:                        # the operator writes the OBSERVED state here
  phase: Running
  primary: orders-db-0
  readyInstances: 3

You kubectl apply this one small file. Behind it, the operator does everything a human DBA would: creates the StatefulSet and storage, designates a primary and configures replication, sets up the backup cron, and writes back status. Change instances: 3 to 5 and re-apply — the operator reconciles by adding two replicas. Change version: "16" to "17" — the operator runs the upgrade in the careful order the runbook specifies. The spec is your intent; the status is the operator reporting reality back to you. That spec/status split is the same observe-and-compare structure as every other controller.

Why this is a big deal

Without an operator	With an operator
Backups/failover/upgrades are runbooks a human executes	Encoded once, run automatically and identically every time
Stateful apps don’t fit Kubernetes’ stateless model well	Domain-specific logic teaches the cluster how to run them
The expert is a single point of failure (and they sleep)	The expertise runs 24/7 in the cluster
Config drift on the database is invisible	The operator continuously reconciles it back to spec

The deepest win is the same one the whole book keeps circling: a self-healing system needs no human in the loop for routine operations. A node dies and takes the Postgres primary with it; with an operator, failover happens in seconds, automatically, while everyone’s asleep — because the desired state (“a healthy primary plus two replicas”) is continuously re-asserted by a loop, not by a person on call.

Common real operators follow this exactly:

• Database operators run stateful clusters (Postgres, MySQL, Kafka, etc.) — provisioning, backups, failover, and upgrades as reconciled spec.
• Certificate operators treat a TLS certificate as a custom resource: declare “I want a valid cert for this hostname,” and the operator obtains it, installs it, and renews it before it expires — turning the classic 2-a.m.-expired-certificate outage into a loop nobody has to think about. (It’s the mTLS/identity machinery, automated.)

Use an operator when reconciliation needs domain knowledge

Not everything needs an operator. If a built-in object plus a Helm chart captures your app, use that — it’s simpler. The operator earns its complexity when running the thing correctly requires ongoing, stateful, domain-specific decisions — failover order, backup-then-upgrade sequencing, replica promotion — that a static template can’t make. The test: would you otherwise need a human expert watching this thing? Then encode the expert.

An operator is software you now own and operate

You’ve moved the runbook into code — which means a bug in the operator is now an automated, repeated, fleet-wide mistake. An operator that “fails over” incorrectly does so confidently and fast, exactly the control-loop failure mode of being wrong at machine speed. Operators that manage stateful data deserve the same testing rigor — and the same chaos testing — as the data they manage.

It actually happened — Microsoft Teams’ expired certificate (February 2020)

The cert-operator example above — “renew it before it expires (or, too often, just after)” — is not hypothetical. In February 2020, Microsoft Teams went globally unavailable for several hours because Microsoft had let an authentication certificate lapse without renewing it. One expired certificate took down a service used by millions, and the fix was the most mundane operations task there is: issue a new cert. This is exactly the failure a certificate operator deletes: you declare “I want a valid cert for this hostname,” and a control loop obtains, installs, and renews it on its own, well before expiry, so the outage simply can’t happen. The expensive lesson is that the most damaging incidents are often not exotic — they’re a routine renewal that no human owned.

The thread

The manual, error-prone step operators remove is a human expert executing a runbook — the DBA paged to fail over a primary, the engineer who remembers the exact safe order to upgrade a cluster, the person who renews the certificate before it expires (or, too often, just after). All of that is operational knowledge trapped in a head and a wiki page, available only when that person is awake and reachable. An operator encodes it once into a control loop and runs it continuously, identically, automatically. Production gets safer because the routine-but-critical operations that used to depend on the right human being available now happen on their own, the same correct way every time, in seconds — and because the cluster continuously corrects drift on stateful systems it previously couldn’t reason about at all.

The cost: you’ve written and must now operate a piece of software with privileged access to important state, where a bug is a fast, repeated, automated error. That’s a real trade — which is why operators are worth it exactly when the alternative is “we need an expert on call for this forever.” The next page zooms out from encoding one expert’s knowledge to encoding the whole organization’s: building an internal platform that turns operators, pipelines, and golden paths into a product the rest of engineering can self-serve — platform engineering.

The architect’s lens

Five questions before you write an operator:

• Why does it exist? Because Kubernetes reconciles a Deployment but has no idea how to run your Postgres — backup, failover, safe upgrade — knowledge that otherwise lives only in a human’s head and a runbook.
• What problem does it solve? It moves that runbook out of the 3 a.m. expert and into the cluster: a CRD adds the noun (PostgresCluster), the operator is the controller that reconciles it, so failover and cert renewal happen automatically and identically every time (the certificate that never lapses — recall Teams 2020).
• What are the trade-offs? You’ve written software with privileged access to important state, where a bug is an automated, repeated, fleet-wide mistake made confidently and fast — the control-loop “wrong at machine speed” failure.
• When should I avoid it? When a built-in object plus a Helm chart captures the app — reach for an operator only when running it correctly needs ongoing, stateful, domain-specific decisions.
• What breaks if I remove it? Stateful operations revert to a human expert who must be awake and reachable, and the cluster can’t reason about — or self-heal — the stateful system at all.

Check your understanding

1. Distinguish a CRD from an operator. Which one adds a noun to the API, and which one is the verb?
2. The reconciliation-loop page said operators “extend the pattern to your own objects.” Map a database operator onto the observe/compare/act loop — what plays desired and actual?
3. In the PostgresCluster YAML, what’s the difference between spec and status, and which one does the operator write? Why does that mirror every other controller?
4. State the operator pattern in one sentence (“an operator is …”). When is an operator worth its complexity, and when is a Helm chart enough?
5. Using the book’s thread, what manual role does an operator remove — and what new risk do you take on by encoding a runbook as an automated control loop?

Show answers

1. A CRD extends the Kubernetes API with a new object type (the noun, e.g. PostgresCluster) so the API server stores and serves it. An operator is the custom controller (the verb) — a control loop you write that reconciles instances of that type to your desired state.
2. Desired = the PostgresCluster spec (3 nodes, version 16, nightly backups). Actual = the real cluster’s observed state. The operator observes the DB’s real condition, compares it to the spec, and acts — provisioning, failing over, backing up, upgrading — then loops, forever.
3. spec is your declared desired state; status is the observed reality the operator writes back. The operator writes status. It mirrors every controller because that’s exactly the observe-and-compare split — declared intent vs reported actual.
4. “An operator encodes the operational knowledge of a human expert running an application into a control loop that runs it continuously and automatically.” It’s worth the complexity when running the thing correctly needs ongoing, stateful, domain-specific decisions (failover, upgrade ordering); a Helm chart is enough when a static template captures the app.
5. It removes a human expert executing a runbook (failover, safe upgrades, cert renewal) — knowledge that’s only available when that person is awake. The new risk: a bug in the operator is an automated, repeated, fleet-wide mistake, made confidently and fast (the control-loop “wrong at machine speed” failure).