Secrets Management

Config & Secrets ended on a sharp warning: a Kubernetes Secret is base64, not encryption, and “it’s in a Secret” is not the same as “it’s protected.” That page named four layers of real protection and promised this one would treat the topic properly. Here it is — the why and the how of keeping credentials out of the wrong hands.

A secret is any value whose disclosure compromises the system: a database password, an API token, a TLS private key, a signing key. The defining property of a secret is that the blast radius of a leak is enormous and often invisible — a leaked key can be used quietly for months. So the entire discipline is about shrinking two things: how many places a secret exists, and how long any copy of it is valid.

Why the easy places are the dangerous places

The tempting places to put a secret are exactly the ones that leak.

   WHERE SECRETS GO TO DIE                         WHY IT'S A BREACH
   ──────────────────────                          ─────────────────
   committed to git        ──►  in history FOREVER, even after you delete it
   .env file on disk       ──►  copied to laptops, backups, CI logs, Slack
   baked into an image     ──►  anyone who can `docker pull` reads every layer
   pasted in CI config      ──►  printed in build logs, shared with every job

• Git. A secret committed to git is leaked forever, because git history is immutable by design. Deleting the file in a later commit does nothing — the value is still in the history, in every clone, in every fork. (This is precisely why the pre-commit secret scanners exist: the only cheap fix is to never let it in.)
• Env files. A .env on disk gets copied into backups, synced to laptops, and accidentally printed into CI logs. It’s plaintext sitting in a dozen places you’ve forgotten about.
• Images. Bake a secret into a Docker layer and anyone who can pull the image can read it — layers are not encrypted, and docker history shows the build commands. Multi-stage builds don’t save you if the secret lands in the final stage.

git rm does not un-leak a secret

Once a credential is in git history, treat it as compromised and rotate it immediately — do not just delete the file and move on. Rewriting history is painful, incomplete (forks and clones already have it), and still leaves the window where it was exposed. Rotation is the real remedy; prevention is the only cheap one.

A secret store: one source of truth, access-controlled

The fix for “a secret exists in twelve places” is a secret store (or secrets manager): a single, access-controlled, audited service that holds the master copy. Applications and pipelines request a secret from the store at runtime instead of carrying their own copy. Examples: HashiCorp Vault, and the cloud-native managers — AWS Secrets Manager, GCP Secret Manager, Azure Key Vault — often backed by a KMS (Key Management Service) that holds the encryption keys.

What a real secret store buys you over a .env file:

Property	.env / git	Secret store
Copies of the secret	Many, uncontrolled	One source of truth
Encryption at rest	None (plaintext)	Encrypted under a KMS key
Access control	Filesystem permissions	Per-identity policy, fine-grained
Audit	None	Every read is logged: who, what, when
Rotation	Manual, everywhere	Centralized, often automatic

Encryption at rest is the floor: the store keeps secrets encrypted on disk, decrypting only to serve an authorized request. The encryption keys themselves live in a KMS, so even someone who steals the storage backend gets ciphertext. (This is the same fix Config & Secrets called “table stakes” for etcd — encrypt before it hits disk.)

The bigger leap: dynamic, short-lived secrets

Encrypting a static password is good. Not having a long-lived password at all is better. The most powerful idea in modern secrets management is the dynamic secret: instead of storing a permanent database credential, the store generates a fresh, short-lived credential on demand and revokes it automatically when its lease expires.

   STATIC SECRET                         DYNAMIC SECRET
   ─────────────                         ──────────────
   one password, lives for years         app asks Vault for db creds
   shared by every instance              Vault CREATES a new db user
   leak = valid until someone notices    creds valid for 1 hour (a LEASE)
   rotation = scary, manual, rare        lease expires → Vault DELETES the user
                                         leak = useless in an hour

This collapses the second variable — how long a copy is valid — to near zero:

• Leasing. Every dynamic secret comes with a lease (a TTL). When the lease ends, the credential is revoked at the source. An app renews its lease while it’s running; a leaked credential simply expires.
• Rotation. Because credentials are generated per-request and short-lived, rotation is continuous and automatic rather than a terrifying quarterly fire drill. There’s no “the password changed, go update seventeen services” — each service just requests a fresh one.
• Revocation. Suspect a compromise? Revoke the lease and every credential issued under it dies at once, without redeploying anything.

The goal is a secret that’s useless when stolen

Static-secret thinking asks “how do we keep the password from leaking?” Dynamic-secret thinking asks “so what if it leaks?” — because a credential that’s already expired is worthless to an attacker. Aim to make leaks boring, not just unlikely.

Secrets in GitOps: SOPS and Sealed Secrets

There’s a real tension with GitOps, where git is the source of truth for everything the cluster runs — including Secret manifests. But you just learned plaintext secrets must never touch git. The resolution is to commit secrets encrypted, so git holds ciphertext only and the cluster decrypts at apply time.

• SOPS (Secrets OPerationS) encrypts the values in a YAML/JSON file while leaving the keys readable, so the file stays diff-friendly. Encryption keys come from a KMS or age/PGP key. You commit the encrypted file; a controller or the deploy step decrypts it.
• Sealed Secrets uses a controller in the cluster that holds a private key. You encrypt a Secret with the matching public key into a SealedSecret resource that only that cluster can decrypt, then commit it safely. Even a public repo is fine — the ciphertext is useless without the cluster’s key.

# Sealed Secrets: encrypt locally, commit the result, the cluster decrypts.
kubectl create secret generic db-creds \
  --from-literal=password='s3cr3t' --dry-run=client -o yaml \
  | kubeseal --format yaml > sealed-db-creds.yaml   # safe to commit to git

A complementary pattern is the External Secrets Operator: don’t store secrets in git at all, store a reference, and have an in-cluster operator fetch the real value from Vault or a cloud manager and materialize a short-lived Kubernetes Secret. Git holds the pointer; the secret store holds the truth.

It actually happened — Uber’s keys in a private repo (2016)

In 2016, attackers reached a private GitHub repository used by Uber engineers and found AWS access keys committed in the code. Those long-lived keys unlocked an S3 datastore holding personal data on roughly 57 million riders and drivers. Uber quietly paid the attackers 100,000 USD to delete the data and stay silent — a cover-up that later drew regulatory penalties and a criminal conviction for the security executive involved.

Every protection on this page would have blunted it. The credential should never have been in source control at all — a pre-commit scanner catches that — and it should have lived in an audited secret store rather than scattered copies. Most decisively, as a dynamic, short-lived credential it would have expired long before an attacker browsing an old repo could use it. A static key in git is precisely the “valid for years, leaked invisibly” failure this page opened with — made real, at scale.

The thread: what manual step disappears

The manual, error-prone step secrets management removes is the human shuffling of long-lived credentials — pasting passwords into config files, sharing them in chat, copying keys onto new servers, and performing rare, scary, all-hands rotations when something leaks. In its place: a single audited store, encryption at rest by default, and credentials that are generated fresh, leased short, and revoked automatically.

Production gets safer along both axes the page opened with. Where the secret exists shrinks to one controlled, logged place instead of a dozen forgotten copies. How long a copy is valid shrinks from years to an hour, so a leak is contained by the clock rather than by someone noticing. The cost is real operational machinery — a store to run, identities to wire up, an outage in the store now affecting every app that needs a credential — but in exchange, “who has the database password?” finally has an answer: nobody, permanently; the store issues a new one each time. Next, a different trust problem entirely — the code you ship but didn’t write: Supply-Chain Security.

The architect’s lens

Five questions for managing secrets:

• Why does it exist? Because a secret’s blast radius is enormous and a leak can go unnoticed for months, so the whole discipline is shrinking two things: how many places a secret exists and how long any copy stays valid.
• What problem does it solve? Secrets dying in git/env/images/CI logs: a secret store gives one audited, encrypted-at-rest source of truth, and dynamic, short-lived credentials make a leaked secret expire on its own — so rotation is continuous instead of a scary fire drill.
• What are the trade-offs? Real operational machinery — a store to run, identities to wire, and a store outage now hits every app that needs a credential — plus GitOps forcing encrypted-in-git patterns (SOPS, Sealed Secrets, External Secrets Operator).
• When should I avoid it? You don’t avoid it for real secrets; the full Vault/dynamic-secret stack may be overkill for a solo hobby project — but even there, a secret committed to git is leaked forever.
• What breaks if I remove it? Credentials scatter into a dozen forgotten plaintext copies valid for years — exactly the Uber-2016 failure (static AWS keys in a repo, 57M records) — and “who has the password?” has no answer.

Check your understanding

1. Why is a secret committed to git considered leaked forever, and why is deleting the file in a later commit not a fix? What is the only real remedy?
2. List three properties a real secret store gives you that a .env file cannot, and say why each matters.
3. Explain a dynamic secret with leasing. How does it shrink the “how long is a leak useful?” window, and why does that make rotation routine instead of scary?
4. GitOps wants everything in git, but secrets must never be in git in plaintext. Explain how SOPS or Sealed Secrets resolves that tension.
5. Using the book’s thread, name the manual step removed and describe the two axes — where and how long — along which production becomes safer.

Show answers

1. Git history is immutable by design, so the value persists in the history, in every clone, and in every fork even after you delete the file — rewriting history is painful and incomplete. The only real remedy is to rotate the credential (treat it as compromised); prevention via pre-commit scanning is the only cheap option.
2. For example: one source of truth (no scattered copies to forget); encryption at rest under a KMS (stealing the backend yields ciphertext); per-read audit logging (you know who accessed what, when). Fine-grained per-identity access control and centralized rotation are also valid answers.
3. A dynamic secret is generated on demand (e.g. Vault creates a fresh DB user) and comes with a lease (a TTL); when the lease expires the credential is revoked at the source. A leaked credential simply expires within the lease window, so the leak is contained by the clock. Rotation becomes routine because every credential is already short-lived and per-request — there’s no big coordinated swap.
4. Commit the secret encrypted, so git holds only ciphertext. SOPS encrypts the values in a file (keys stay readable) using a KMS/age/PGP key; Sealed Secrets encrypts to a SealedSecret that only the target cluster’s private key can decrypt. Either way the cluster decrypts at apply time; the repo never holds plaintext. (External Secrets Operator goes further: store only a reference.)
5. Removed: the human shuffling of long-lived credentials — pasting them into files, sharing in chat, copying to servers, doing rare scary rotations. Safer on two axes: where the secret exists shrinks to one audited place instead of many forgotten copies; how long a copy is valid shrinks from years to an hour, so leaks are contained automatically.