Config & Secrets

There’s a principle the container Parts kept circling: build the image once, run it everywhere. The same artifact that ran in CI runs in staging runs in production. But “everywhere” differs — a different database URL, a different log level, a different API key. If those values are baked into the image, you need a different image per environment, which throws away the whole point and means rebuilding to change a setting. The manual alternative — SSHing in to edit a config file on each box — is exactly the error-prone toil DevOps exists to kill.

Kubernetes separates configuration from the image with two objects: ConfigMaps for non-sensitive config, Secrets for sensitive values. Same shape, different intent.

ConfigMaps: config as data

A ConfigMap is just key/value data living in the cluster, decoupled from any Pod:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  LOG_LEVEL: "info"
  FEATURE_CHECKOUT_V2: "true"
  app.properties: |
    timeout=30
    retries=3

kubectl create configmap app-config --from-literal=LOG_LEVEL=info
kubectl get configmap app-config -o yaml

Now the image carries code; the ConfigMap carries settings. Promote the identical image from staging to production and swap which ConfigMap it reads — no rebuild.

Two ways to inject: env vars vs volumes

A Pod consumes a ConfigMap (or Secret) in one of two ways, and the choice matters.

   ConfigMap / Secret                  Pod
   ┌───────────────┐    env vars   ┌─────────────────────┐
   │ LOG_LEVEL=info│ ────────────► │ process environment  │  (read once at start)
   │ app.properties│               │                      │
   └───────────────┘    volume     │ /etc/config/         │  (files; can update live)
                    ────────────►  │   app.properties     │
                                   └─────────────────────┘

As environment variables — simple, but read once at process start. Change the ConfigMap and the running Pod keeps the old value until it restarts.

spec:
  containers:
    - name: app
      image: myapp:1.4.2          # one immutable image
      envFrom:
        - configMapRef: { name: app-config }   # every key becomes an env var
      env:
        - name: LOG_LEVEL                       # or pick one key explicitly
          valueFrom:
            configMapKeyRef: { name: app-config, key: LOG_LEVEL }

As mounted files (a volume) — the ConfigMap appears as files in a directory, and Kubernetes updates those files when the ConfigMap changes (eventually, within a sync period). Apps that re-read config files can pick up changes without a restart:

spec:
  containers:
    - name: app
      image: myapp:1.4.2
      volumeMounts:
        - name: config
          mountPath: /etc/config
  volumes:
    - name: config
      configMap: { name: app-config }

Rule of thumb

Env vars for simple, restart-on-change settings; volumes for config files or anything you want to update without recreating the Pod. Either way, the value lives outside the image.

Secrets: same mechanics, sensitive intent

A Secret looks almost identical to a ConfigMap and is consumed the same two ways — but it’s meant for passwords, tokens, TLS keys. Kubernetes gives Secrets slightly different handling (kept out of some logs, mountable as tmpfs in memory) to signal “treat this carefully.”

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
stringData:                       # plaintext in; stored base64-encoded
  DB_PASSWORD: "s3cr3t-p@ss"

kubectl create secret generic db-credentials \
  --from-literal=DB_PASSWORD='s3cr3t-p@ss'

kubectl run app --image=myapp --env="DB_PASSWORD=$(...)"   # ✗ don't; use a ref

Inject by reference, never by pasting the value into a Pod spec:

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef: { name: db-credentials, key: DB_PASSWORD }

The trap: a Secret is base64, not encrypted

Here is the misconception that bites everyone, and the reason this page exists. A Kubernetes Secret is not encrypted by default — it is merely base64-encoded. Base64 is not a security measure; it’s a reversible text encoding so binary data survives in YAML. Anyone who can read the object reads the value:

kubectl get secret db-credentials -o jsonpath='{.data.DB_PASSWORD}' | base64 -d
# s3cr3t-p@ss   ← trivially recovered, no key required

Worse, by default Secrets are stored in etcd (the cluster’s data store) as base64 too. A backup of etcd, or read access to it, is a backup of your plaintext secrets.

   What "Secret" implies        What it actually is by default
   ┌────────────────┐           ┌──────────────────────────────┐
   │  🔒 encrypted   │   vs.     │  base64(value)  ← reversible  │
   │     at rest     │           │  stored in etcd, often plain  │
   └────────────────┘           └──────────────────────────────┘

base64 ≠ encrypted

If you remember one thing: base64 -d undoes a Secret with no key. “It’s in a Secret” is not the same as “it’s protected.” Encryption and access control are separate jobs you must turn on.

What to actually do about it

Four layers, increasingly serious:

1. Encryption at rest for etcd. Configure the API server’s EncryptionConfiguration so Secrets are encrypted before they hit etcd. Now an etcd backup isn’t plaintext. This is table stakes.
2. RBAC. Lock down who can get Secrets — most workloads and humans never should. See Identity, RBAC & Network Policy.
3. An external secrets manager. Keep the source of truth in Vault, AWS Secrets Manager, or similar, and sync into the cluster (e.g. the External Secrets Operator). The cluster holds short-lived copies, not the master record.
4. Keep secrets out of Git. GitOps stores manifests in Git — and a plaintext Secret committed to a repo is a leak forever. Use Sealed Secrets or SOPS to encrypt before commit. The whole topic gets its own treatment in Secrets Management.

Why this makes production safer

Back to the thread: what manual, error-prone step does this remove? Without ConfigMaps and Secrets, per-environment settings get baked into images (forcing rebuilds) or hand-edited on servers (drift and typos), and credentials end up pasted into scripts and chat. ConfigMaps make configuration a reviewed, versioned object that travels separately from the immutable image. Secrets do the same for sensitive values — provided you remember base64 is not encryption and add real protection on top. The win is that “change a setting” or “rotate a password” becomes a declarative change to data, not a risky manual edit on a live box. Next, the harder problem: data that must survive a Pod’s death — storage and stateful workloads.

The architect’s lens

Step back from the YAML and judge ConfigMaps and Secrets as the decoupling tools they are:

• Why do they exist? Because build once, run everywhere breaks if per-environment values (DB URL, log level, API key) are baked into the image — you’d need a different image per environment. They keep code in the image and settings in the cluster as data.
• What problem do they solve? They turn “change a setting” or “rotate a password” from an SSH-and-edit on a live box into a reviewed, versioned object — injected by env var (read once at start) or by mounted volume (updated live within a sync period), the identical image promoted across environments.
• What are the trade-offs? A Secret is base64, not encrypted — base64 -d recovers it with no key, and by default it sits in etcd that way, so an etcd backup is a backup of your plaintext. The object signals “treat carefully” but doesn’t itself protect.
• When should I avoid relying on them as-is? Whenever the value is truly sensitive and you haven’t layered real protection — encryption at rest, RBAC on get secret, or an external manager (Vault, the External Secrets Operator) — a raw Secret in Git is a leak forever (use Sealed Secrets or SOPS).
• What breaks if I remove them? Per-environment settings collapse back into the image (forcing rebuilds) or hand-edited server files (drift and typos), and credentials end up pasted into scripts and chat — the exact toil storage-and-stateful and the rest of the Part are built to retire.

Check your understanding

1. Why does baking environment-specific config into an image defeat the “build once, run anywhere” principle? What do ConfigMaps restore?
2. Compare injecting config as environment variables vs as a mounted volume. When does each pick up a change to the underlying ConfigMap?
3. A teammate says “the password is safe, it’s in a Kubernetes Secret.” What’s wrong with that reasoning? Show the command that disproves it.
4. Where are Secrets stored by default, and why does that make an etcd backup a security concern? What’s the first mitigation?
5. Using the book’s thread, explain how ConfigMaps and Secrets remove manual, error-prone configuration steps — and the one caveat you must add for the safety to be real.

Show answers

1. Because environment-specific values baked in force a different image per environment, so a setting change means a rebuild — defeating “build once, run everywhere.” ConfigMaps restore it by carrying the settings outside the image as data, so the identical artifact runs in staging and production and you just swap which ConfigMap it reads.
2. Env vars are read once at process start — change the ConfigMap and a running Pod keeps the old value until it restarts. Mounted volumes appear as files that Kubernetes updates live (within a sync period), so an app that re-reads its config files can pick up changes without a restart.
3. A Secret is base64-encoded, not encrypted — base64 is reversible with no key, so anyone who can read the object reads the value. Disprove it with: kubectl get secret db-credentials -o jsonpath='{.data.DB_PASSWORD}' | base64 -d.
4. Secrets are stored in etcd, base64-encoded by default — so an etcd backup (or read access to it) is a backup of your plaintext secrets. The first mitigation is encryption at rest for etcd (configure the API server’s EncryptionConfiguration) so Secrets are encrypted before they hit the store.
5. They turn per-environment settings and credentials from baked-into-images or hand-edited-on-servers (rebuilds, drift, typos, passwords pasted into scripts and chat) into reviewed, versioned objects that travel separately from the immutable image — so “change a setting” or “rotate a password” is a declarative data change. The caveat: base64 is not encryption, so you must add real protection (encryption at rest, RBAC, an external manager) for the safety to be real.