Security in Depth

Identity, RBAC & Network Policy made the case for least privilege and showed the objects: Roles, bindings, NetworkPolicies. Config & Secrets warned that a Secret is base64, not encryption. This page goes underneath both — to the mechanisms that make those promises real. How does the apiserver actually verify who a request comes from? Where does a Pod’s identity physically come from, and why is the old token a liability? What kernel features turn “runs as non-root” from a label into an enforced confinement? And how do you finally encrypt etcd?

Security isn’t one control; it’s a stack of them, each catching what the last missed. The useful frame is the 4 C’s.

The 4 C’s: defense in depth as nested boxes

   ┌──────────────────────────────────────────────────────────┐
   │ CLOUD   — the account, VPC, IAM, the physical/datacenter   │
   │  ┌─────────────────────────────────────────────────────┐  │
   │  │ CLUSTER  — apiserver authn/authz, RBAC, etcd, PSA     │  │
   │  │  ┌────────────────────────────────────────────────┐  │  │
   │  │  │ CONTAINER — image provenance, no root, dropped  │  │  │
   │  │  │   capabilities, seccomp/AppArmor confinement     │  │  │
   │  │  │  ┌──────────────────────────────────────────┐   │  │  │
   │  │  │  │ CODE — your app: input validation, deps,  │   │  │  │
   │  │  │  │   secrets handling, no SQL injection       │   │  │  │
   │  │  │  └──────────────────────────────────────────┘   │  │  │
   │  │  └────────────────────────────────────────────────┘  │  │
   │  └─────────────────────────────────────────────────────┘  │
   └──────────────────────────────────────────────────────────┘
   A weak outer layer can't be saved by a strong inner one — and vice versa.

Each layer is a different team’s problem on a different timescale, and a breach usually walks inward: a leaked cloud credential, then the cluster, then a container, then your data. This page lives mostly in the cluster and container layers; the code layer is shift-left’s job and the cloud layer is Part 9’s.

Authentication: how the apiserver knows who you are

The apiserver authenticates every request, but — surprisingly — Kubernetes has no user database. There is no User object you can kubectl create. Instead the apiserver runs a chain of authenticator plugins; the first one that recognizes a credential wins and produces a username + groups. The common methods:

• Client certificates (mTLS). You present a TLS client cert signed by the cluster’s CA. The apiserver reads the certificate’s Common Name (CN) as the username and Organization (O) fields as groups. This is how kubectl admin access and the control-plane components themselves authenticate.
• Bearer tokens. A Authorization: Bearer <token> header. ServiceAccount tokens (below) are the main case; static token files exist but are discouraged.
• OIDC. The apiserver is configured to trust an external identity provider (Okta, Google, Entra). The user logs in there, gets a signed JWT, and the apiserver verifies the issuer’s signature and reads the username/groups from claims. This is how you give humans SSO without minting a cert per person.

   request ──► authenticator chain ──► (username, groups) ──► [ AUTHZ / RBAC ]
              ├ client cert?  CN=alice, O=devs
              ├ bearer token? system:serviceaccount:prod:payments
              └ OIDC JWT?     alice@corp.com, groups=[platform]

The username is just a string — RBAC binds to it

Authentication’s only output is a username and a set of groups, both opaque strings. RBAC bindings then grant permissions to those exact strings — subjects: [{kind: User, name: "alice@corp.com"}] or a group. This is why authn and authz stay cleanly separated: authn decides the string, authz decides what the string may do. There is no “admin” flag on an identity; admin is just a binding to cluster-admin.

Authorization: RBAC and deny-by-default

Once authenticated, the request hits the authorizer chain. RBAC is the dominant authorizer, and its single most important property is that it is deny-by-default and additive-only: nothing is permitted unless a RoleBinding or ClusterRoleBinding explicitly grants it, and there are no deny rules. You cannot write “allow everything except deleting Secrets” in RBAC — you compose the allow set up from zero. (The Identity & RBAC page covered the four objects; here’s the mechanism underneath them.)

   Can serviceaccount:prod:payments delete pods in prod?
   ──► scan every (Cluster)RoleBinding that names this subject
       ──► union of all their rules: do ANY grant verb=delete, resource=pods, ns=prod?
           ──► yes → ALLOW    no → DENY (the default)

Because it’s additive, RBAC is safe to reason about: granting a binding can only ever add power, never silently remove it elsewhere, so there’s no interaction between rules to puzzle over. The flip side — no deny rules — means you express “everything except X” by simply never granting X, which is why scoping Roles narrowly matters so much. Audit what you’ve granted with:

kubectl auth can-i delete pods --as=system:serviceaccount:prod:payments -n prod
kubectl auth can-i --list --as=system:serviceaccount:prod:payments -n prod

ServiceAccounts and the death of the forever-token

Every Pod runs as a ServiceAccount — its identity to the apiserver. The question is how that identity is delivered to the Pod, and the answer changed in an important way.

The old way: when you created a ServiceAccount, a controller minted a Secret holding a JWT that never expired, and mounted it into the Pod. A token that’s valid forever and sits on disk in every Pod is a dream credential to steal — exfiltrate it once and it works indefinitely, for any caller, anywhere.

The modern way is the TokenRequest API with bound, projected tokens. The kubelet requests a token for that specific Pod from the apiserver; the token is time-limited (auto-rotated, often hourly), audience-scoped (only valid for the audience it was issued for), and bound to the Pod’s lifetime (it becomes invalid the moment the Pod is deleted). It’s delivered via a projected volume, not a static Secret:

spec:
  serviceAccountName: payments
  containers:
    - name: app
      image: payments:2.1
      volumeMounts:
        - name: token
          mountPath: /var/run/secrets/tokens
  volumes:
    - name: token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              audience: vault              # this token is only valid for Vault
              expirationSeconds: 3600       # the kubelet rotates it before expiry

   OLD: ServiceAccount Secret = JWT that never expires, mounted forever
        steal it once → valid forever, any audience               ✗
   NEW: TokenRequest → bound token: short-lived + audience-scoped + tied to the Pod
        steal it → expires in an hour, useless outside its audience, dead when Pod dies  ✓

This is the same “short-lived and verifiable beats long-lived and stealable” principle the security Part preached, now made concrete in the token machinery. It’s also what makes workload identity federation possible: an external system (a cloud IAM, Vault) trusts the apiserver’s OIDC signature on the bound token and grants access to that identity — no static cloud credential stored in the Pod at all.

Turn the token off when the Pod never calls the API

Most application Pods never talk to the Kubernetes API — yet they get a token mounted anyway, a credential with no purpose but plenty of risk. Set automountServiceAccountToken: false on the Pod (or the ServiceAccount). One less stealable credential per Pod, for free.

Confining the container: securityContext and the kernel

RBAC and tokens govern the API. But a compromised process inside a container attacks the node — and the defenses there are Linux kernel features the namespaces & cgroups page introduced. The Pod’s securityContext is how you switch them on declaratively.

spec:
  securityContext:
    runAsNonRoot: true            # refuse to start if the image's user is root (UID 0)
    runAsUser: 10001
    fsGroup: 10001                # group that owns mounted volumes
    seccompProfile:
      type: RuntimeDefault        # block the rarely-needed, dangerous syscalls
  containers:
    - name: app
      image: app:1.0
      securityContext:
        allowPrivilegeEscalation: false   # setuid binaries can't gain more than the process has
        readOnlyRootFilesystem: true      # the image's filesystem is immutable at runtime
        capabilities:
          drop: ["ALL"]                    # start from zero Linux capabilities
          add: ["NET_BIND_SERVICE"]        # add back only the one you truly need

Each setting closes a specific door:

• runAsNonRoot / runAsUser — root inside a container is, by default, mapped to real root on the host kernel. A container escape from root is a host compromise; from an unprivileged UID it’s far less. runAsNonRoot: true makes the kubelet refuse to start a Pod whose image would run as UID 0.
• Dropped capabilities — Linux splits root’s power into ~40 capabilities (NET_ADMIN, SYS_ADMIN, CHOWN…). Containers get a subset by default; dropping ALL and adding back only what’s needed means a compromised process can’t, say, load kernel modules or rewrite firewall rules.
• allowPrivilegeEscalation: false — blocks the setuid trick where a process gains more privilege than its parent. readOnlyRootFilesystem: true stops an attacker writing a payload to the image’s disk.
• seccomp — a seccomp profile filters which syscalls the kernel will even accept from the container. RuntimeDefault blocks dozens of dangerous-and-rarely-used syscalls; a custom profile can go further. AppArmor (or SELinux on some distros) is the complementary lever — a mandatory-access-control profile restricting which files/capabilities the process can touch, regardless of file permissions.

   Container escape attempt, layered defenses:
   process runs as UID 10001 ─► not host root, limited damage
        │ tries privilege escalation ─► blocked (allowPrivilegeEscalation:false)
        │ tries a dangerous syscall  ─► blocked (seccomp RuntimeDefault)
        │ tries to write a payload   ─► blocked (readOnlyRootFilesystem)
        │ tries an admin capability  ─► blocked (capabilities drop ALL)
   Each layer alone is bypassable; together they make escape expensive.

This is exactly the restricted Pod Security Standard profile from Admission Control & Policy — and PSA exists precisely to enforce that these fields are set, since an unenforced securityContext is just a suggestion an overworked developer will skip.

privileged: true discards all of it

A container with securityContext.privileged: true gets nearly all capabilities, full device access, and sees host namespaces — it’s effectively root on the node. It’s occasionally necessary (some CNI/CSI agents, node-level tooling) but is the single most dangerous field in a Pod spec. Treat any privileged: true in an application workload as a finding, and let PSA’s baseline profile reject it for you.

Encrypting Secrets at rest: closing the etcd hole

The Config & Secrets page left a hole open: Secrets land in etcd base64-encoded, so an etcd backup is a plaintext credential dump. The fix is encryption at rest, configured via an EncryptionConfiguration the apiserver reads. It encrypts Secret values before writing them to etcd:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets"]
    providers:
      - kms:                              # best: a KMS provider holds the key, not the cluster
          apiVersion: v2
          name: cloud-kms
          endpoint: unix:///var/run/kmsplugin/socket.sock
      - identity: {}                      # fallback for reads of not-yet-migrated objects

The provider choice is the whole point:

Provider	Where the key lives	Trade-off
identity	nowhere — no encryption	the default; base64 only
aescbc / aesgcm	in the config file on the control-plane node	better than nothing, but a host compromise leaks the key too
kms (v2)	an external KMS (cloud KMS, HSM, Vault)	the cluster never holds the root key; envelope encryption

A KMS provider uses envelope encryption: the KMS holds a root key that never leaves it, the apiserver uses it to encrypt a per-object data key, and only the encrypted data key travels to etcd. An attacker with the etcd backup and the config file still can’t decrypt without calling the KMS — which you can revoke. This is the difference between “we encrypted etcd” and “we encrypted etcd with a key the same attacker can read.”

Encryption is lazy — you must rewrite existing Secrets

Turning on encryption only affects writes. Secrets already in etcd stay in their old form until rewritten. After enabling a provider, force a rewrite of every Secret so none are left in the clear: kubectl get secrets -A -o json | kubectl replace -f -. Verify by reading the raw key straight from etcd and confirming it’s no longer base64-readable plaintext.

It actually happened — the Tesla cryptojacking breach (2018)

The 4 C’s “a breach walks inward” is not hypothetical. In early 2018, researchers (RedLock) found that Tesla had a Kubernetes dashboard exposed to the internet with no authentication. That single open door at the cluster layer was enough: attackers launched cryptomining Pods and — walking inward — discovered AWS credentials stored in the cluster, reaching the cloud layer. They were deliberately quiet about it, mining against a private pool behind a CDN with CPU throttled to evade the usual detection. The lesson maps straight onto this page: a weak outer layer (an unauthenticated console) is not saved by anything inside it, and a long-lived credential sitting one layer in is exactly what turns a foothold into a full compromise. It is the case for bound, short-lived tokens and least privilege, told as an incident instead of a principle.

The thread: what manual step disappears

The recurring question: what manual, error-prone step does this remove — and how does it make production safer? The manual step is trusting the inside: hand-distributed long-lived credentials, containers run as root because “it’s just internal,” and a hope that nobody reads the etcd backup. Each mechanism here replaces a hope with an enforced property — bound tokens that expire and can’t be replayed instead of a forever-token copied into a wiki; a securityContext the kubelet refuses to ignore instead of “we usually don’t run as root”; KMS-backed encryption so a stolen backup is ciphertext instead of a password list. None of it prevents a breach outright; defense in depth assumes a breach and ensures each layer the attacker clears costs them more and yields less. The cost is operational weight — managing certs and OIDC, tuning seccomp profiles, running a KMS — which is real, and exactly why the platform automates so much of it (auto-rotated tokens, RuntimeDefault profiles, default-deny PSA). Next, we descend to where the data actually lives: Storage Internals (CSI).

The architect’s lens

Step back from the individual controls and answer the five questions behind defense in depth:

• Why does it exist? Because trusting the inside — hand-distributed forever-tokens, containers run as root “because it’s internal,” and hoping nobody reads the etcd backup — is exactly how a foothold becomes a full compromise. The Tesla cryptojacking breach walked cloud → cluster → container → data; each layer assumes the last failed.
• What problem does it solve? It replaces hopes with enforced properties: an authenticator chain yields a username/groups string that deny-by-default RBAC binds to; bound, projected TokenRequest tokens are short-lived, audience-scoped, and die with the Pod; a securityContext the kubelet refuses to ignore confines the container; and KMS envelope encryption keeps the etcd root key off the cluster.
• What are the trade-offs? It’s real operational weight — certs, OIDC, seccomp tuning, running a KMS — which is why the platform automates much of it (auto-rotated tokens, RuntimeDefault, default-deny PSA); and no single control prevents a breach, the layers only make each step cost more and yield less.
• When should I avoid the lax path? Set automountServiceAccountToken: false on any Pod that never calls the API; treat privileged: true in an application workload as a finding; and don’t claim “we encrypt etcd” with aescbc (key on the host) — only a KMS provider resists a host compromise.
• What breaks if I remove it? Authn/authz collapse to implicit trust; a stolen legacy token works forever for any audience; a container escape from root becomes a host compromise; and an etcd backup becomes a plaintext credential dump (a Secret is base64, not encryption).

Check your understanding

1. Kubernetes has no User object, yet the apiserver authenticates every request. How does it know who you are, and what exactly is the output of authentication that RBAC then uses?
2. RBAC is “deny-by-default and additive-only, with no deny rules.” What does that mean in practice, and how do you express “allow everything except deleting Secrets”?
3. Contrast a legacy ServiceAccount Secret token with a TokenRequest bound/projected token across three properties. Why is the modern token far harder to abuse if stolen?
4. A container is compromised and the process tries to escape to the node. Walk through how runAsNonRoot, dropped capabilities, allowPrivilegeEscalation: false, and seccomp each independently raise the cost.
5. Your CISO says “we encrypt etcd at rest.” Using the provider table and envelope encryption, explain why that claim is meaningful only with a KMS provider — and the one operational step people forget after enabling it.

Show answers

1. The apiserver runs a chain of authenticator plugins (client certs, bearer/ServiceAccount tokens, OIDC JWTs); the first to recognize a credential wins. There’s no user database — identity comes from the credential itself (e.g. a cert’s CN/O fields, or OIDC claims). The output is just a username + a set of groups, as opaque strings, which RBAC bindings then grant permissions to. Admin is not a flag; it’s a binding to cluster-admin.
2. Deny-by-default: nothing is allowed unless a (Cluster)RoleBinding explicitly grants it. Additive /no-deny: you can only grant permissions, never write a rule that subtracts one, so granting a binding can only add power and rules never interact. You express “everything except deleting Secrets” by never granting that verb/resource in the first place — there’s no deny rule, so scoping Roles narrowly is how you exclude things.
3. Legacy token: never expires, not audience-scoped, lives in a static Secret independent of any Pod. Bound token: time-limited (auto-rotated, ~hourly), audience-scoped (only valid for its declared audience), and bound to the Pod’s lifetime (dead when the Pod is deleted). Stolen, the legacy token works forever for anyone; the bound token expires in an hour, is useless outside its audience, and dies with the Pod.
4. runAsNonRoot/runAsUser means the process isn’t host root, so an escape yields an unprivileged user, not the box. Dropping capabilities removes the specific powers (load modules, edit firewall) needed for many escalations. allowPrivilegeEscalation: false blocks the setuid path to gaining more than the parent had. seccomp RuntimeDefault rejects the dangerous syscalls an exploit typically needs. Each is individually bypassable; together they make escape expensive and unlikely.
5. With identity there’s no encryption; with aescbc/aesgcm the key sits on the control-plane node, so an attacker who takes the host (and thus the etcd backup) also gets the key — encryption that doesn’t resist the realistic threat. A KMS provider uses envelope encryption: the root key never leaves the external KMS, only encrypted data keys reach etcd, so a stolen backup + config is still undecryptable (and the KMS key is revocable). The forgotten step: rewrite all existing Secrets (e.g. kubectl get secrets -A -o json | kubectl replace -f -), since encryption only applies to new writes.