Resource Management & Node Pressure

Scaling & Scheduling framed requests and limits as a contract: the request is what the scheduler reserves, the limit is the ceiling the kernel enforces. That’s the what. This page is the how — the precise mechanism by which two numbers in your YAML become Linux cgroup settings, how those settings classify your Pod into a quality-of-service tier, and what the kubelet and the kernel do when a node runs out of room. Get this wrong and you get the two most-misdiagnosed failures in Kubernetes: a CPU-starved app that’s “fine” on every dashboard, and OOMKilled Pods that restart for no obvious reason.

The thread runs straight through here. The manual step being removed is an ops engineer babysitting which process gets the CPU and which gets killed when memory runs low — a job humans do badly and inconsistently. Requests, limits, and QoS make that policy declared data the kernel and kubelet enforce identically every time.

From two numbers to cgroup knobs

A container is just a process in a cgroup. Your requests and limits are translated by the kubelet into specific cgroup values when it creates the container. The two resources map to different knobs because CPU is compressible and memory is not.

Spec field	cgroup v2 knob	Effect
cpu request	cpu.weight (proportional share)	relative priority for spare CPU under contention
cpu limit	cpu.max (quota / period)	hard cap — throttled when exceeded
memory request	(no hard knob)	scheduling + OOM-ranking input only
memory limit	memory.max	hard cap — OOM-killed when exceeded

CPU request → share. A request of 250m (a quarter core) becomes a proportional weight. The classic cgroup v1 name was cpu.shares, where 1 core = 1024 shares, so 250m → 256 shares; cgroup v2 expresses the same idea as cpu.weight. The crucial property: shares only matter under contention. If the node is idle, a Pod requesting 250m can use a whole core. The moment two Pods compete, the kernel’s scheduler hands out CPU in proportion to their weights — a 500m Pod beats a 250m Pod 2:1 for the contested cycles. Requests don’t cap CPU; they rank it.

CPU limit → quota. A limit of 500m becomes a CFS quota: “this cgroup may run for 50ms of CPU per 100ms period.” Exhaust the quota and the process is throttled — descheduled until the next period — which shows up as latency, not as a crash. This is why an over-tight CPU limit produces a service that is slow and jittery while its CPU graph sits below the limit on average: the throttling happens in sub-second bursts the averaged metric hides.

   cpu: limit 500m  →  cpu.max = "50000 100000"   (50ms per 100ms)
   ┌──100ms period──┬──100ms period──┬──100ms period──┐
   │ run 50ms |XXXX| │ run 50ms |XXXX| │ run 50ms |XXXX| │   XXXX = throttled, waiting
   └────────────────┴────────────────┴────────────────┘

Memory has no “share.” You cannot give a process “a little less RAM” — memory is incompressible. So the memory request sets no runtime knob; it’s used only for scheduling and for ranking who dies first. The memory limit becomes memory.max: cross it and the kernel’s OOM killer fires inside that cgroup and kills the offending process. That’s the OOMKilled you see in kubectl describe.

Throttling is invisible; OOMKilled is loud

The two failure modes are diagnosed completely differently. A too-low CPU limit silently throttles — no event, no restart, just tail-latency. Check container_cpu_cfs_throttled_periods_total in Metrics, not the average CPU graph. A too-low memory limit is loud — the container restarts with OOMKilled and exit code 137. If you see 137, raise the memory limit (or fix the leak); if a service is “slow but the CPU looks fine,” suspect CPU throttling.

QoS classes: how the request/limit shape decides priority

The kubelet doesn’t store an explicit “priority” per Pod; it derives a QoS class from the shape of your requests and limits. The class then governs what happens under node pressure.

QoS class	Condition	Under pressure
Guaranteed	every container sets request == limit, for both CPU and memory	evicted/killed last
Burstable	at least one request set, but not Guaranteed	evicted by how far over request it is
BestEffort	no requests or limits set at all	evicted first

The lesson is uncomfortable for the common habit of leaving requests and limits unset, or setting requests far below real usage: a BestEffort Pod is first against the wall when the node is squeezed, and a Burstable Pod that’s running far above its request is next. Guaranteed — request equal to limit on both resources — is the only class the kubelet protects, and it’s what you give anything whose death would page someone.

   eviction order under memory pressure:
   BestEffort ──► Burstable (most over request first) ──► Guaranteed (last)
       ↑ no requests          ↑ requests < usage              ↑ request == limit

The OOM killer and oom_score_adj

QoS isn’t only a kubelet concept; the kubelet pushes it down into the kernel. When the node runs out of memory faster than the kubelet’s eviction loop can react (a sudden allocation), the Linux kernel OOM killer steps in directly. It picks a victim by oom_score — and the kubelet biases that score per container by writing oom_score_adj:

• Guaranteed pods get a very negative oom_score_adj (-997) — the kernel avoids them.
• BestEffort pods get 1000 — maximally likely to be picked.
• Burstable pods get a value scaled by their memory request — a Pod requesting more is protected more.

So the QoS class you derived from your YAML literally becomes a kernel kill-ranking. Two failure paths exist and they’re worth distinguishing: a container exceeding its own memory.max is OOM-killed by its cgroup (just that container restarts); a node running out of memory globally triggers either kubelet eviction (graceful, by QoS) or the kernel OOM killer (abrupt, by oom_score_adj).

Eviction is graceful; the kernel OOM killer is not

The kubelet’s node-pressure eviction is proactive and orderly — it watches signals, picks victims by QoS, and terminates them with grace periods and events. The kernel OOM killer is reactive and brutal — it fires when allocation outruns the kubelet, killing a process instantly with no Pod-level grace. A well-tuned node leans on the former (reserve resources so the kubelet acts first) and treats the latter as the safety net of last resort.

Node-pressure eviction and its signals

The kubelet continuously watches a set of eviction signals against configured thresholds. When a signal crosses, the kubelet reclaims by evicting Pods (ranking by QoS and usage-over-request). The signals are real resources, not abstractions:

Signal	Meaning
memory.available	free memory on the node
nodefs.available	free space on the node’s root/kubelet filesystem
imagefs.available	free space where images & writable layers live
nodefs.inodesFree	free inodes (you can run out of these before bytes)
pid.available	free process IDs

Thresholds come in two flavors: soft (cross it and the kubelet waits a grace period, then evicts — giving transient spikes a chance to recover) and hard (cross it and the kubelet evicts immediately, no grace). When the kubelet evicts for disk pressure it first tries cheaper reclamation — deleting dead Pods and unused images — before terminating running Pods. A node under sustained pressure also gets a taint (e.g. node.kubernetes.io/memory-pressure) so the scheduler stops sending it new work.

   kubelet watches signals ──► memory.available < threshold?
                                      │ soft → wait grace period, recheck
                                      │ hard → evict now
                                      ▼
                        rank Pods (BestEffort → Burstable-over-request → Guaranteed)
                                      ▼
                        evict lowest-priority Pod, reclaim, re-check signal

PodDisruptionBudgets: protecting availability during voluntary disruption

Eviction and the OOM killer are involuntary disruptions — the system reacting to pressure. The other kind is voluntary: a node drain for an upgrade, a Cluster Autoscaler scale-down, an admin running kubectl drain. A PodDisruptionBudget (PDB) is your guardrail against voluntary disruptions taking out too many replicas at once.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: web }
spec:
  minAvailable: 2            # never let voluntary eviction drop web below 2 Pods
  selector: { matchLabels: { app: web } }

A drain or autoscaler scale-down calls the Eviction API, which respects PDBs: if evicting a Pod would drop the matching set below minAvailable (or above maxUnavailable), the eviction is denied and the drainer waits. This is exactly what makes the Cluster Autoscaler’s scale-down safe, and what keeps a rolling node upgrade (Cluster Lifecycle & Operations) from briefly zeroing your service.

A PDB can block a drain forever

If minAvailable equals the replica count, no Pod can ever be voluntarily evicted — a node drain will hang indefinitely, because removing even one Pod violates the budget. A PDB protects availability and can deadlock maintenance. The fix is a budget with genuine slack (minAvailable: 2 for a 3-replica service), so the cluster can always make progress while keeping a quorum up.

LimitRange and ResourceQuota: governing a namespace

Everything above is per-Pod. Two namespace-scoped objects govern the aggregate and set sane defaults so individual teams can’t accidentally — or deliberately — consume the whole cluster:

• LimitRange — per-Pod/per-container defaults and bounds within a namespace. It can inject default requests/limits onto Pods that declare none (turning would-be BestEffort Pods into Burstable), and reject Pods whose requests fall outside [min, max]. This is how you stop the “no requests set” problem from the source.
• ResourceQuota — a namespace-wide cap on totals: sum of all CPU/memory requests and limits, and counts of objects (Pods, Services, PVCs). Once the namespace hits its quota, new Pods are rejected at admission until something is freed.

apiVersion: v1
kind: ResourceQuota
metadata: { name: team-a }
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    pods: "100"

ResourceQuota forces honest requests

A subtle, powerful interaction: once a namespace has a requests.cpu/requests.memory quota, the API server refuses any Pod that doesn’t declare those requests — it has no number to count against the quota. So a single quota object makes “no requests set” impossible in that namespace, which in turn makes the scheduler’s bin-packing and the HPA’s math reliable. One governance object fixes the root cause of a whole class of resource bugs.

The thread, made concrete: requests, limits, QoS, eviction, and quotas replace a human deciding, ad hoc and inconsistently, who gets CPU and who dies when memory is tight. The kernel and kubelet now enforce that policy identically every time, from a declaration in YAML. What it costs you is the discipline to set honest numbers — because the same machinery that protects a Guaranteed Pod will, just as reliably, kill a BestEffort one first.

With per-node resource behavior understood, we zoom out to the cluster as a whole: how it’s bootstrapped, made highly available, and upgraded without downtime — Cluster Lifecycle & Operations.

By the numbers — the exact knob behind “scaled by their memory request”

“Burstable pods get a value scaled by their request” is a precise formula, not a hand-wave. The kubelet computes each Burstable container’s oom_score_adj as:

   oom_score_adj = 1000 − (1000 × memoryRequest / nodeAllocatableMemory)   clamped to [2, 999]
   request 10% of node memory → 1000 − 100 = 900   (killed readily)
   request 50% of node memory → 1000 − 500 = 500
   request 90% of node memory → 1000 − 900 = 100   (well protected)

BestEffort sits at the maximum (1000, first to die) and Guaranteed at the protective floor the OOM section above quoted. So the more memory a Burstable Pod honestly reserves, the lower its kill-ranking — the kernel rewards an accurate request with survival, the same incentive the scheduler and HPA rely on, pushed all the way down into the OOM killer.

The architect’s lens

Step back from the cgroup knobs and answer the five questions behind requests, limits, and QoS:

• Why does it exist? Because the manual step is an ops engineer babysitting which process gets the CPU and which gets killed when memory runs low — a job humans do badly and inconsistently. Requests, limits, and QoS make that policy declared data the kernel and kubelet enforce identically every time.
• What problem does it solve? It translates two YAML numbers into cgroup settings — CPU request → cpu.weight (ranks contested cycles), CPU limit → cpu.max quota (throttles), memory limit → memory.max (OOM-kills) — and derives a QoS class (Guaranteed/Burstable/BestEffort) that orders eviction and biases the kernel’s oom_score_adj.
• What are the trade-offs? The two failure modes diagnose oppositely: a tight CPU limit throttles silently (check container_cpu_cfs_throttled_periods_total, not the average), while a tight memory limit is loud (OOMKilled, exit 137); and a PDB with minAvailable equal to the replica count protects availability but can deadlock a node drain forever.
• When should I avoid the easy path? Don’t leave requests blank — a BestEffort Pod is evicted first and breaks the scheduler’s bin-packing and the HPA’s math; CPU and memory are not symmetric, so don’t reach for a memory “share” that doesn’t exist; and a ResourceQuota on requests.* is the cheapest way to force honest requests across a namespace.
• What breaks if I remove it? Resource arbitration reverts to a human picking victims ad hoc; without requests the scheduler can’t reserve and the HPA can’t divide; without QoS and oom_score_adj the kernel kills arbitrarily under pressure; and without a PDB a node drain can zero a service mid-upgrade.

Check your understanding

1. A service has cpu: limit 200m and is reported as slow, but its average CPU graph sits well below 200m. What’s almost certainly happening, and which metric — not the average — confirms it?
2. Explain why a memory request sets no runtime cgroup knob while a CPU request does. Tie it to “compressible vs incompressible.”
3. You leave both requests and limits blank on every container. What QoS class do your Pods land in, and why is that dangerous under node pressure?
4. Distinguish kubelet node-pressure eviction from the kernel OOM killer. When does each fire, and which one respects PodDisruptionBudgets?
5. A team keeps shipping Pods with no resource requests, breaking scheduling and the HPA. Which single namespace object can make that impossible, and by what mechanism?

Show answers

1. The container is being CPU-throttled: the 200m limit is a CFS quota (20ms per 100ms period), and it’s hitting the cap in sub-second bursts that the averaged graph smooths away. Confirm with container_cpu_cfs_throttled_periods_total (throttling counters), not the average CPU utilization.
2. CPU is compressible — the kernel can hand out fewer cycles, so a request becomes a proportional weight (cpu.weight/cpu.shares) that ranks who gets contested CPU. Memory is incompressible — there’s no “a little less RAM,” so a memory request can’t be a runtime knob; it’s used only for scheduling and to bias the OOM ranking. The limit (memory.max) is the only hard memory knob.
3. BestEffort — a Pod with no requests and no limits is BestEffort, which is evicted first under pressure and carries the kernel’s maximum oom_score_adj (1000): the cheapest Pod to kill, and one the scheduler can’t reserve for. (Setting only limits is different: Kubernetes copies each limit as the request, so a limits-only Pod actually lands in Guaranteed.)
4. Kubelet eviction is proactive and orderly — it watches signals (memory.available, nodefs, etc.), waits grace periods on soft thresholds, ranks victims by QoS, and respects PDBs via the Eviction API. The kernel OOM killer is reactive and abrupt — it fires when allocation outruns the kubelet, killing a process instantly by oom_score_adj, ignoring PDBs and grace periods.
5. A ResourceQuota that sets requests.cpu/requests.memory. Once it exists, the API server has to count every Pod’s requests against the quota, so it rejects at admission any Pod that omits them — making “no requests set” impossible in that namespace and restoring the scheduler/HPA math. (A LimitRange with defaults achieves a softer version by injecting requests.)