Scaling & Scheduling

You’ve declared what should run. This page is about where it runs and how many of it run — the two decisions that used to be a human’s job. Capacity planning by hand meant an ops engineer staring at a spreadsheet of CPU and RAM, picking a server with room, SSHing in to start the process, and at 2 a.m. during a traffic spike, manually spinning up more. Every step was slow, error-prone, and reactive. Kubernetes turns both decisions into control loops — but only if you give it the one piece of information it needs: how much each Pod actually wants.

Requests and limits: the contract that makes scheduling possible

The scheduler cannot place a Pod sensibly if it doesn’t know the Pod’s appetite. So every container should declare two numbers per resource:

• request — the amount the Pod is guaranteed. The scheduler reserves this; it’s the basis for placement.
• limit — the ceiling the Pod may use. Exceed it and the kernel intervenes; it’s the basis for isolation.

spec:
  containers:
    - name: app
      image: myapp:1.4.2
      resources:
        requests:                 # scheduler reserves this much
          cpu: "250m"             # 0.25 of a core
          memory: "256Mi"
        limits:                   # hard ceiling at runtime
          cpu: "500m"
          memory: "512Mi"

The two behave differently, and the difference matters:

   CPU            compressible  → over the limit = THROTTLED (slowed, not killed)
   Memory     incompressible  → over the limit = OOM-KILLED (the kernel kills it)

This is the cgroups contract from Part 3, surfaced at the cluster level: limits are the kernel-enforced ceiling, now scheduled across many machines. CPU is compressible — the kernel can hand out less and the process just runs slower. Memory is incompressible — there’s no “a little less RAM,” so crossing the limit means the OOM killer. That’s why an under-set memory limit shows up as mysterious OOMKilled restarts.

No requests, no good decisions

A Pod with no resource requests is invisible to the scheduler’s math — it gets packed anywhere and can starve neighbors. Requests that are too high waste money on reserved-but-idle capacity. Getting these numbers right is one of the highest-leverage things you do on a cluster.

The scheduler: a filter-then-score loop

When a Pod is created it has no node yet. The scheduler (introduced in Kubernetes Architecture) watches for unscheduled Pods and runs a two-phase decision — itself a reconciliation loop closing the gap between “Pod exists” and “Pod is running somewhere”:

   All nodes ──► [ FILTER ]  drop nodes that can't fit:
                              • not enough free CPU/memory (vs requests)
                              • taints not tolerated, selectors unmet
              ──► [ SCORE ]   rank the survivors:
                              • spread across nodes, prefer least-loaded
              ──► bind Pod to the highest-scoring node

Filter answers can it run here?; score answers where is best? Then the kubelet on the chosen node pulls the image and starts the container. No human picked the box.

Steering placement: selectors, affinity, taints, tolerations

The default scoring is sensible, but sometimes you need to influence where Pods land. Kubernetes gives you several knobs, from blunt to expressive:

• nodeSelector — the blunt one: “only nodes with this label.” disktype: ssd, gpu: "true".
• Affinity / anti-affinity — richer rules. Node affinity: prefer/require nodes by label. Pod anti-affinity: “don’t put two replicas of me on the same node” — so one node failure doesn’t take out every copy. This is how you spread for availability.
• Taints and tolerations — the inverse logic, and the one that confuses people. A taint on a node repels Pods: “stay off unless you explicitly tolerate me.” A toleration on a Pod is its permission slip. Taints are how you reserve nodes (GPU nodes, dedicated workloads) and how Kubernetes keeps ordinary Pods off the control plane.

# Reserve a node: nothing schedules here without the matching toleration
kubectl taint nodes gpu-node-1 dedicated=gpu:NoSchedule

spec:
  tolerations:                    # this Pod is allowed onto the tainted node
    - key: "dedicated"
      operator: "Equal"
      value: "gpu"
      effect: "NoSchedule"
  affinity:
    podAntiAffinity:              # spread replicas across nodes
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector: { matchLabels: { app: web } }
            topologyKey: kubernetes.io/hostname

Mnemonic

nodeSelector / affinity = the Pod asking to be near certain nodes (attraction). Taints = the node pushing Pods away (repulsion); a toleration is the Pod’s exception. Attraction lives on the Pod; repulsion lives on the node.

The Horizontal Pod Autoscaler: scaling out automatically

Scheduling places Pods; the Horizontal Pod Autoscaler (HPA) decides how many there should be. It watches a metric — classically CPU utilization against the request — and adjusts a Deployment’s replica count to keep the metric near a target. This is precisely the 2 a.m. manual scale-up, turned into a control loop.

kubectl autoscale deployment web --cpu-percent=70 --min=2 --max=20

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: web }
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 70 }

The loop: observe average CPU → if above 70%, add replicas → if below, remove them (within min/max). Notice the dependency — the HPA’s target is a percentage of the request, so it literally cannot work without requests set. Two more layers exist: VPA right-sizes a single Pod’s requests, and the Cluster Autoscaler adds nodes when Pods can’t be scheduled for lack of room. HPA scales Pods, Cluster Autoscaler scales the machines under them — the deeper cloud picture is in Autoscaling.

   load rises ─► CPU > 70% ─► HPA adds Pods ─► no node has room
                                            ─► Cluster Autoscaler adds a node
   load falls ─► CPU < 70% ─► HPA removes Pods ─► nodes empty ─► CA removes a node

Why this makes production safer

The thread once more: what manual, error-prone step does this remove? Without requests and the scheduler, placing a workload is a human guessing which server has room — and guessing wrong means a starved app or a wasted machine. Without the HPA, surviving a traffic spike means someone awake, watching graphs, and typing scale commands fast enough. Kubernetes converts both into declared intent reconciled by control loops: the scheduler packs Pods to honor their reservations, taints and affinity encode placement policy as data instead of tribal knowledge, and the HPA absorbs load swings without anyone paged. The error-prone, reactive, human-in-the-loop parts of capacity management disappear — provided you feed it honest requests. The last gap is operational: dozens of these YAML files per app. Helm & Packaging closes it.

By the numbers — the HPA’s scaling formula

The HPA isn’t a vague “add some Pods when busy.” It computes a target replica count from one formula:

   desiredReplicas = ceil( currentReplicas × (currentMetric / targetMetric) )

   Example: 4 replicas, average CPU at 90% of request, target 70%
            desiredReplicas = ceil(4 × 90/70) = ceil(5.14) = 6   → scale to 6
   Next tick: 6 replicas now average ~60%, target 70%
            desiredReplicas = ceil(6 × 60/70) = ceil(5.14) = 6   → hold

Two things fall out of the math. First, scaling is proportional — load at twice the target asks for roughly twice the replicas in one step, not a timid +1. Second, the HPA ignores moves inside a small tolerance (about 10% by default) so it doesn’t thrash on noise — and it can only form a ratio at all because currentMetric is measured against the request, which is why an unset request breaks it entirely. The full control theory is in Autoscaling Internals.

The architect’s lens

Step back from requests, taints, and the HPA and judge automated capacity management:

• Why does it exist? Because placement and replica count used to be a human’s job — staring at a spreadsheet to find a server with room, then scaling up by hand at 2 a.m. The scheduler (filter → score) and the HPA turn both into control loops.
• What problem does it solve? It converts reactive, error-prone capacity work into declared intent: the scheduler packs Pods to honor reservations, taints/affinity encode placement policy as data instead of tribal knowledge, and the HPA absorbs load swings (desiredReplicas = ceil(current × currentMetric/target)) without anyone paged.
• What are the trade-offs? It only works if you feed it honest requests — they’re the denominator the HPA divides by and the math the scheduler packs with. Memory is incompressible, so an under-set limit shows up as mysterious OOMKilled restarts; CPU is compressible, so over-limit merely throttles.
• When should I avoid it (or tune it)? Don’t run Pods with no requests — they’re invisible to the scheduler and starve neighbors; don’t over-request either, which wastes money on reserved-but-idle capacity. The HPA also won’t fix a single oversized Pod (that’s VPA) or add machines (that’s the Cluster Autoscaler).
• What breaks if I remove it? Placement reverts to a human guessing which box has room — guess wrong and you get a starved app or a wasted machine — and surviving a spike again means someone awake watching graphs and typing scale commands fast enough.

Check your understanding

1. What is the difference between a resource request and a limit, and which one does the scheduler use to place a Pod?
2. Why does exceeding a CPU limit only throttle a Pod while exceeding a memory limit kills it? Connect this to “compressible vs incompressible.”
3. Describe the scheduler’s two phases. What question does each phase answer?
4. Explain the difference between a nodeSelector/affinity and a taint. If a node is tainted, what must a Pod have to run there?
5. Using the book’s thread, explain how the HPA replaces a manual scaling step — and why it depends on resource requests being set.

Show answers

1. A request is the amount a Pod is guaranteed and reserved — the basis for placement. A limit is the ceiling it may use, enforced by the kernel — the basis for isolation. The scheduler uses the request to decide which node has room.
2. CPU is compressible — the kernel can simply hand out fewer cycles, so over-limit just runs slower (throttled). Memory is incompressible — there’s no “a little less RAM,” so crossing the limit leaves nothing to do but OOM-kill the container. (This is why an under-set memory limit shows up as mysterious OOMKilled restarts.)
3. Filter then score. Filter answers can it run here? — drop nodes lacking free CPU/memory (vs requests) or with untolerated taints/unmet selectors. Score answers where is best? — rank survivors, preferring spread and least-loaded, then bind the Pod to the winner.
4. nodeSelector/affinity is attraction — the Pod asking to be near certain nodes. A taint is repulsion — the node pushing Pods away (“stay off unless you tolerate me”). To run on a tainted node, a Pod must carry a matching toleration (its permission slip).
5. The HPA turns the 2 a.m. manual scale-up (someone awake, watching graphs, typing scale commands) into a control loop: observe average CPU, add replicas above the target, remove them below, within min/max. It depends on requests because its target is a percentage of the request — with no request set there’s no denominator, so it literally cannot compute utilization.