Autoscaling Internals

Scaling & Scheduling introduced the Horizontal Pod Autoscaler as a control loop that watches CPU and adjusts a Deployment’s replica count. That page told you what it does. This one opens the box: where the number it compares against actually comes from, the exact arithmetic it runs, why it dampens its own output, and how four different autoscalers — HPA, VPA, Cluster Autoscaler, KEDA — stack into one system that scales Pods, right-sizes them, and adds machines underneath, all without a human watching a graph.

Autoscaling is the purest expression of the book’s thread. The manual step it removes is a person on call, reading dashboards, and typing kubectl scale fast enough to survive a spike — and, on the way down, remembering to scale back so you don’t pay for idle capacity at 3 a.m. Every autoscaler below is a reconciliation loop whose “desired” is a target metric and whose “actual” is what the cluster is currently observing.

The metrics pipeline: where the numbers come from

The HPA never measures anything itself. It reads metrics from an aggregated API, and something else is responsible for populating that API. Kubernetes exposes three metrics APIs, each backed by a different supplier:

API group	Example metric	Supplied by
metrics.k8s.io (resource)	Pod CPU / memory	metrics-server
custom.metrics.k8s.io	requests-per-second on this Pod	a custom-metrics adapter (e.g. Prometheus Adapter)
external.metrics.k8s.io	SQS queue depth, Kafka lag (off-cluster)	an external-metrics adapter / KEDA

For plain CPU/memory autoscaling you need exactly one component: metrics-server. It is not installed by default on a bare cluster — a fact that surprises people whose first HPA reports <unknown> for utilization. metrics-server scrapes the /metrics/resource summary endpoint on every kubelet (which in turn reads it from cAdvisor inside the kubelet), keeps the latest sample in memory only, and serves it through the aggregation layer as the metrics.k8s.io API.

   kubelet (cAdvisor) ──scrape──► metrics-server ──serves──► metrics.k8s.io
        ▲                              (in-RAM, last sample)        │
        │ per-node                                                  │ aggregated API
   container cgroup stats                                           ▼
                                                          HPA reads utilization

metrics-server is for autoscaling, not observability

metrics-server keeps only the most recent sample, in memory, and is meant to feed the HPA/VPA and kubectl top. It is not a monitoring system — you cannot query history, alert, or build dashboards from it. For that you want a real time-series database; see Metrics. The two coexist: Prometheus for humans, metrics-server (or a Prometheus adapter) for the autoscaler.

The moment you want to scale on anything other than raw CPU/memory — requests-per-second, p99 latency, queue depth — you install an adapter that registers the custom.metrics.k8s.io or external.metrics.k8s.io API and translates the HPA’s queries into your metrics backend. The Prometheus Adapter is the common one: you write a rule mapping a PromQL query to a metric name, and the HPA can then target it. Custom metrics are attached to a Kubernetes object (per-Pod RPS); external metrics describe something outside the cluster entirely (a cloud queue).

The HPA control-loop algorithm

The HPA controller runs inside the controller-manager and re-evaluates every target on a fixed period (--horizontal-pod-autoscaler-sync-period, default 15s). Each tick it does the canonical observe → compare → act, and the “compare” is one formula:

   desiredReplicas = ceil( currentReplicas × (currentMetricValue / desiredMetricValue) )

If average CPU is 90% of request and your target is 50%, then ratio = 1.8, so 10 replicas becomes ceil(10 × 1.8) = 18. That is the whole core. Everything else is guardrails around this multiply, and the guardrails are what separate a stable autoscaler from one that oscillates:

• Tolerance. If the ratio is within a small band of 1.0 (default ±10%, --horizontal-pod-autoscaler-tolerance=0.1), the HPA does nothing. Without this, a metric jittering by 1% would churn replicas every 15 seconds.
• Ready/missing Pod handling. Pods that are not yet Ready, or are missing metrics, are excluded or treated conservatively so a fleet that’s still warming up isn’t counted as “doing work.”
• Stabilization window. For scale-down, the HPA looks back over a window (default 300s) and uses the highest desiredReplicas it computed in that window. This is the single most important anti-flapping mechanism: it makes the autoscaler quick to scale up and deliberately slow to scale down, so a brief lull doesn’t tear down capacity you’ll need again in a minute.
• Behavior policies. spec.behavior lets you cap the rate of change — e.g. “add at most 4 Pods or 100% every 60s,” “remove at most 10% per minute” — independently for scaleUp and scaleDown.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: web }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: web }
  minReplicas: 2
  maxReplicas: 50
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 50 } }
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300   # look back 5m, take the max desired
      policies: [ { type: Percent, value: 10, periodSeconds: 60 } ]
    scaleUp:
      stabilizationWindowSeconds: 0     # react immediately on the way up
      policies: [ { type: Pods, value: 4, periodSeconds: 60 } ]

Why “fast up, slow down” is the right asymmetry

The cost of scaling up too eagerly is a few extra Pods for a few minutes. The cost of scaling down too eagerly is dropping capacity right before a second wave — a latency spike or outage. The penalties are asymmetric, so the algorithm is deliberately asymmetric: a 0s up-window and a 300s down-window encode “when in doubt, keep the capacity.”

Because the target is a percentage of the resource request, the HPA cannot compute utilization if the Pod has no CPU request set — exactly the dependency Scaling & Scheduling flagged. Requests aren’t just for the scheduler; they’re the denominator the HPA divides by.

VPA: scaling the Pod up, not out

The HPA changes how many Pods you run. The Vertical Pod Autoscaler (VPA) changes how big each one is — it recommends (and optionally applies) better requests based on observed historical usage. It exists because hand-tuning requests is guesswork: set them too low and Pods get throttled or OOM-killed; too high and you reserve idle capacity. The VPA has three pieces:

• Recommender — watches usage history and computes a target request per container.
• Updater — evicts Pods whose requests are far off so they get recreated with the new values.
• Admission controller — a mutating webhook that rewrites the requests on the new Pod as it’s created.

Don’t run HPA-on-CPU and VPA on the same workload

They fight. The HPA scales replicas by CPU utilization relative to the request; the VPA changes the request out from under it, moving the very denominator the HPA divides by. Use VPA in recommendation-only mode alongside a CPU HPA, or let VPA own resources while the HPA scales on a different signal (e.g. RPS). Vertical scaling has also traditionally meant a restart — the VPA’s Updater evicts Pods to apply new requests — though in-place Pod resize (stable since Kubernetes 1.35) is beginning to lift that restriction.

Cluster Autoscaler: adding and removing nodes

The HPA and VPA assume there’s somewhere to put Pods. When the scheduler can’t place a Pod because no node has room, it stays Pending — and that Pending Pod is the Cluster Autoscaler’s trigger.

• Scale-up is a simulation. The CA notices unschedulable Pods, then for each node group it asks: “if I added one more node of this shape, would the scheduler be able to place these Pods?” It runs the real scheduler’s predicates against a hypothetical node. If yes, it asks the cloud provider to grow that group by one (or more). It does not look at CPU utilization — it reasons purely about schedulability.
• Scale-down is eligibility-based. A node becomes a removal candidate when its utilization sits below a threshold (default ~50%) and every Pod on it could be rescheduled elsewhere. A node is not eligible if it hosts Pods that block eviction: Pods with no controller, Pods using local storage, Pods not covered by a PodDisruptionBudget that would be violated, or anything marked cluster-autoscaler.kubernetes.io/safe-to-evict: "false".

   Pod Pending (no room) ──► Cluster Autoscaler simulates "+1 node?"
                                    │ scheduler predicates pass on hypothetical node
                                    ▼
                            cloud API: grow node group ──► node joins ──► Pod scheduled

   Node <50% used AND all Pods reschedulable ──► cordon ──► drain ──► cloud API: remove node

The CA scales the machine layer; the HPA scales the Pod layer. They compose without coordination because they communicate through one shared signal: a Pending Pod. The HPA creates Pods that may not fit; the unfit Pod’s Pending status tells the CA to add a node; the node lets the scheduler place the Pod. Each loop minds only its own gap.

Karpenter: provisioning without fixed node groups

The classic Cluster Autoscaler grows pre-defined node groups of a fixed shape. Newer provisioners (Karpenter on AWS, and the general “just-in-time node provisioning” pattern) instead look at the pending Pods’ actual requirements and provision a node shaped to fit them — choosing instance type and size on the fly. Same trigger (Pending Pods), more flexible response.

KEDA: event-driven scaling, including scale-to-zero

The HPA scales on resource and metric levels. A huge class of workloads — queue consumers, stream processors, cron-driven jobs — should instead scale on events: the depth of a Kafka topic, the number of messages in an SQS queue, rows awaiting processing in a database. KEDA (Kubernetes Event-Driven Autoscaling) fills this gap. It is not a replacement for the HPA; it’s a layer on top of it.

You declare a ScaledObject listing triggers (a Kafka lag, an SQS queue, a cron schedule). KEDA’s operator reads those external sources via scalers, and then:

1. Generates and manages an HPA for you, feeding it through the external.metrics.k8s.io API — so the actual scaling 1→N is still the HPA algorithm above.
2. Adds the one thing the HPA can’t do alone: scale to zero. The HPA’s floor is minReplicas: 1; KEDA’s activation logic scales the Deployment from 0→1 when the first event arrives and back to 0 when the source is empty, so an idle consumer costs nothing.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: order-worker }
spec:
  scaleTargetRef: { name: order-worker }
  minReplicaCount: 0          # scale to zero when idle
  maxReplicaCount: 30
  triggers:
    - type: aws-sqs-queue
      metadata: { queueURL: https://sqs.../orders, queueLength: "5" }

That queueLength: "5" reads “run one Pod per 5 in-flight messages” — the same desiredReplicas multiply, but the metric is queue depth instead of CPU.

How they compose

The four autoscalers form a stack, each owning a different axis and communicating through the layer below it:

   KEDA            ── translates events (queue/lag/cron) into a metric, and 0↔1
     ▼ feeds external metric
   HPA             ── chooses how MANY Pods (replica count)
     ▼ creates Pods that may not fit
   Cluster AS      ── chooses how many NODES (machine count)
   VPA  (sideways) ── chooses how BIG each Pod is (requests)

A queue-backed service might run all four: KEDA wakes it from zero on the first message and feeds queue-depth to an HPA that scales 1→N; the HPA’s new Pods go Pending when nodes fill, prompting the Cluster Autoscaler to add machines; and a VPA in recommend-only mode tells you the per-Pod requests are 2× too high. The thread, stated plainly: every reactive, manual, error-prone capacity decision — more Pods, bigger Pods, more nodes, wake it from idle — becomes a declared target reconciled by a loop. What it costs you is honesty about your numbers (requests and targets) and discipline about the dampening (windows and PDBs), or the loops will oscillate or evict the wrong thing.

The next page goes one level deeper into the resource numbers these autoscalers depend on: what requests and limits actually do to a running container, and what the node does when it runs out of room — Resource Management & Node Pressure.

By the numbers — why the HPA isn’t instant

Reading the formula, scaling looks immediate; in practice a spike takes a visible budget of time to become serving capacity, because three independent delays stack up:

   load spikes
     + up to ~15 s   metrics-server scrapes the kubelet (default resolution)
     + up to  15 s   the HPA controller's next sync tick reads the new value
     + scheduling + image pull + readiness   (seconds … to minutes for a cold image)
   ─────────────────────────────────────────────────────────────────────────
   ≈ 30 s before a new Pod even starts; often 1–3 min before it serves traffic

This is why the metrics pipeline’s own latency matters, and why bursty workloads pre-provision headroom (a higher minReplicas) rather than trusting the HPA to catch a sudden spike — the loop is reactive, and reaction has a floor.

The architect’s lens

Step back from the four autoscalers and answer the five questions that decide how — and whether — you wire them up:

• Why does it exist? Because the manual step is a person on call reading dashboards and typing kubectl scale fast enough to survive a spike — then remembering to scale back so you don’t pay for idle capacity at 3 a.m. Each autoscaler is a reconciliation loop whose desired is a target metric.
• What problem does it solve? It turns every reactive capacity decision into a declared target: more Pods (the HPA’s ceil(replicas × current/target)), bigger Pods (VPA), more machines (Cluster Autoscaler reacting to a Pending Pod), and wake-from-idle (KEDA’s scale-to-zero) — composing without coordination through shared signals.
• What are the trade-offs? The loop is reactive, so a spike takes ≈30 s to a new Pod and 1–3 min to serve traffic (metrics-server’s ~15 s scrape + the HPA’s 15 s sync + scheduling/pull); it only works on honest numbers — the HPA divides by the CPU request — and survives only with dampening (the 300 s scale-down window, behavior policies, PDBs) or it oscillates.
• When should I avoid it? Never run a CPU-based HPA and a VPA on the same workload — the VPA moves the very denominator the HPA divides by; pre-provision a higher minReplicas for bursty traffic rather than trusting the reactive loop to catch a sudden wave; and don’t treat metrics-server as monitoring — it keeps only the last sample.
• What breaks if I remove it? Capacity goes back to being hand-arbitrated — you either over-provision and pay for idle Pods and nodes at 3 a.m., or under-provision and drop traffic on the next wave; idle queue consumers can’t fall to zero; and a Pending Pod never makes a node appear.

Check your understanding

1. A teammate’s first HPA shows TARGETS: <unknown>/50% and never scales. Walk the metrics pipeline and name the two most likely causes.
2. Average CPU is 80% of request, the target is 40%, and there are 6 replicas. What does the desiredReplicas formula produce, and what role does tolerance play in whether the HPA acts?
3. Why is the scale-down stabilization window (default 300s) the key anti-flapping mechanism, and why is the up/down asymmetry deliberate rather than a bug?
4. Why should you avoid running a CPU-based HPA and a VPA on the same workload? What’s the underlying conflict?
5. The Cluster Autoscaler ignores node CPU utilization when deciding to scale up. What does it look at instead, and how does this let it compose with the HPA with no direct coordination?

Show answers

1. The HPA reads the metrics.k8s.io API, which is populated by metrics-server. The two usual causes are (a) metrics-server isn’t installed (it’s not default on a bare cluster), so nothing serves the API; or (b) the target container has no CPU request set, so there’s no denominator to compute utilization as a percentage of request. Both surface as <unknown>.
2. ratio = 80/40 = 2.0, so desiredReplicas = ceil(6 × 2.0) = 12. Tolerance (default ±10%) is the dead-band: if the ratio were within 0.9–1.1 the HPA would do nothing, preventing churn from tiny metric jitter. Here 2.0 is far outside the band, so it scales.
3. For scale-down the HPA takes the maximum desiredReplicas over the look-back window, so a brief dip in load doesn’t immediately tear down Pods you’ll need again seconds later. The asymmetry (≈0s up, 300s down) is intentional because the penalties are asymmetric: over-scaling up costs a few idle Pods briefly; over-scaling down risks dropping capacity right before another wave — an outage. “When in doubt, keep capacity.”
4. The HPA scales replicas on CPU utilization relative to the request; the VPA changes the request, which is the HPA’s denominator. They fight — the VPA moves the very number the HPA divides by. Use VPA in recommend-only mode with a CPU HPA, or have VPA own requests while the HPA scales on a different signal like RPS.
5. It looks at schedulability: it simulates adding a node and asks whether the real scheduler’s predicates would then place the Pending Pods. They compose through that shared signal — the HPA creates Pods that may not fit, the resulting Pending status triggers the CA to add a node, and each loop only ever reconciles its own gap.