Autoscaling

Core primitives gave you compute you can create and destroy with an API call. This page is about the thing that becomes possible only because of that: letting a program create and destroy compute for you, in response to load. That’s autoscaling, and it’s the cloud’s headline trick — the one on-prem hardware could never do, because you can’t grow a server farm at 2 a.m. when traffic spikes and shrink it again at dawn.

You’ve already met the core of this idea twice. Scaling & Scheduling introduced the Horizontal Pod Autoscaler, and the reconciliation loop named the pattern underneath it. Autoscaling is that pattern pointed at capacity: observe load, compare it to a target, act by adding or removing capacity, repeat — forever. Everything below is a variation on that single loop.

Horizontal vs vertical: more boxes, or bigger boxes

There are exactly two ways to give a system more capacity, and the difference matters enormously:

   VERTICAL (scale up)              HORIZONTAL (scale out)
   ───────────────────             ──────────────────────
   ┌────┐      ┌────────┐          ┌──┐         ┌──┐┌──┐┌──┐┌──┐
   │ 2  │ ───► │   8    │          │ 1│  ───►   │ 1││ 1││ 1││ 1│
   │ CPU│      │  CPU   │          └──┘         └──┘└──┘└──┘└──┘
   └────┘      └────────┘          one instance   four instances
   one bigger machine               more machines, same size

• Vertical (scale up) — give the same instance more CPU/RAM. Simple, but it hits a ceiling (the biggest instance type), usually requires a restart, and gives you no redundancy — it’s still one box, one failure away from zero.
• Horizontal (scale out) — run more instances behind a load balancer. No hard ceiling, and N instances means losing one costs you 1/N of capacity, not all of it.

Horizontal scaling is the cloud default, and it’s why the stateless, cattle-not-pets design discipline matters: you can only add and remove identical instances freely if no instance holds unique state. State lives in storage and managed services; compute stays disposable. That’s the precondition that makes autoscaling possible at all.

Why “stateless” is the price of admission for autoscaling

An autoscaler adds and kills instances on a whim — that’s its whole job. If an instance holds the only copy of something (a user’s session, an uploaded file, a write-ahead log), killing it loses data and adding one doesn’t help. So horizontal autoscaling requires that instances be interchangeable: state pushed out to a database, cache, or object store. Design for disposability first; the scaling follows.

Reactive scaling: a control loop on a metric

The most common autoscaler is reactive: it watches a live metric and reacts when the metric crosses a threshold. This is exactly the Kubernetes HPA and the cloud’s own instance-group autoscalers. The loop:

   ┌──────────────── every 15–60s ─────────────────┐
   │  OBSERVE   read the metric (CPU%, RPS, queue)   │
   │  COMPARE   metric vs target (e.g. CPU 70%)       │
   │  ACT       above target → add instances          │
   │            below target → remove instances        │
   └──────────────────┬──────────────────────────────┘
                       └── repeat forever; capacity tracks load

That is the reconciliation loop with replica count as the thing being reconciled and a load metric as the signal. The art is choosing the right metric. CPU is the default but often wrong — a web app bottlenecked on a slow database can be at 100% latency and 20% CPU. A better signal is usually requests per second per instance or queue depth, because those track the actual work, not a proxy for it.

Reactive scaling is always a step behind

A reactive autoscaler only acts after load has already risen and the metric has crossed the line — and then new instances take time to boot, pull an image, and warm up. During that gap, the existing instances are overloaded. For gentle ramps this is fine. For a sharp spike (a marketing blast, a flash sale, the top of the hour) reactive scaling can be permanently chasing a load it never catches. That’s the gap the next two approaches fill.

Predictive and scheduled scaling: get ahead of known load

When load is predictable, you don’t have to wait for it:

• Scheduled scaling — scale on a clock. “Run 20 instances 9am–6pm on weekdays, 4 overnight.” If your traffic has a daily shape you can see in a graph, encode it. Dead simple, no machine learning, and it beats reactive scaling for known patterns because the capacity is already there when load arrives.
• Predictive scaling — use historical patterns to forecast load and pre-provision for it. The provider learns “Mondays at 8am spike” and scales up before the spike. More sophisticated, more failure-prone (forecasts can be wrong), but it closes the reactive gap for repeating patterns.

The honest framing: scheduled and predictive scaling handle the expected load; reactive scaling is the safety net for the unexpected on top of it. Real systems run both — a predicted baseline plus a reactive loop for surprises.

Cluster autoscaling: scaling the machines under the Pods

There’s a subtlety the HPA page flagged. The HPA adds Pods — but a Pod needs a node to run on, and if every node is full, the new Pods sit Pending, unscheduled, doing nothing. The Cluster Autoscaler closes that gap by scaling the nodes:

   load rises ─► HPA adds Pods ─► no node has room ─► Pods stuck Pending
                                                   ─► Cluster Autoscaler adds a NODE
                                                   ─► scheduler places the Pods
   load falls ─► HPA removes Pods ─► a node goes empty ─► Cluster Autoscaler removes the NODE

Two loops, stacked: the HPA reconciles Pod count to a metric; the Cluster Autoscaler reconciles node count to “are there unschedulable Pods / empty nodes?” The first scales your app; the second scales the fleet your app runs on. You need both, because adding Pods to a full cluster accomplishes nothing.

Scale-to-zero and serverless: the deepest cut

Every approach so far keeps a minimum number of instances running, even idle, because spinning up takes time. Scale-to-zero drops that floor to nothing: when there’s no traffic, run zero instances and pay for nothing. This is the heart of serverless (AWS Lambda, Cloud Run, Cloud Functions) — you don’t provision capacity at all; the platform runs your code only when a request arrives and tears it down after.

   no traffic        request arrives          steady traffic       traffic stops
   ──────────        ───────────────          ──────────────       ─────────────
   0 instances  ──►  COLD START: spin up  ──► instances warm,  ──►  scale back
   $0/idle           (latency penalty)        serving fast          to 0

The payoff is enormous for spiky or low-traffic workloads: a service used a few times an hour costs essentially nothing between requests, instead of an idle VM billing 24/7. The cost implications are the reason serverless exists.

The catch is the cold start. When scaled to zero, the first request after idle has to wait for the platform to allocate compute, load your code, and initialize the runtime — adding anywhere from tens of milliseconds to several seconds of latency before that request is served. Subsequent requests hit a warm instance and are fast. Cold starts are the price of paying nothing while idle, and they’re why serverless fits background jobs, spiky APIs, and event handlers better than a latency-critical user-facing path — unless you pay to keep a minimum warm (which is just scale-to-zero with the floor lifted back up).

Every option is the same dial, set differently

Vertical/horizontal, reactive/scheduled/predictive, minimum-instances vs scale-to-zero — they’re all one dial: how aggressively do you trade idle cost for response speed? Keep instances warm and you pay for idle but never cold-start; scale to zero and you pay nothing but eat cold-start latency. There’s no free setting — only the right point on that trade-off for this workload’s traffic shape and latency budget.

By the numbers — the reactive gap

A reactive autoscaler can’t help until the metric crosses the line and fresh capacity is warm. Add up the lag for a sudden doubling of traffic:

   poll / observe interval   ~15–60 s    (notice the breach)
   provision + boot          ~30–90 s    (request, schedule, start)
   image pull + warm-up      ~20–120 s   (cold caches, JIT, connections)
   ───────────────────────────────────────────────────────────────────
   time to first new capacity   ≈ 1–4 minutes of running hot

For one to four minutes the existing fleet absorbs the entire spike alone. That’s survivable for a gentle ramp and brutal for a flash sale — which is exactly why scheduled and predictive scaling exist: to put the capacity in place before the metric ever moves.

The thread: the pager that no longer rings

The manual, error-prone step autoscaling removes is a human watching graphs and typing capacity commands — the on-call engineer who, at 2 a.m. during a spike, SSHes in to launch more servers, and who (more often) forgets to scale back down afterward, leaving expensive idle capacity running for weeks. Autoscaling turns capacity into a control loop: observe load, reconcile capacity to it, in both directions, forever, with nobody paged.

Production gets safer in two ways at once. Availability: the system absorbs a traffic spike by growing instead of falling over, and survives an instance death by being 1/N down instead of all-the-way down. Cost safety: scaling back down automatically — all the way to zero where it fits — removes the human’s most expensive mistake, forgetting to clean up. The cost of all this is real: reactive scaling lags, cold starts hurt latency, and an autoscaler tuned wrong can thrash (scaling up and down repeatedly) or hit a limit you didn’t set and stop. Which is the perfect segue: capacity that scales itself is also a bill that scales itself. Next, Cost & FinOps.

The architect’s lens

Five questions for putting capacity on a control loop:

• Why does it exist? Because compute you can create and destroy with an API call makes a new thing possible — letting a program add and remove capacity in response to load, the trick on-prem hardware never could do.
• What problem does it solve? A human watching graphs at 2 a.m. (and forgetting to scale back down): it runs the reconciliation loop on a load metric, reconciling capacity in both directions — availability when load spikes, cost safety when it falls.
• What are the trade-offs? It’s one dial — idle cost vs response speed. Reactive scaling is always a step behind (~1–4 min to warm new capacity), scale-to-zero swaps that idle cost for cold-start latency, a mistuned autoscaler can thrash — and it all requires stateless instances.
• When should I avoid it? A latency-critical path can’t eat cold starts (keep a warm floor); steady, predictable load may be cheaper on a fixed reserved baseline than a churning reactive loop.
• What breaks if I remove it? Capacity goes back to a human SSHing in to launch servers under a spike — and forgetting to remove them — so you topple on spikes and bleed money on idle.

Check your understanding

1. Distinguish horizontal from vertical scaling. Why is horizontal the cloud default, and what design property must your instances have for it to work?
2. Reactive autoscaling is described as a reconciliation loop. What plays the role of desired, actual, and the signal? Why is reactive scaling “always a step behind”?
3. The HPA adds Pods but your new Pods are stuck Pending. What’s happening, and which autoscaler fixes it? Describe how the two loops stack.
4. What is scale-to-zero, what does it buy you, and what is the cold-start cost? Name one workload it fits well and one it fits poorly.
5. Using the book’s thread, explain the manual step autoscaling removes and the two distinct production-safety wins — then name one way an autoscaler tuned wrong can hurt you.

Show answers

1. Vertical = a bigger instance (more CPU/RAM on one box); horizontal = more instances of the same size behind a load balancer. Horizontal is the default because it has no hard ceiling and losing one of N instances costs only 1/N of capacity. It requires instances to be stateless / disposable — no instance can hold unique state, since the autoscaler kills and creates them freely.
2. Desired = the target metric value (e.g. CPU 70%); actual = the current metric reading; the autoscaler acts by changing replica count to close the gap. It’s a step behind because it only reacts after the metric crosses the threshold, and new instances then need time to boot and warm up — during that gap the running instances are overloaded.
3. Every node is full, so the new Pods have nowhere to be scheduled and sit Pending. The Cluster Autoscaler fixes it by adding a node. Two stacked loops: the HPA reconciles Pod count to a metric; the Cluster Autoscaler reconciles node count to “are there unschedulable Pods or empty nodes?” You need both because adding Pods to a full cluster does nothing.
4. Scale-to-zero drops the running-instance floor to zero when there’s no traffic, so you pay nothing while idle (the basis of serverless). The cost is the cold start: the first request after idle waits for the platform to allocate compute and initialize the runtime, adding latency. It fits spiky/low-traffic work (background jobs, event handlers, rarely-hit APIs) and fits poorly a latency-critical user-facing path.
5. It removes a human watching graphs and manually launching/retiring servers — including the common, costly mistake of forgetting to scale back down. The two safety wins are availability (grow to absorb spikes, survive instance loss as 1/N) and cost safety (automatic scale-down, even to zero). Tuned wrong, an autoscaler can thrash (rapidly scale up and down) or hit an unset/low limit and stop scaling when you need it.