Multi-Region & Disaster Recovery

Autoscaling survives an instance dying. Managed services survive a machine dying. This page is about the failure those don’t cover: an entire region going dark — a data-center fire, a fibre cut, a provider-wide outage, a botched config push that takes down a whole zone. It happens, to every provider, more often than the marketing implies. Disaster recovery (DR) is your plan for the day it happens to you.

DR is also where the whole IaC Part pays off. Why IaC ended on disaster recovery as the payoff that makes engineers believers, and immutable infrastructure ended on rebuildability — “recovery is just redeploying a known-good artifact.” This page is that promise cashed at the scale of a whole region. But first you have to decide how fast you need to recover and how much data you can afford to lose — because those two numbers determine everything, including the bill.

RTO and RPO: the two numbers that define your DR

Every DR conversation reduces to two questions, and you must answer them in minutes and bytes, not adjectives:

• RTO — Recovery Time Objective — how long can you be down? The maximum acceptable time from disaster to service restored. Minutes? Hours? A day?
• RPO — Recovery Point Objective — how much data can you lose? The maximum acceptable amount of recent data lost, measured as a window of time. Zero? The last 5 minutes? The last 24 hours?

                    DISASTER
                       │
   ◄──── RPO ──────────┤────────── RTO ──────────►
   last good           │                          back
   data point          │                          online
   "how much data      │      "how long are
    can we lose?"      │       we down?"
   (data-loss window)  now    (downtime window)

RPO looks backward from the disaster (how far back is your last safe data), RTO looks forward (how long until you’re up again). They are independent — you can have a tiny RPO with a large RTO (you lose no data but take hours to restore) or vice versa. And here is the iron law of DR: smaller RTO and RPO cost more money. “Zero downtime, zero data loss” is achievable, and it is expensive. Most of DR is choosing the cheapest strategy that meets the RTO/RPO the business actually needs — not the heroic one.

The DR ladder: four strategies, trading cost for recovery speed

There are four standard DR postures. They form a ladder: each rung recovers faster (smaller RTO) and loses less data (smaller RPO) than the one below — and costs more to keep running.

   COST  &  RECOVERY SPEED
   ▲
   │  ACTIVE-ACTIVE      both regions serve live traffic; one dies, the
   │                     other already has the load. RTO ≈ 0.
   │  WARM STANDBY       a smaller copy runs in region 2, scaled down;
   │                     on failover, scale it up. RTO: minutes.
   │  PILOT LIGHT        only the core (DB replica) runs in region 2;
   │                     spin up the rest on failover. RTO: tens of min.
   │  BACKUP & RESTORE   backups stored in another region; rebuild from
   │                     scratch on disaster. RTO: hours. Cheapest.
   ▼

Strategy	What’s running in the standby region	RTO	RPO	Cost
Backup & restore	nothing — only stored backups	hours	hours (last backup)	lowest
Pilot light	the core only (a replicating DB)	tens of minutes	minutes (replication lag)	low
Warm standby	a scaled-down full copy	minutes	seconds–minutes	medium
Active-active	a full copy serving live traffic	~zero	~zero	highest

• Backup & restore — you keep backups in a second region and, on disaster, rebuild the whole stack there from scratch. Cheapest (you pay only for stored backups) and slowest (rebuilding takes hours). Viable only because of IaC: “rebuild the whole stack” is terraform apply against a fresh region, not a multi-day manual reconstruction.
• Pilot light — keep the irreplaceable core warm: a database replica continuously receiving data in the second region. Everything else (the stateless compute) is defined in code but not running. On failover you launch the rest around the already-current data. Faster, because the slow part — getting the data there — is already done.
• Warm standby — a complete but scaled-down copy of the system runs in the second region, taking data replication, ready to serve. On failover you scale it up to full size and redirect traffic. Minutes to recover, because everything’s already running, just small.
• Active-active — both regions run full-scale and serve live traffic simultaneously. A region failing means the survivor already has the data and the traffic; recovery is near-instant. The most resilient and the most expensive — you’re running (at least) two of everything, all the time.

The hard part: data and the consistency cost

Notice that the strategies are easy to describe for stateless compute — it’s just immutable images you redeploy anywhere — and hard for data. That’s the crux of multi-region: getting your data to the standby region, current enough to meet your RPO.

The catch is physics. Regions are far apart, and replicating data between them takes time. This forces a choice the rest of distributed systems never lets you escape:

• Asynchronous replication — the primary acknowledges a write immediately, then ships it to the other region in the background. Fast for users (no cross-region wait on every write), but if the primary dies before a write is shipped, that write is lost. Your RPO is the replication lag — typically seconds to minutes of potential data loss.
• Synchronous replication — the write isn’t acknowledged until both regions have it. RPO is zero — no committed write is ever lost. But every single write now pays the cross-region round-trip in latency, which can be tens of milliseconds added to every write, everywhere.

   ASYNC: ack fast, may lose the last few writes   →  small RPO, low latency
   SYNC:  ack only when both regions have it        →  zero RPO, high latency
                                                        (every write waits)

You cannot have low latency, zero data loss, and multi-region all at once

This is the consistency cost, and it’s not a tooling problem you can buy your way out of — it’s the speed of light. Synchronous cross-region replication gives you RPO zero but taxes every write with the round-trip latency. Asynchronous keeps writes fast but accepts a non-zero RPO. Active-active makes it harder still: two regions taking writes at once can conflict, and resolving that needs either careful data partitioning or conflict resolution. “Multi-region” is never free; it’s a standing latency-or-data- loss trade you design around.

Failover: pointing traffic at the survivor

Having a standby is only half of DR; you also need to send traffic to it when the primary fails. That redirection is failover, and it’s usually done at the DNS or global-load-balancer layer with health checks: probe the primary, and when it fails, resolve traffic to the standby instead.

Failover can be automatic (health checks flip traffic with no human) or manual (a human confirms the disaster is real and triggers it). Automatic is faster but risks flapping — a brief blip triggers a failover you didn’t want, or worse, a split-brain where both regions think they’re primary and both take writes. Many teams keep the decision manual for exactly that reason, even when the mechanism is automated: a human confirms “yes, the region is really gone” before the switch, trading a few minutes of RTO for protection against a false alarm doing more damage than the outage.

Why DR only works if you rehearse it

Here is the single most important sentence on this page: an untested DR plan is not a DR plan — it’s a hope. A runbook nobody has executed, a standby nobody has failed over to, a backup nobody has restored: all of these feel like safety and provide none. They fail in the ways that only show up under real conditions — the backup that won’t restore, the IAM role missing in region 2, the DNS TTL that’s too long to fail over quickly, the one manual step the runbook forgot.

This is precisely why DR and immutable, rebuildable infrastructure are the same idea wearing different clothes. If your platform is genuinely re-derivable from a repo — if terraform apply against a fresh region actually produces a working system — then DR isn’t a fragile binder of manual steps; it’s a command you can run on a normal Tuesday to test it. Rebuildability is what makes DR rehearsable, and rehearsability is what makes DR real.

Rehearse the disaster, on purpose, on a schedule

The discipline of deliberately triggering failures to prove your recovery works — game days, scheduled failover drills, even continuously failing things in production — is its own field: chaos engineering. The principle is simple and uncomfortable: the only DR plan you can trust is one you’ve executed, recently, and watched succeed. Rehearse it before the disaster forces you to, when the stakes are a calendar invite, not the company.

It actually happened — the typo that broke us-east-1 (2017)

On 28 February 2017 an authorized AWS engineer, debugging the S3 billing system, ran a standard runbook command to remove a few servers — and a typo in the command removed far more than intended, including servers running two core S3 subsystems in us-east-1. Restarting them took hours, and because an enormous slice of the internet stored objects and assets in that one region, sites everywhere broke for roughly four hours. The cruelest detail: AWS’s own status dashboard couldn’t show the outage, because its status icons were themselves served from the failed S3. The lessons are this page’s whole argument — a single region is a single blast radius, hidden dependencies surface only under failure, and one mistyped manual command can take down a continent of services. us-east-1 has had further large outages since — a notable control-plane disruption in December 2021, and an October 2025 DynamoDB DNS failure that degraded much of the region (and the services built on it) for the better part of a day — which is why “what happens when this entire region goes dark?” is a question you answer before it does — with the DR ladder above and a rehearsed failover.

The thread: from heroics to a routine you’ve already run

The manual, error-prone step DR-done-right removes is reconstructing your platform from memory under maximum pressure — the catastrophe response where exhausted engineers rebuild infrastructure by hand at 3 a.m., guessing at config they never wrote down, while the business bleeds. Multi-region architecture and rebuildable IaC replace that with a rehearsed, tested procedure: data is already replicated (pick your RPO), the standby is already defined in code (pick your RTO and pay for it), and failover is a switch you’ve flipped before in a drill.

Production gets safer in the most literal sense — it survives events that would otherwise end it — but only to the degree you’ve honestly matched your strategy to your RTO/RPO and rehearsed the recovery. The cost is equally literal: every rung up the DR ladder is more standing infrastructure billing every hour, and every reduction in RPO is latency or money spent on replication. DR is the purest expression of this Part’s theme — you are quite precisely trading cost for recovery speed, and the only wrong answer is the untested one.

This is the last page of Part 9. You can now reason about the cloud end to end: what it is, what it’s made of, how it scales, what to buy versus build, what it costs, and how it survives a region going dark — all of it tamed as code. Next we go past the fundamentals into the techniques that define modern platforms: Part 10 · Advanced.

The architect’s lens

Five questions for sizing your disaster-recovery posture:

• Why does it exist? Because autoscaling survives an instance dying and managed services survive a machine dying, but neither covers an entire region going dark — which happens to every provider (us-east-1, 2017).
• What problem does it solve? It turns “reconstruct the platform from memory under pressure” into a rehearsed procedure, sized to RTO (how long down) and RPO (how much data lost) — the cheapest rung on the backup → pilot-light → warm-standby → active-active ladder that meets the business’s real numbers.
• What are the trade-offs? Every rung up is more standing infrastructure billing every hour, and every cut to RPO is paid in latency or money — synchronous replication buys RPO-zero but taxes every write with the cross-region round-trip (the speed-of-light limit you can’t buy past).
• When should I avoid the higher rungs? When the business genuinely tolerates hours of RTO and a backup-window of RPO — don’t pay active-active prices for a workload that only needs backup-and-restore.
• What breaks if I never rehearse it? An untested DR plan is a hope — the backup that won’t restore, the missing region-2 IAM role, the too-long DNS TTL all surface only under real failure; rebuildable IaC is what lets you drill it on a calm Tuesday.

Check your understanding

1. Define RTO and RPO precisely, including which direction each looks relative to the disaster. Give an example of a system with a small RPO but a large RTO.
2. Name the four DR strategies from cheapest to most resilient, and state in one phrase what’s running in the standby region for each.
3. Explain the consistency cost of multi-region data. Why can’t you have low write latency, zero data loss, and multi-region all at once?
4. What is failover, and why do many teams keep the decision manual even when the mechanism is automated? Name the failure modes they’re guarding against.
5. Using the book’s thread, explain why “an untested DR plan is a hope, not a plan,” and how rebuildable IaC makes DR rehearsable.

Show answers

1. RTO (Recovery Time Objective) is the maximum acceptable downtime — it looks forward from the disaster to “back online.” RPO (Recovery Point Objective) is the maximum acceptable data loss as a time window — it looks backward to your last safe data point. A backup-and-restore system can have a small RPO (frequent backups, little data lost) but a large RTO (rebuilding from scratch takes hours).
2. Cheapest → most resilient: backup & restore (nothing running — only stored backups), pilot light (only the core, e.g. a replicating database), warm standby (a scaled-down full copy taking replication), active-active (a full copy serving live traffic simultaneously).
3. Replicating data across distant regions takes time. Synchronous replication acks a write only once both regions have it — RPO zero, but every write pays the cross-region round-trip latency. Asynchronous acks immediately and ships in the background — fast writes, but a crash before shipping loses the last writes (RPO = replication lag). The limit is the speed of light, not tooling, so you can’t get all three at once — you pick which to sacrifice.
4. Failover is redirecting traffic (usually via DNS or a global load balancer with health checks) from the failed primary to the standby. Teams keep the decision manual to guard against flapping (a brief blip triggering an unwanted failover) and split-brain (both regions believing they’re primary and both taking writes) — a human confirms the region is really gone before the switch, trading a little RTO for protection against a false alarm causing more damage than the outage.
5. Because a plan nobody has executed fails in ways only real conditions reveal — a backup that won’t restore, a missing IAM role in region 2, a too-long DNS TTL, a forgotten manual step. It removes rebuilding from memory under pressure. Rebuildable IaC makes DR rehearsable because if terraform apply against a fresh region genuinely produces a working system, you can run the recovery as a drill on a calm day and watch it succeed — turning DR from a fragile binder into a tested, routine command.