Notable Outages of the 2020s

Incident Response & Postmortems argued that a failure is the most valuable data you’ll get all quarter — if you turn it into a lesson the system enforces. The cheapest way to buy that lesson is to learn it from someone else’s worst day. The 2020s have provided an unusually instructive set of large, public, well-documented outages, and the striking thing is how few of them were caused by an attacker or an exotic failure. Almost all were ordinary mechanisms — a latent bug, a routine command, an automated process — amplified by a system with too little blast-radius containment. This page tours three canonical ones and the single systemic lesson each teaches, then points at the case study that earns its own page.

Fastly — a latent bug, woken by a valid change (8 June 2021)

Fastly is a major CDN; an enormous slice of the web sits behind it. On 8 June 2021, a large part of the internet returned errors at once — Amazon, Reddit, Spotify, GitHub, the UK government’s gov.uk, major news sites — for roughly an hour.

There was no attack. Weeks earlier, a software deployment had introduced a latent bug — dormant, harmless, undetected. On 8 June a customer pushed an entirely valid configuration change that happened to meet the precise conditions to trigger that dormant bug, and a large majority of Fastly’s network began serving errors. Fastly detected it within about a minute and identified and disabled the offending configuration within roughly an hour, with the network largely recovering after that.

The systemic lesson: valid input can detonate a latent defect. You cannot point only at the customer (their config was legal) or only at the trigger (the bug was already there). The pairing is the hazard — which is why progressive delivery and canary rollouts matter even for changes that “can’t break anything”: something deployed weeks ago may be waiting for the exact input that does. It is also a lesson in concentration risk — when much of the web shares one edge provider, that provider’s one-hour bug is the web’s one-hour bug.

Facebook/Meta — the network that erased itself (4 October 2021)

On 4 October 2021, Facebook, Instagram, WhatsApp, Messenger, and Oculus all vanished from the internet for roughly six hours — not slow, not erroring, simply gone, unreachable and unresolvable. This one is worth understanding mechanically, because it’s a beautiful, terrible cascade.

During routine maintenance, a command intended to assess backbone capacity inadvertently took down all of Facebook’s backbone connections, disconnecting its data centers from each other and the world. An audit tool that should have rejected such a command had a bug and didn’t. Then came the twist that turned an internal problem into a global vanishing act: Facebook’s DNS servers are designed to withdraw their BGP route advertisements if they lose their connection to the data centers (a health-check — “if I can’t reach home, stop telling the internet to send traffic to me”). With the backbone down, the DNS servers concluded they were unhealthy and withdrew Facebook’s routes from BGP. At that point Facebook’s nameservers were unreachable, so no one could even resolve facebook.com — the company had effectively erased itself from the internet’s address book.

   routine command (backbone capacity check)
        │  audit tool bug ──► doesn't block it
        ▼
   backbone connections drop ──► data centers isolated
        │
        ▼
   DNS servers: "can't reach home" ──► withdraw BGP routes (by design)
        │
        ▼
   nameservers unreachable ──► facebook.com can't be RESOLVED, not just reached ──► global outage

By the numbers — and why recovery was slow

The outage ran roughly six hours on 4 October 2021 and took down Facebook’s entire family of apps globally at once. The brutal part was recovery: because so much rode on the same internal network, the tools engineers needed to diagnose and fix it were themselves down, and reports at the time described physical access systems (badge readers) failing, slowing the people who had to reach hardware in person. A surge of failed lookups also hammered DNS resolvers worldwide as devices retried relentlessly. The incident is the textbook example of a circular dependency: the systems you rely on to recover depended on the very thing that was broken.

The systemic lesson: beware automation and dependencies that can sever your ability to recover. A single command, an audit tool that failed open, and a sensible-in-isolation DNS health-check chained into a self-inflicted global outage — and the recovery was slowed because monitoring, access, and tooling all depended on the network that was down. Build out-of-band recovery paths that do not share fate with the system they’re meant to fix. (For the DNS and routing fundamentals underneath this, see DNS and The Network Stack.)

AWS us-east-1 — the region the internet leans on (December 2021)

us-east-1 (Northern Virginia) is AWS’s oldest and busiest region, and — critically — the default home for the control planes of many global AWS services. So when it struggles, the damage isn’t local. On 7 December 2021, an automated scaling activity triggered unexpected behavior in an internal AWS network, producing a surge of connection activity and congestion that impaired a wide range of services for several hours. The same congestion degraded AWS’s own monitoring and operational tooling, which slowed diagnosis — the same circular-dependency theme as Facebook, in a different vendor. (December 2021 actually saw several us-east-1 disruptions; the 7 December one was the most severe.)

The systemic lesson: a control plane is a shared dependency, and “one region” can be a global single point of failure. Many teams that believed they were multi-region still depended on a global service whose control plane lived in us-east-1, so a “regional” event reached them anyway. The defenses are the ones from Multi-Region & DR: know your true dependencies (including the hidden control- plane ones), avoid defaulting everything to a single region, and test that your failover doesn’t itself route through the thing that’s down.

The recurring shape — monitoring that shares fate with the outage

Notice the pattern across Facebook and AWS (and the recovery phase of CrowdStrike): the tools meant to detect, diagnose, and fix the outage depended on the very system that was down. Facebook’s internal tools and badge readers, AWS’s own dashboards — each failed with the thing they were supposed to observe. This is the observability lesson turned inside out: your monitoring, alerting, and break-glass recovery paths must be able to function when your main system can’t. If your status page runs on the infrastructure it reports on, it will go dark exactly when you need it.

Reading them together

Outage	Date	What failed	Systemic lesson
Fastly	8 Jun 2021	Valid config triggered a latent bug; most of the CDN served errors (~1 h)	Valid input can detonate a dormant defect; concentration risk at the edge
Facebook/Meta	4 Oct 2021	Command + failed audit tool → backbone down → DNS withdrew BGP routes (~6 h)	Circular dependencies and automation can sever your ability to recover
AWS us-east-1	7 Dec 2021	Automated scaling → internal network congestion; ops tooling impaired (hours)	A control plane is a global single point of failure; recovery tooling must not share fate
CrowdStrike	19 Jul 2024	Faulty kernel-agent content update → BSOD on ~8.5 M Windows hosts	Content needs rings too — see the dedicated case study

Three throughlines connect them. None was an attack — they were ordinary mechanisms (a latent bug, a routine command, an automated process, a config push) that lacked blast-radius containment. Recovery was the hard part, repeatedly slowed by tooling that shared fate with the failure. And the fixes are boring: staged rollout, out-of-band recovery, honest dependency mapping, and not defaulting everything to one shared thing. The unglamorous controls are the ones that would have helped.

The thread: turning other people’s outages into your guardrails

The manual, error-prone step this page removes is learning reliability only from your own scars — waiting until you take down production to discover that valid input triggers latent bugs, that your recovery tooling shares fate with the outage, or that “multi-region” still leaned on one control plane. A postmortem is reusable: each of these public failures hands you a guardrail you can adopt before you earn it the hard way — canary even “safe” changes, build out-of-band break-glass access, map your hidden dependencies, decentralize the single point everyone leans on.

Production gets safer because each studied outage becomes a check the system enforces, not a lesson a team has to relive. That is the blameless-postmortem philosophy applied across companies: treat the industry’s worst days as free, pre-paid action items. The most detailed of these — a content update that crashed millions of machines and demanded a manual fix on each — gets its own page: Case Study: The CrowdStrike 2024 Outage.

Check your understanding

1. In the Fastly outage, the customer’s config was valid and the bug already existed. Why does the page insist the pairing is the hazard, and what rollout discipline addresses it?
2. Walk through the Facebook/Meta cascade from the maintenance command to facebook.com becoming unresolvable. Why is “unresolvable” worse than “unreachable”?
3. What is the “circular dependency” that appears in both Facebook and AWS, and what general defense does the page prescribe for it?
4. Why can an outage “in one region” (us-east-1) hit teams that believed they were multi-region? Name the concept that makes a single region a global single point of failure.
5. Using the book’s thread, what manual step does studying other companies’ postmortems remove, and how is that the blameless-postmortem idea applied across the whole industry?

Show answers

1. Neither the valid config nor the dormant bug is dangerous alone — the hazard is that a legal input met the exact conditions to trigger a latent defect deployed weeks earlier. Since you can’t rely on inputs being “safe,” progressive/canary rollout limits exposure: a change reaches a small slice first, so a triggered latent bug is caught before it hits the whole network.
2. A routine backbone-capacity command (not blocked by a buggy audit tool) took down the backbone, isolating the data centers; Facebook’s DNS servers withdrew their BGP routes by design when they lost the backbone; with the nameservers’ routes gone, the internet couldn’t even resolve facebook.com. “Unresolvable” is worse than “unreachable” because clients can’t get an address at all — there’s nothing to retry against, and the failure looks total.
3. The recovery tooling depended on the system that was down: Facebook’s internal tools and badge readers, AWS’s own monitoring/operational dashboards. The defense is out-of-band recovery paths and monitoring that don’t share fate with the primary system, so you can still diagnose and fix when it’s broken.
4. Many global AWS services run their control plane in us-east-1, so teams depending on those services were affected even with workloads elsewhere. The concept is the control plane as a shared dependency / single point of failure — “multi-region” data plane doesn’t help if a global control plane is down.
5. It removes having to learn each reliability lesson from your own production outage — instead you adopt the guardrail (canary, out-of-band access, dependency mapping, decentralization) from someone else’s failure first. It’s the blameless-postmortem principle scaled up: treat the industry’s public worst days as free, pre-paid action items that become checks your system enforces.