AWS US-East-1 cascade failure patterns

TL;DR

• US-East-1 (N. Virginia) is AWS's oldest and most feature-rich region, meaning it ships new services first and encounters novel failure modes before anyone else.
• Three major outages (2011 EBS, 2017 S3, 2020 Kinesis) all followed the same cascade pattern: trigger, retry storm, 10x load amplification, dependent service failures.
• The 2020 Kinesis outage took down Cognito (auth) and CloudWatch (monitoring) because both depend on Kinesis internally, leaving teams blind and locked out during the incident.
• AWS's own status dashboard went dark during the 2017 S3 outage because the dashboard itself was hosted on S3.
• Building for "US-East-1 is down" requires multi-region active-active with pre-provisioned capacity and locally cached control plane data. Cold failover takes 30+ minutes, and that is 30 minutes too many.

---

What Happened

US-East-1 is not just another AWS region. It was the first region Amazon launched, it hosts more services than any other region, and it receives new feature launches before the rest of the fleet. That "first and biggest" status also means US-East-1 carries unique operational risk.

Between 2011 and 2020, US-East-1 suffered three high-profile cascading failures. Each one followed a recognizable pattern, yet each one exploited a different dependency chain that nobody had fully mapped.

I've worked with teams that picked US-East-1 because "it has everything" without considering that "everything" includes a higher concentration of correlated failure risk. The incidents below explain why that matters.

Each outage was triggered by something different: a network change, a typo, a capacity addition. But the amplification mechanism was always the same. Retries without backpressure, recovery mechanisms without resource budgets, and dependencies that nobody had mapped until they all failed at once.

2011: EBS Storage Network Collapse

A routine network change in one Availability Zone caused EBS volumes to lose contact with their replica mirrors. EBS's automatic safety mechanism (re-mirroring) tried to rebuild those replicas immediately. The problem: re-mirroring consumed the exact same network and I/O resources that were already degraded.

The more volumes that lost their mirrors, the more re-mirroring traffic flooded the network, which caused more volumes to lose their mirrors. A self-amplifying loop that ran for roughly 12 hours before manual intervention broke the cycle.

2017: S3 Increased Error Rates

An engineer was debugging S3's billing system and executed a playbook command to remove a small set of servers. The command's input parameter had no upper bounds validation. The engineer accidentally specified a value that removed a critical mass of servers from two S3 subsystems.

S3 is so deeply embedded in the AWS ecosystem that its failure cascaded to Lambda, ECS, EC2 instance launches, and even AWS's own health dashboard. AWS literally could not tell customers what was happening because the tool they use to communicate was down. They resorted to posting updates on Twitter.

2020: Kinesis Thread Limit Cascade

A routine capacity addition to Kinesis Data Streams in US-East-1 triggered an unexpected failure. The newly added front-end servers exceeded operating system thread count limits during their initialization. This caused a partial failure of the Kinesis front-end fleet.

Here is where the cascade gets interesting. Kinesis is not just a customer-facing service. Internally, AWS uses Kinesis as the event streaming backbone for CloudWatch (metrics, logs, alarms) and Cognito (authentication events, user pool operations). When Kinesis went down, it dragged both of these services with it.

I still remember the chaos reports from that morning. Teams could not authenticate users (Cognito down), could not see their dashboards (CloudWatch down), and could not figure out why because the monitoring tool was the thing that was broken.

---

How the System Worked Before

AWS regions are divided into Availability Zones (AZs), each with independent power, cooling, and networking. The design promise: a failure in AZ-a should not take down AZ-b.

AZ isolation works well for compute and storage faults. But the control plane services (S3, Kinesis, CloudWatch, Cognito, IAM) operate at the regional level, not the AZ level. A control plane failure affects every AZ in the region simultaneously.

This is the architectural gap that each US-East-1 outage exploited. The data plane was designed for AZ isolation. The control plane was not.

Why US-East-1 Specifically?

US-East-1 is not inherently less reliable than other regions. It fails more visibly for three reasons:

1. Scale. More customers, more traffic, more services. The same bug that goes unnoticed in ap-southeast-2 triggers a cascade in US-East-1 because the fleet is 5-10x larger.
2. New service launches. US-East-1 gets features first. New code means new bugs. Services running their first month in production are more likely to hit edge cases.
3. Customer concentration. Many companies default to US-East-1 because it was the only option when they started on AWS. They never migrated. This means a US-East-1 outage affects a disproportionate share of the internet.

The lesson is not "avoid US-East-1." The lesson is: build your architecture assuming any single region can have a bad day, and US-East-1 has the most data points proving that it will.

---

The Failure Cascade

Every major US-East-1 outage followed the same five-stage cascade pattern. The trigger differs, but the amplification mechanism is identical.