Post-mortem: AWS Kinesis cascade 2020

Incident Summary

Date: November 25, 2020 Duration: ~8 hours of significant customer impact Region: US-East-1 (Northern Virginia) Services affected: Kinesis, CloudWatch, Cognito, Lambda, API Gateway, EventBridge, ECS, Elastic Beanstalk, Personal Health Dashboard, and dozens more Root cause: Capacity addition to Kinesis front-end fleet triggered OS thread limit exhaustion, causing a cascading failure across all services that depend on Kinesis internally

This was not a "Kinesis outage." It was a demonstration of what happens when a foundational service fails and every other service has a hidden dependency on it. I keep coming back to this incident because it teaches more about cascading failures than any textbook example.

The incident also exposed a fundamental assumption in AWS's architecture: that internal platform services (Kinesis, in this case) are always available. When that assumption broke, the blast radius was not proportional to Kinesis's importance. It was proportional to the number of services that had quietly built dependencies on it over the years.

For your interview preparation: this is the canonical example of a cascading failure caused by hidden dependencies. If an interviewer asks "tell me about a real-world cascading failure," this is the one to cite.

What Happened: The Timeline

Here is the minute-by-minute sequence of how a routine capacity operation became an 8-hour multi-service outage.

The total customer impact was roughly 8 hours. But the real story is in the cascade, not the timeline.

Several things stand out about this timeline. The time from trigger (07:44) to customer impact (08:15) was only 31 minutes. The time from first impact to root cause identification (08:15 to 11:00) was nearly 3 hours. And the time from identifying the root cause to full recovery (11:00 to 16:00) was 5 hours. This ratio (fast to break, slow to diagnose, even slower to fix) is characteristic of cascading failures.

The diagnosis delay was not because the engineers were slow. It was because their primary diagnostic tool (CloudWatch) was down. They were trying to fix a system while the system they use to understand system health was itself broken. This is the operational equivalent of trying to find a leak in a dark room when the flashlight batteries are dead.

This timeline distribution is something interviewers look for when you discuss incident response. Engineers who mention only "we found the bug and fixed it" miss the operational reality. A strong answer breaks the timeline into three phases: detection (how fast did you know something was wrong?), diagnosis (how did you find the root cause, and what slowed you down?), and recovery (why did the fix take so long even after you knew the cause?). This incident hits all three phases hard, which is what makes it such a useful reference.

Another subtlety: the 07:44 trigger was not a deployment. It was a capacity addition. In an interview setting, this matters because it demonstrates that operational changes (not just code changes) can cause production outages. Capacity additions, configuration changes, certificate rotations, DNS updates: these operational actions often lack the same safety nets (CI/CD pipelines, automated rollback) that code deployments have. When an interviewer asks "what kinds of changes can cause outages?", the answer should include operational changes, not just code pushes.

The Architecture Before the Incident

Kinesis is a streaming data service: producers push records, Kinesis stores them in shards, consumers pull records. The front-end fleet is the API layer that accepts all Kinesis API calls and routes them to the correct back-end shards.

The critical detail: those "producer" boxes at the top are not just customer workloads. CloudWatch, Cognito, and Lambda all use Kinesis internally to move data. This dependency is invisible to customers and, as we learned, partially invisible to AWS engineers operating those services.

When I say "hidden dependency," I mean something specific. The Kinesis API documentation does not say "CloudWatch requires Kinesis." The CloudWatch documentation does not say "this service will degrade if Kinesis is unavailable." The dependency existed in the implementation but not in the service's operational model. This is the most dangerous kind of dependency: one that is architecturally real but operationally invisible.

In any large organization, these hidden dependencies accumulate over time. A team building CloudWatch's metrics ingestion pipeline five years ago chose Kinesis because it was the best tool for the job. That was a good technical decision. But no one updated the operational runbooks to say "if Kinesis goes down, CloudWatch will be impaired." Over the years, the dependency became infrastructure: invisible, assumed, and untested for failure.

Root Cause: Thread Limit Exhaustion During Capacity Addition

The Kinesis front-end fleet needed additional capacity. AWS operations added new servers to the fleet. This is a routine scaling operation that had been performed many times before.

The problem: the new servers were a larger instance type than the existing fleet. More CPUs per server.

Each front-end server initializes its routing cache at startup. The cache initialization code creates threads proportional to the number of CPU cores available. The formula roughly:

threads_created = num_cpus × threads_per_cpu × num_routing_entries

On the old instance type (fewer CPUs), this formula produced a thread count well within the Linux OS thread limit. On the new instance type (more CPUs), it produced a thread count that exceeded the OS ulimit for maximum threads per process.

To put numbers on this: if the old instance type had 16 CPUs and the initialization created 50 threads per CPU per routing entry, and there were 100 routing entries, that is 16 x 50 x 100 = 80,000 threads. The OS limit may have been set at 100,000, leaving comfortable headroom. The new instance type with 32 CPUs creates 32 x 50 x 100 = 160,000 threads, well past the same 100,000 limit. (These are illustrative numbers; AWS did not publish the exact figures.)

The result: new servers started, began initializing their cache, hit the thread limit, and crashed. But it was worse than that. The operating system thread limit exhaustion did not just affect the new servers. As the fleet tried to redistribute work, the cascading restarts and retry storms pushed existing servers past their thread limits too.

Within minutes, a significant portion of the Kinesis front-end fleet was down. The servers that remained healthy were now handling the load of the entire fleet, which pushed their thread counts higher from increased API handling. Some of those surviving servers crossed their own thread limits and went down too.

Here is the core failure sequence:

The positive feedback loop (crash, redistribute, overload, crash) is the key lesson here. Once enough front-end servers failed, the remaining servers were handling so much traffic that they too approached the thread limit, and some of them failed as well.

This is a classic example of a "cascading resource exhaustion" pattern. The failure of some servers increases the load on the remaining servers, which pushes them closer to their own resource limits. If the added load pushes even one more server past its limit, the cycle accelerates. In the worst case, the entire fleet collapses to zero healthy servers. The math is merciless: if you lose 30% of your fleet and the remaining 70% was already at 60% capacity, the remaining servers are now at ~86% capacity (60% / 0.7). If any of those servers were already closer to 70% capacity, they are now over 100% and start failing too.

The only way to break this cycle is to reduce the total load (turn away traffic) or add capacity that does not trigger the same failure mode. AWS had to do the former first, then carefully do the latter.

The Cascade: How Kinesis Took Down CloudWatch, Cognito, and Lambda

This is where the incident becomes a masterclass in hidden dependencies.

CloudWatch: Amazon CloudWatch uses Kinesis Data Streams internally for metrics ingestion. When customers (or AWS services) publish CloudWatch metrics, those metrics flow through Kinesis before reaching CloudWatch's storage layer. When Kinesis became unavailable, CloudWatch could not ingest new metrics.

The consequence of CloudWatch going down was devastating for the recovery effort. Engineers trying to diagnose the Kinesis problem could not see CloudWatch metrics, alarms, or dashboards. They were flying blind.