What happens when a hot key melts your cache

The Problem Statement

Interviewer: "A celebrity with 50 million followers posts a tweet. Within seconds, your Redis cache starts returning errors. Five minutes later, your database is overwhelmed and the entire site is down. Walk me through what happened, step by step, and how you would prevent it."

This question tests whether you understand the single-shard bottleneck problem in distributed caches, how cascading failures propagate from cache to database, and whether you know the real production techniques (key splitting, local caching, read replicas) used by companies like Twitter, Instagram, and DynamoDB to handle hot keys.

I find this is one of the most revealing interview questions because it exposes whether a candidate truly understands distributed systems or just memorizes architecture diagrams. The hot key problem is simple to describe but devastating in production, and the solutions are non-obvious.

Most candidates know "cache the popular stuff," but the hot key problem is what happens when the cache itself becomes the bottleneck. The cache is supposed to protect the database. When a single key gets so many requests that it overwhelms one cache node, the protection fails and the entire system collapses.

Clarifying the Scenario

You: "Let me make sure I understand the scenario correctly."

You: "When you say 'the cache starts returning errors,' are we talking about a Redis cluster where the hot key lives on a single shard, or a standalone Redis instance?"

Interviewer: "Redis Cluster with hash-slot-based sharding."

You: "Got it. And is the failure limited to reads (everyone reading the celebrity's tweet), or are writes also involved (likes, retweets, replies)?"

Interviewer: "Primarily reads, but I want you to mention what happens with writes too."

You: "One more: are we dealing with Redis only, or should I also discuss how this affects key-value stores like DynamoDB?"

Interviewer: "Start with Redis, then briefly mention DynamoDB. I want to see the general pattern."

You: "OK. I will structure my answer in three parts: first, the cascade failure from hot key to full outage. Then, the detection strategies to catch hot keys early. Finally, the mitigation techniques including key splitting, local caches, and read replicas."

My Approach

I break this into five parts:

1. What creates a hot key: Understanding why certain keys attract disproportionate traffic (viral content, celebrity accounts, flash sales, global config keys).
2. The single-shard bottleneck: In any distributed cache using hash-based sharding, one key maps to one shard. All traffic for that key hits a single node.
3. The cascade failure: When the cache node overloads, requests fail. Clients retry. The application falls through to the database. The database gets the full unfiltered traffic and dies.
4. Detection: Monitoring key access frequency, identifying hot keys before they cause damage.
5. Mitigation: Key splitting, local in-process caches, read replicas, and DynamoDB adaptive capacity.

The mental model I keep coming back to is this: a distributed cache is only as strong as its weakest shard. If all traffic concentrates on one shard, you have effectively reduced your entire cache cluster to a single node. The "distributed" part becomes meaningless.

The Architecture

Here is what a normal distributed cache looks like, and what happens when a hot key concentrates all traffic on a single shard.

Here is the step-by-step cascade:

1. The celebrity posts a tweet. The tweet content is cached in Redis under key tweet:12345.
2. Hash slot calculation: CRC16("tweet:12345") % 16384 = slot 7923. This slot lives on Shard 2.
3. Within seconds, 50 million followers start loading their timelines. Each timeline includes this tweet. Each includes a cache read for tweet:12345.
4. Shard 2 goes from 5K requests per second to 200K requests per second. Its CPU hits 100%. Network buffers fill. Latency spikes from 1ms to 500ms.
5. At 500ms latency, application servers start timing out. The Redis client library retries, doubling the load on Shard 2.
6. Cache misses and timeouts trigger the application to fall through to the database. The database was designed for 10K reads per second (because the cache was supposed to absorb 95% of reads). It now receives 200K reads per second.
7. The database connection pool is exhausted within seconds. All queries queue up. The entire application hangs, not just the celebrity's tweet, but every request that needs the database.

This is the "one hot key takes down everything" pattern. The cache was supposed to be the shield, but the shield itself cracked.

The Hot Key Cascade Failure Pattern

Let me trace the full failure timeline to show exactly how fast things go wrong. This is the sequence that every engineer should be able to draw on a whiteboard.

The key insight: the total time from "celebrity posts" to "full outage" can be under 30 seconds. This is why hot key detection and mitigation must be proactive, not reactive. By the time you see the alert, the cascade has already started.

Key Splitting and Local Cache Strategies

Let me show exactly how key splitting works at the implementation level. This is the section that turns a whiteboard answer into a production deployment.

Here is the implementation logic as pseudocode:

SPLIT_FACTOR = 8  // Number of key copies

function readHotKey(keyBase):
    // Layer 1: Check local in-process cache
    localValue = localCache.get(keyBase)
    if localValue != null:
        return localValue

    // Layer 2: Read from Redis with random suffix
    suffix = random(0, SPLIT_FACTOR - 1)
    splitKey = keyBase + ":" + suffix
    value = redis.get(splitKey)

    // Populate local cache with short TTL
    localCache.set(keyBase, value, ttl=2_seconds)
    return value

function writeHotKey(keyBase, value):
    // Write to ALL split keys (fan-out write)
    for i in range(0, SPLIT_FACTOR):
        redis.set(keyBase + ":" + i, value, ttl=3600)
    // Invalidate local caches via pub-sub
    redis.publish("cache:invalidate", keyBase)

The write path is the critical detail most candidates miss. When the celebrity's tweet is updated (edited, deleted, or engagement counts change), you must write to all N split keys. This means writes are amplified by the split factor. For a read-heavy hot key (which is the common case), this tradeoff is overwhelmingly worthwhile: you amplify writes by 8x to reduce the read bottleneck by 8x.

Detecting Hot Keys Before They Cause Damage

The best mitigation is prevention. If you can identify a hot key before it takes down the shard, you can proactively split it or add it to the local cache allow-list.

There are three approaches to detection, and each operates at a different layer:

For DynamoDB, the detection story is different. DynamoDB publishes ThrottledRequests and ConsumedReadCapacityUnits metrics per partition key to CloudWatch. You can set alarms on these metrics. DynamoDB also has "adaptive capacity," which automatically redistributes throughput from cold partitions to hot ones, but it has limits and takes time to kick in.

The Tricky Parts

1. Write amplification with split keys: When the hot key's value changes (tweet edited, engagement count updated), you must write to all N split copies. If N = 8 and the value changes 100 times per second, that is 800 writes per second spread across shards. For read-heavy hot keys this is fine, but for write-heavy hot keys (like a real-time counter), key splitting makes writes worse. The solution for write-heavy hot keys is different: use local counters and periodic aggregation.

2. Stale local cache during burst writes: The local in-process cache has a 1-3 second TTL. During those 1-3 seconds, the app server might serve slightly stale data. For a tweet's text content, this is unacceptable (a deleted tweet must disappear immediately). For engagement counts (likes, retweets), 2 seconds of staleness is fine. The solution: use different TTLs for different data types, and add a pub-sub invalidation channel for critical keys.

3. Unpredictable hot keys: A celebrity tweet is predictable (the celebrity has 50M followers). But sometimes a random user's tweet goes viral due to a retweet cascade. You cannot pre-split keys for unknown-in-advance viral content. The automatic detection system must react within seconds, not minutes.

4. Cascading hot keys: The celebrity's tweet becomes hot, but so does their profile, their avatar image URL, and every parent tweet in a reply thread. One hot entity creates 5-10 hot keys. Your detection system must handle this cluster pattern, not just individual keys.

5. The DynamoDB partition split lag: DynamoDB detects hot partitions and splits them, but the split takes minutes. During those minutes, the partition throttles at its provisioned capacity. If you provisioned 3,000 RCU per partition and a hot key needs 30,000, you get throttled for 90% of requests until the adaptive capacity kicks in.

What Most People Get Wrong

How I Would Communicate This in an Interview

Here is how I would actually say this:

"The hot key problem happens when a single cache key gets so many requests that it overwhelms the cache shard responsible for it. In a Redis Cluster, each key maps to exactly one shard via hash slots. When a celebrity with 50 million followers posts a tweet, that tweet's cache key might get 200K reads per second, all hitting one shard.

The cascade goes like this: the shard CPU maxes out, latency spikes, requests time out, the application retries (making things worse), then falls through to the database. The database was designed for 10K reads per second behind the cache, not 200K raw reads. It gets overwhelmed and the entire site goes down.

I would defend against this with two layers. First, a local in-process cache on every app server with a 2-second TTL. For any key accessed more than once per second on that server, the second and subsequent reads are served from local memory with zero network round-trip. This alone absorbs 95% of the reads.

Second, for keys that are genuinely hot across the cluster, I would use key splitting. Instead of one key, I create eight copies with different suffixes. Each copy lands on a different shard. The application picks a random suffix for each read, spreading the load evenly.

For detection, I would instrument the Redis client to sample 1% of commands and feed key names into a Count-Min Sketch. When any key exceeds 10K reads per second, automatically push it to the local cache allow-list and trigger an alert.

The reason this is hard is that you cannot predict which keys will become hot. A random tweet can go viral. The detection and mitigation must be automatic, not manual."

Interview Cheat Sheet

• Trigger: "hot key" or "single key bottleneck" → Say: "One key maps to one shard. No amount of horizontal scaling helps that specific key. You need key splitting or a local cache tier."
• Trigger: "cache failure cascade" → Say: "Hot shard overloads, timeouts trigger retries (amplifying load), cache falls through to DB, DB connection pool exhausts, full outage within 30 seconds."
• Trigger: "how to split keys" → Say: "Create N copies with different suffixes (tweet:123:0 through tweet:123:7). Read a random copy. Write to all copies. Spreads load across N shards."
• Trigger: "local cache" → Say: "In-process cache (Caffeine, Guava, lru-cache) with 1-3 second TTL on every app server. Absorbs 95%+ of repeat reads with zero network overhead."
• Trigger: "detect hot keys" → Say: "Client-side 1% sampling feeding into a Count-Min Sketch. Detects keys exceeding a threshold within seconds. No Redis server-side impact."
• Trigger: "DynamoDB hot partitions" → Say: "Each partition has a throughput limit. Adaptive capacity redistributes but takes minutes. Pre-provision with on-demand mode or use DAX (DynamoDB Accelerator) as a read cache."
• Trigger: "thundering herd" → Say: "Related but different. Thundering herd is all clients missing cache simultaneously after expiry. Hot key is all clients hitting the same shard even when the cache is present."
• Trigger: "write-heavy hot key" → Say: "Key splitting amplifies writes. For counters, use local in-process counters and flush aggregated totals to Redis periodically (every 1-5 seconds)."
• Trigger: "real-world examples" → Say: "Twitter celebrity tweets, Amazon flash sales (Lightning Deals), Reddit front page posts, Spotify Wrapped launch day, DynamoDB GSI hot partitions."
• Trigger: "monitoring" → Say: "Redis SLOWLOG for symptoms, client-side sampling for detection, per-shard CPU and network metrics for capacity planning."

Test Your Understanding

Quick Recap

1. A hot key is a single cache key that receives so many requests it overwhelms the cache shard responsible for it.
2. In hash-based sharding (Redis Cluster, DynamoDB), one key always maps to one shard. Adding more shards does not help that specific key.
3. The cascade failure progresses in under 30 seconds: shard overloads, timeouts trigger retries (amplifying load), cache falls through to database, database connection pool exhausts, full outage.
4. Key splitting creates N copies of the hot key across different shards, spreading the read load evenly.
5. Local in-process caches (1-3 second TTL) on every app server absorb 95%+ of repeat reads with zero network overhead.
6. Detection uses client-side sampling with Count-Min Sketch to identify hot keys within seconds of their emergence.
7. The thundering herd problem is different: that is about cache expiry causing stampedes to the database. Hot key is about overwhelming a shard even when the cache is present.
8. DynamoDB handles hot partitions with adaptive capacity, but it takes minutes, so proactive measures like DAX or write-sharding are needed.

Related Concepts

• Cache stampede / thundering herd: What happens when a cache key expires and all clients miss simultaneously. Related but distinct from the hot key problem.
• Consistent hashing with virtual nodes: The sharding mechanism that determines which node owns each key. Virtual nodes improve distribution but do not eliminate hot keys.
• Circuit breaker pattern: Protects downstream services from cascading failures. Used here to prevent retry amplification when a Redis shard is overloaded.
• Count-Min Sketch: The probabilistic data structure used for frequency estimation in hot key detection. Bounded error, fixed memory, no false negatives.
• DynamoDB adaptive capacity: DynamoDB's built-in mechanism to redistribute throughput from cold partitions to hot ones. Useful but has latency and limits.