What happens when you add a node to a Redis cluster

The Problem Statement

Interviewer: "You have a Redis Cluster with 6 nodes and you need to add a 7th node to handle growing traffic. What actually happens during resharding? How do the keys move between nodes, and what does a client see during the migration?"

This question tests three things: your understanding of Redis Cluster's hash slot model and how data is distributed across nodes, your knowledge of the MIGRATING/IMPORTING state machine that moves slots between nodes without downtime, and whether you can explain how clients handle the ASK and MOVED redirects that occur during and after migration.

Most candidates know that Redis Cluster uses hash slots. The strong answer explains the step-by-step migration protocol: how a slot transitions through states, how individual keys are atomically migrated, and what the client-side experience looks like (including the brief period where reads for migrating keys hit the new node via ASK redirects).

I have seen this question in interviews at companies running large Redis clusters (Stripe, DoorDash, Shopify) and at Redis Labs itself. It separates people who have read the docs from people who have operated a production cluster.

Clarifying the Scenario

You: "Good question. Let me clarify a few things before I walk through the process."

You: "When you say 'add a 7th node,' are we talking about adding a new master that will own some hash slots, or adding a replica for an existing master?"

Interviewer: "A new master. We want to rebalance the data."

You: "Got it. And should I assume this cluster is serving live traffic during the resharding? We are not taking it offline?"

Interviewer: "Correct. The cluster must remain available during the migration."

You: "One more. Are we using Redis's built-in cluster resharding (redis-cli --cluster reshard), or a custom migration tool?"

Interviewer: "The built-in mechanism. I want to understand what happens at the protocol level."

You: "OK. I will structure my answer in four parts: the hash slot model that determines which node owns which keys, the slot-level migration state machine (MIGRATING and IMPORTING states), the key-level migration process (the MIGRATE command), and the client-side redirect behavior (ASK vs MOVED) that keeps queries working during the move."

This scoping is critical. Without it, candidates tend to wave their hands and say "the slots move." The interviewer wants the protocol-level detail.

My Approach

I break this into four parts:

1. The hash slot model: Redis Cluster divides the keyspace into 16,384 hash slots. Every key hashes to one slot (CRC16 mod 16384), and each slot is owned by exactly one master node. This is how the cluster knows where to find any key.
2. Adding the new node: When you join a new master to the cluster, it starts with zero slots. The resharding tool decides which slots to move from existing nodes to the new node. A typical rebalance moves roughly 16384/7 ≈ 2340 slots to the new node.
3. The migration state machine: Slots are migrated one at a time. The source node marks a slot as MIGRATING. The target node marks it as IMPORTING. During this state, both nodes collaborate to serve requests for keys in that slot.
4. Client-side behavior: Clients receive ASK redirects for keys that have already moved to the new node but whose slot is still being migrated. After migration completes, the slot ownership updates and clients receive MOVED redirects (which they cache permanently).

The beauty of this design is that the cluster stays fully available during resharding. No key is ever unreachable. The cost is slightly higher latency for keys in actively migrating slots (due to the redirect hop).

Let me put some numbers on this. Here is the math for a typical rebalance:

This is a rough guide. Real numbers depend on key sizes, network bandwidth, and how much throttling you apply to protect production latency.

One important subtlety: the reshard tool does not move slots from all source nodes simultaneously. It processes one slot at a time (though some tools like redis-cli --cluster rebalance parallelize across source nodes). This sequential approach is intentional. Moving one slot at a time limits the impact: at most one slot on each source node is in the MIGRATING state at any time, so at most one slot's worth of keys experience ASK redirects.

Here is the timeline for a realistic resharding operation:

The "client convergence" phase is often overlooked. Even after the cluster knows the new mapping, some clients still have stale caches. They will get one MOVED redirect per migrated slot on their first request. After that, they are up to date. This is normal and expected.

The Architecture

Here is the high-level flow when you add Node 7:

1. You run redis-cli --cluster add-node to join Node 7 to the cluster. At this point, Node 7 is a master with zero slots. It participates in gossip protocol but owns no data.
2. You run redis-cli --cluster reshard and specify how many slots to move (typically ~2340 for an even 7-way split) and which source nodes to take them from.
3. The resharding tool migrates slots one at a time from the source nodes to Node 7. Each slot migration goes through the MIGRATING/IMPORTING state machine.
4. After all slots are migrated, the cluster is rebalanced. Clients update their slot map cache and route new requests directly to Node 7.

The entire process can take minutes to hours depending on how many keys exist in the migrating slots. The cluster remains fully available throughout.

A key detail that candidates often miss: the redis-cli --cluster reshard command is just a convenience wrapper. Under the hood, it sends plain Redis commands (CLUSTER SETSLOT, CLUSTER GETKEYSINSLOT, MIGRATE) to the cluster nodes. You could do the entire resharding process manually with a script. Understanding the raw commands is what separates "I used the tool" from "I understand the protocol."

Here is the command sequence the reshard tool executes for each slot:

# Step 1: Prepare the target
redis-cli -h target CLUSTER SETSLOT 42 IMPORTING source-node-id

# Step 2: Prepare the source
redis-cli -h source CLUSTER SETSLOT 42 MIGRATING target-node-id

# Step 3: Get keys in slot (batch of 100)
redis-cli -h source CLUSTER GETKEYSINSLOT 42 100

# Step 4: Migrate each key
redis-cli -h source MIGRATE target-host 6379 key 0 5000
# ... repeat for all keys ...

# Step 5: Finalize
redis-cli -h target CLUSTER SETSLOT 42 NODE target-node-id
redis-cli -h source CLUSTER SETSLOT 42 NODE target-node-id
# ... send to all other nodes too ...

Knowing these raw commands is valuable because it lets you script custom migration strategies (like migrating only during off-peak hours or with custom throttling) when the default tool is too aggressive.

The Hash Slot Migration State Machine

This is the core of the resharding process. Each slot migrates through a precise sequence of states that ensures no key is ever lost or unreachable.

Think of it like moving offices. The source office (old node) and the target office (new node) both stay open during the move. Files (keys) are moved one box at a time. If someone calls the old office asking for a file that has already been moved, the old office says "that file is at the new office, try there" (ASK redirect). After all files are moved, the old office updates the building directory to point to the new office permanently (MOVED).

The critical invariant is: every key is reachable at every point during migration. A key is either on the source (and served directly), or on the target (and reached via ASK redirect from the source), or the migration is complete (and reached via MOVED redirect or directly). There is no window where a key is "in transit" and unreachable.

Let me walk through the protocol step by step.

Here is each step in detail:

Step 1: Mark the slot as IMPORTING on the target. The reshard tool sends CLUSTER SETSLOT 42 IMPORTING <source-node-id> to the target node. This tells the target "you are about to receive keys for slot 42. Accept MIGRATE commands and ASKING-prefixed queries for this slot."

Step 2: Mark the slot as MIGRATING on the source. The tool sends CLUSTER SETSLOT 42 MIGRATING <target-node-id> to the source. This tells the source "slot 42 is being moved. If a client asks for a key in this slot that you no longer have, redirect them to the target with an ASK redirect."

Step 3: Get the keys. The tool calls CLUSTER GETKEYSINSLOT 42 100 to get up to 100 keys at a time that belong to slot 42 on the source node.

Step 4: Migrate each key. For each key, the tool sends MIGRATE <target-host> <target-port> <key> 0 5000. The MIGRATE command is atomic: it serializes the key on the source (DUMP), transfers it to the target (RESTORE), and deletes it from the source, all in one operation. If MIGRATE fails, the key remains on the source. No data is lost.

Step 5: Finalize. After all keys have been migrated, the tool sends CLUSTER SETSLOT 42 NODE <target-node-id> to both nodes (and ideally to all nodes in the cluster). This permanently assigns slot 42 to the target. From this point, the slot is no longer in a transitional state.

Here is a timeline of what a single slot migration looks like from start to finish:

The migration duration in phase 2 depends entirely on the number and size of keys in the slot. Empty slots migrate instantly. Slots with 100K small keys take a few minutes.

Client Behavior During ASK Redirects

The client-side handling of redirects during resharding is where most candidates get confused. There are two redirect types, and confusing them breaks your answer.

I'll be direct: if you cannot explain the difference between ASK and MOVED in an interview, you do not understand Redis Cluster resharding at a level worth discussing. This is the core of the protocol, and it is what the interviewer is testing for.

The fundamental distinction is about intent and duration:

• ASK says "this one key has already moved, but the slot migration is still in progress."
• MOVED says "this slot has permanently moved. Update your records and do not come back here for this slot."

This distinction matters because it prevents the client's slot map cache from thrashing during a migration that moves keys one at a time over minutes.

The two redirect types have fundamentally different meanings:

The ASKING command is crucial. Before the client sends the actual GET/SET to the target node, it must first send ASKING. This tells the target node "I know this slot is in IMPORTING state and you do not officially own it yet, but I was redirected here by the source." Without ASKING, the target node would reject the request because its slot map says it does not own slot 42.

My recommendation for your interview: always explain the difference between ASK and MOVED. It is the detail that separates "I read a blog post" from "I understand the protocol."

Here is what the protocol looks like on the wire for a single key access during migration:

# Client sends GET to source (Node A) for a migrated key
Client -> Node A: GET user:1001
Node A -> Client: -ASK 42 10.0.0.7:6379

# Client follows ASK redirect to target (Node B)
Client -> Node B: ASKING
Node B -> Client: +OK
Client -> Node B: GET user:1001
Node B -> Client: $5\r\nhello

# Client does NOT update its slot map
# Next request for a different key in slot 42 still goes to Node A

Compare this to what happens after migration is complete:

# Client sends GET to stale node (Node A) after migration finished
Client -> Node A: GET user:1003
Node A -> Client: -MOVED 42 10.0.0.7:6379

# Client updates slot map: slot 42 -> Node B
# All future requests for slot 42 go directly to Node B
Client -> Node B: GET user:1003
Node B -> Client: $5\r\nworld

Notice the asymmetry: ASK requires the extra ASKING prefix and does not update the slot map. MOVED is a permanent reroute with no prefix needed.

Performance Impact and Live Migration Throttling

Resharding is not free. Every key migration consumes CPU (serialization), network bandwidth (data transfer), and memory (temporary duplication during MIGRATE). In production, you need to control the migration speed to avoid degrading the cluster's latency for normal traffic.

I have seen teams run an unthrottled reshard during peak hours and watch their p99 spike from 1ms to 200ms. The fix was simple: add throttling. But the damage (SLA violation, angry downstream services) was already done. Always reshard during low-traffic windows, and always monitor latency in real time during the migration.

The cost model breaks down into three parts:

1. CPU cost: Serializing (DUMP) and deserializing (RESTORE) each key. For complex data structures like sorted sets with 100K members, this is significant.
2. Network cost: Transferring the serialized payload between nodes. For intra-datacenter migration, this is usually sub-millisecond per key for small keys.
3. Event loop blocking cost: The MIGRATE command runs synchronously. While it is executing, the source node cannot process any other commands. This is the most dangerous cost because it directly impacts all other clients.

The MIGRATE command blocks the Redis event loop on the source node for the duration of the key transfer. For small keys (< 1 KB), this is sub-millisecond. For large keys (10 MB+), this can block for hundreds of milliseconds, during which the source node cannot serve any other requests.

Key performance facts:

The Tricky Parts

1. Slot migration is not atomic across the cluster: When the reshard tool sends CLUSTER SETSLOT NODE to finalize the migration, it sends it to each node individually. During the brief window where some nodes have updated their slot map and others have not, different nodes disagree about who owns the slot. This is resolved by gossip protocol propagation, but it means clients connected to different nodes might get different redirect responses for a few seconds.

2. Race condition: write to source after key migrated: A client sends a SET for a key in a MIGRATING slot. The key has already been migrated to the target. The source sees the SET, notices the key is not local, and sends an ASK redirect. But if the client had a stale connection and did not receive the redirect, the write could be lost. Smart client libraries handle this by always retrying ASK redirects, but naive clients (like a raw TCP connection) can lose writes.

3. Large key blocking: The MIGRATE command blocks the Redis event loop. A single 100 MB sorted set takes hundreds of milliseconds to serialize and transfer. During this time, the source node is completely unresponsive. There is no built-in mechanism to stream large keys incrementally. The only mitigation is to avoid large keys or migrate them during off-peak hours.

4. Replica lag during migration: If the source node has replicas, they replicate the DEL commands (key deleted after migration) but not the MIGRATE itself. The replica's dataset diverges briefly. If the source fails during migration and a replica is promoted, the promoted replica might have keys that were already migrated to the target, causing duplicates that are resolved by the cluster's conflict detection but can confuse clients temporarily.

5. Client library maturity matters: Not all Redis client libraries handle ASK redirects correctly. Some treat ASK like MOVED (updating the slot map prematurely). Some do not send the ASKING prefix. Some do not retry at all. In production, verify that your client library (Jedis, Lettuce, ioredis, redis-py) handles the full ASK protocol correctly before resharding live traffic.

Here is a compatibility snapshot for common client libraries:

If you are using a client library that does not handle ASK correctly, you have three options: upgrade the library, add a wrapper that handles ASK manually, or accept degraded performance during resharding (requests for migrated keys will fail until migration completes and MOVED kicks in).

6. Multi-key operations during migration: Commands like MGET or MSET that span multiple keys can fail if some keys in the batch have migrated and others have not. Redis returns a CROSSSLOT error if the keys hash to different slots, but within the same slot, some keys might be on the source and some on the target. The source cannot atomically redirect for some keys and serve others. The client must fall back to individual GET/SET commands for keys in migrating slots.

Here is the impact summary for different workload patterns:

7. Hash tags and co-located keys: Keys with hash tags like {user:1001}.profile and {user:1001}.sessions hash to the same slot (only the tagged portion is hashed). During migration, these keys move together as part of the slot. This is actually a feature: co-located keys stay co-located after resharding. But it also means hash-tagged namespaces create "fat slots" with many keys, which take longer to migrate.

Here is a pre-resharding checklist I use in production:

What Most People Get Wrong

How I Would Communicate This in an Interview

Here is how I would actually say this:

"Redis Cluster distributes data across nodes using 16,384 hash slots. Every key hashes to one slot via CRC16 mod 16384, and each slot is owned by exactly one master node.

When you add a new master, it joins with zero slots. The resharding process moves slots from existing nodes to the new one. This happens live, with no downtime.

Each slot migration works like this. The source marks the slot as MIGRATING, and the target marks it as IMPORTING. Then the reshard tool iterates through every key in that slot and runs MIGRATE, which atomically serializes the key on the source, sends it to the target, and deletes it from the source.

During migration, clients that hit the source for an already-migrated key get an ASK redirect. They follow the redirect to the target node, prefixed with an ASKING command, and get the value. Crucially, they do not update their slot map on ASK. Only when migration completes and they receive a MOVED redirect do they permanently update the mapping.

The main performance concern is that MIGRATE blocks the Redis event loop. Small keys migrate in sub-millisecond time, but large keys (10 MB+) can block for hundreds of milliseconds. In production, you want to throttle migration speed and monitor p99 latency to avoid impacting normal traffic.

After all slots are migrated, you send CLUSTER SETSLOT NODE to finalize, the cluster gossip propagates the new mapping, and the new node is fully operational."

I usually pause here and ask if the interviewer wants me to go deeper on any specific part. The answer is almost always "tell me more about the ASK redirect" or "what about large keys." Having both deep dives ready shows you have operational depth, not just theoretical knowledge.

Here is a condensed version for time-constrained interviews (30 seconds):

"Redis Cluster has 16,384 hash slots. Adding a node means moving some slots. Each slot migrates one key at a time using MIGRATE, which is atomic. During migration, clients get ASK redirects for already-moved keys, which is temporary, and MOVED redirects after completion, which is permanent. The cluster stays fully available throughout. The main risk is large keys blocking the event loop during MIGRATE."

That is 6 sentences. It covers the model, the mechanism, the client experience, and the risk. Enough for a first pass, and detailed enough to prompt follow-up questions that you are prepared to answer.

Interview Cheat Sheet

• "How does Redis Cluster shard data?" → 16,384 hash slots. CRC16(key) mod 16384 maps every key to a slot. Each slot is owned by one master.
• "What happens when you add a node?" → New node joins with zero slots. Reshard tool moves slots from existing nodes. Keys migrate one at a time via MIGRATE.
• "What is the MIGRATING state?" → Source node marks a slot as MIGRATING. It still serves keys it has locally but sends ASK redirects for keys already moved.
• "What is the IMPORTING state?" → Target node marks a slot as IMPORTING. It accepts MIGRATE commands and ASKING-prefixed client queries for that slot.
• "ASK vs MOVED?" → ASK is temporary (follow redirect, do not cache). MOVED is permanent (update slot map forever). Confusing them causes routing thrash.
• "How does MIGRATE work?" → Atomic: DUMP on source, network transfer, RESTORE on target, DEL on source. Blocks event loop for the duration.
• "What about large keys?" → MIGRATE blocks event loop. 10 MB key can block for 100ms+. Pre-scan for large keys and migrate them during off-peak.
• "Is there downtime?" → No. The cluster serves all requests during migration. Migrating keys cost one extra redirect hop (ASK). Non-migrating keys are unaffected.
• "How does the client know where to send requests?" → Client caches slot-to-node mapping. Updates on MOVED. Follows ASK without caching. Refreshes full map periodically.
• "What finalizes the migration?" → CLUSTER SETSLOT NODE sent to all nodes. Gossip protocol propagates. Slot permanently assigned to new owner.

Test Your Understanding

Quick Recap

1. Redis Cluster uses 16,384 hash slots (CRC16 mod 16384) to map every key to a slot, and each slot is owned by exactly one master node.
2. Adding a new node starts with zero slots. The reshard tool moves slots from existing masters to the new one, key by key.
3. Each migrating slot enters MIGRATING state on the source and IMPORTING state on the target, allowing both nodes to serve requests collaboratively.
4. The MIGRATE command is atomic (DUMP + transfer + RESTORE + DEL) but blocking. Large keys block the Redis event loop and can cause p99 spikes.
5. ASK redirects are temporary (do not cache the routing change) while MOVED redirects are permanent (update the slot map cache). Confusing them causes routing thrash.
6. Clients must send an ASKING command before querying the IMPORTING target node, otherwise the target rejects the request.
7. Migration throttling is essential in production: monitor p99 latency, limit batch sizes, and pre-scan for large keys.
8. After migration completes, CLUSTER SETSLOT NODE finalizes the slot assignment and gossip protocol propagates the update cluster-wide.

Related Concepts

• How Redis works internally: Covers the single-threaded event loop, data structures, and persistence that underpin the MIGRATE command's behavior during resharding.
• How the hot key problem happens: Hash slot distribution does not guarantee traffic distribution. Hot keys can overload a single node even with perfectly balanced slots.
• How database failover works: Redis Cluster uses epoch-based leader election for replica promotion, directly relevant to what happens when a source node fails mid-resharding.
• How connection draining works: The concept of gracefully migrating traffic away from a node while keeping it available mirrors the MIGRATING/IMPORTING state machine in Redis Cluster.
• How the etl pipeline works: Large-scale data migration patterns (batching, throttling, checkpointing) apply directly to Redis resharding strategies for clusters with millions of keys.