How Slack delivers messages in real time

The Problem Statement

Interviewer: "You open Slack and send a message in a channel with 10,000 members. Walk me through exactly what happens from the moment you press Enter to when every member sees that message in their channel."

This question tests three things: your understanding of persistent connection protocols (WebSockets), how fan-out works at scale when one message must reach thousands of recipients, and whether you understand the ordering and consistency guarantees a messaging system needs. Most candidates describe a basic pub/sub model and stop there. Strong candidates talk about connection management, edge caching, message ordering, and graceful degradation.

The best answers also acknowledge trade-offs explicitly: why at-least-once delivery is acceptable here, why different event types get different delivery guarantees, and what breaks at 10K-member channel scale that does not break at 10-member channel scale. I find this one of the most layered everyday system questions because Slack looks deceptively simple on the surface.

The question is also a good test of how you think about the boundary between "message sent" and "message delivered." In most HTTP APIs, a 200 response means the work is done. In a real-time messaging system, a 200 from the Message Service means the message is persisted, but delivery to recipients is asynchronous and best-effort. The interviewer wants to see whether you understand that distinction and can articulate when it matters.

Clarifying the Scenario

You: "Before I dive into the architecture, I want to clarify the scope."

You: "When you say 'delivers messages in real time,' are we focused on the push delivery path over WebSocket, or should I also cover how Slack handles offline users who open the app later?"

Interviewer: "Cover both, but focus primarily on the real-time push path."

You: "Got it. Should I assume we are talking about Slack's workspace model where a single workspace can have thousands of channels, and some channels have 10K-plus members?"

Interviewer: "Yes, exactly."

You: "One more clarification: should I address what happens when the underlying WebSocket connection is unavailable, for example in corporate environments that block the protocol?"

Interviewer: "Yes, briefly touch on that."

You: "OK. I will structure my answer in four parts: the WebSocket connection lifecycle, the message persistence and ordering layer, the channel fan-out mechanism, and what happens when connections drop or WebSocket is unavailable."

I would also ask whether file uploads need to be treated like messages for delivery purposes, since large files go to a separate CDN pipeline and only a metadata event is fanned out. Lastly, I would clarify whether "real-time" means sub-second or sub-100ms, because those two requirements lead to different Gateway fleet sizing decisions.

My Approach

I break this into five layers, each solving a distinct problem:

1. Connection management: How each Slack client maintains a persistent WebSocket connection through the Gateway Service
2. Message ingestion: How a sent message gets persisted to the database and assigned a sequence number
3. Channel fan-out: How one message reaches all members of a channel, whether that is 3 people or 10,000
4. Edge caching with Flannel: How Slack's edge cache reduces backend load for channel metadata and recent messages
5. Degradation and recovery: What happens when the WebSocket drops, and how the client catches up on missed messages

The mental model I use: imagine a post office (Message Service) that receives a letter, stamps it with a sequence number, puts it in a filing cabinet (MySQL), then hands it to a delivery truck (Fan-out Service). The truck does not drive to every house individually. It drops letters at neighborhood sorting centers (Gateway instances), and each center delivers locally. If someone was not home (offline), they get a slip in the mailbox telling them to pick it up later.

Not all events in Slack use the same delivery pipeline. This table shows how the four main event types differ:

Understanding this table is essentially understanding the entire architecture. Durable events get the full pipeline. Ephemeral events skip it entirely.

At Slack's scale, the distinction between synchronous and asynchronous steps matters enormously. The message persist step (MySQL write) is synchronous: the sender's client does not show the "message sent" confirmation until the write succeeds. The fan-out step is asynchronous: delivery to recipients begins immediately after the write but the sender does not wait for every recipient to receive it. This asymmetry is a deliberate design choice. Making fan-out synchronous would mean a message to a 10K-member channel takes as long as the slowest recipient connection, transforming a 200ms operation into a seconds-long one.

The Architecture

Here is the message lifecycle walkthrough:

Step 1: Client sends message. The client sends the message payload over its existing WebSocket connection to the Gateway Service. The payload includes the channel ID, message text, and a client-generated nonce for deduplication.

Step 2: Gateway routes to Message Service. The Gateway authenticates the request and forwards the message to the Message Service.

Step 3: Message Service persists. The Message Service validates permissions, then writes the message to MySQL/Vitess. The database assigns a monotonically increasing per-channel sequence number, the source of truth for ordering.

Step 4: Fan-out begins. The Fan-out Service looks up channel membership from Memcached, determines which Gateway instance holds each member's WebSocket connection, then batches recipients by Gateway instance and sends one batch push per Gateway rather than one push per user.

Step 5: Gateway pushes to clients. Each Gateway iterates its local connection map and pushes the message down the WebSocket. The client inserts the message at the correct position using the sequence number.

Step 6: Offline handling. For members who are not connected, the Fan-out Service enqueues a job to send a push notification. When they reconnect, the client fetches missed messages via Flannel.

The Gateway Service is not a single server. It is a fleet of hundreds of servers, each holding a fraction of the total WebSocket connections. Each Gateway server can hold roughly 100K-500K concurrent connections before memory becomes the bottleneck. Each connection consumes approximately 20-50 kilobytes of state. At 10 million concurrent connections during peak hours, Slack needs a Gateway fleet of at least 20-100 servers. The connection registry in Memcached rather than on the Gateway itself makes the fleet stateless and safely replaceable.

The Presence Service is a separate component that deserves mention. It manages online/offline state and typing indicators. When a user starts typing, the client sends a presence event to the Gateway, which routes it to the Presence Service. The Presence Service fans these out to relevant channel members with a much looser SLA than messages. A 1-second delay on seeing someone's typing indicator is unnoticeable, so the Presence Service batches and throttles its fan-out aggressively, coalescing multiple state changes into a single delivery. This architectural separation keeps presence-related traffic from competing with message delivery on the same pipeline.

At Slack's operating scale, these design choices are not theoretical. Slack serves tens of millions of daily active users with hundreds of thousands of messages sent per minute during peak hours. Every architectural decision in this system is a decision about how to serve those numbers reliably. Understanding which component is the bottleneck for a given scenario is what distinguishes a strong answer from a weak one. For connection count problems, the Gateway fleet is the constraint. For ordering problems, the database write serialization is the constraint. For reconnection storms, Flannel's cache depth is the constraint.

Deep Dive 1: Fan-Out at Scale

The hardest part of Slack's architecture is not receiving a message or persisting it. It is delivering one message to 10,000 channel members within 200ms. This is the fan-out problem, and how you solve it determines your system's scalability ceiling.

If a Slack workspace has 10,000 members and 5 percent are online at any time, that is 500 active connections to push to when a message is sent. At 100 messages per hour in a busy channel, that is 50,000 delivery events per hour from a single channel. A large company might have 50 such channels. That is 2.5 million delivery events per hour from channels alone, before counting DMs.

Understanding why fan-out is the dominant cost in this system requires thinking about ratios. A message write is one database record. A message fan-out to a 10K-member channel with 30% online generates 3,000 delivery events. The ratio of delivery work to write work is 3,000:1. This means the fan-out tier needs to be much more heavily scaled than the write tier. I find candidates underestimate this ratio consistently. They think of "sending a message" as a single operation. It is actually one write plus N deliveries, and N can be in the thousands.

The fan-out is parallelized across Gateway instances. The Fan-out Service groups recipients by their Gateway instance and sends one batch message per Gateway, which then distributes locally. This is the difference between O(members) network calls and O(gateway_instances) network calls.

The workspace is the natural isolation boundary for the entire fan-out system. Every channel in a workspace lives in the same Vitess shard and all membership lookups stay workspace-local. The Fan-out Service never needs to cross workspace boundaries to resolve a delivery. Workspace-level sharding is not just a performance optimization; it is a tenant isolation guarantee.

There is a subtlety I have seen trip up experienced engineers. The fan-out for a DM (2 people) and the fan-out for a 10K-member channel go through the same Fan-out Service code, but the cost profiles are completely different. A DM is one lookup and one push. A large channel is a full member enumeration, a grouping step, and dozens of batch pushes. Slack optimizes by checking channel size first: channels under roughly 50 members go direct, channels above 1K go pub/sub, and everything in between gets the grouped-batch treatment. Knowing this threshold when answering questions about Slack's fan-out shows that you understand the implementation, not just the concept.

The subscription management in the pub/sub tier requires care at scale. When a user opens Slack and joins their active channels, the Gateway for that user's connection subscribes to the pub/sub topics for those channels. When the user goes idle or disconnects, those subscriptions are removed. The latency of joining a new channel (in large organizations where channel joins are common) needs to be fast enough that the user does not miss messages that arrive between "join channel" and "subscription active." Slack handles this with a sequenced join: the user is subscribed to the channel's pub/sub topic first, then shown the channel's message history. If any messages arrive between the two steps, they are caught by the catch-up protocol the next time the client syncs its cursors.

Deep Dive 2: Message Ordering and the ts Field

Two users in the same channel send messages at nearly the same instant. Which message appears first? If different recipients see different orderings, the conversation becomes incoherent. This is why Slack assigns per-channel sequence numbers at the database level, not at the application level.

Every channel has a monotonically increasing counter in MySQL/Vitess. When a message is persisted, it gets the next sequence number atomically. Message ordering is determined by the order of database writes, not by the order of WebSocket arrivals at the Gateway. Two messages sent 5ms apart always appear in the same order for every recipient, everywhere.

Flannel is Slack's edge cache between clients and the backend. It caches recent messages per channel, channel metadata, and user presence data. When a client reconnects after a brief disconnect, it hits Flannel instead of the database. This absorbs reconnection storms where thousands of clients disconnect simultaneously and would otherwise slam MySQL.

The ts field in the Slack API appears as a decimal string like 1512085950.000216. The integer portion is a Unix timestamp in seconds and the fractional portion is a microsecond-precision counter making the identifier unique within the channel. Flannel indexes messages by ts range to support the exact query reconnecting clients need: "give me all messages in channel C where ts is greater than X." Because ts values sort correctly via lexicographic string comparison, Flannel answers these range queries directly from its in-memory index in under a millisecond.

The ts field format is cleverly designed for multiple purposes simultaneously. The integer part gives human-readable timestamps that support time-based queries ("show me messages from this week"). The fractional part is a counter that makes each identifier unique even when multiple messages arrive within the same second. The combined value sorts lexicographically as a string, which means sorted string comparisons give you chronological ordering without any numeric parsing. Flannel can do range scans on ts strings directly, which keeps the indexing logic simple and fast. This is a tight engineering decision where one field design solves three problems: uniqueness, chronological ordering, and efficient range queries.

Another dimension: the ts value is also the public identifier for a message in the Slack API. When clients send reactions, replies, or edit events, they reference the original message by its ts value. This means the ts is also used as a foreign key in other tables (reactions, thread replies). The format was chosen to be human-readable (the Unix timestamp part) while still being unique enough to serve as a primary key without a separate auto-increment column.

A question I often get asked here: why MySQL rather than Cassandra or Redis? Cassandra handles high write throughput but does not give easily serialized sequence numbers within a partition. Redis could store sequences in memory but loses durability without careful AOF configuration. MySQL with Vitess gives a serialized write order within each shard, strong transactional writes, and efficient range queries by sequence number.

Flannel is worth dwelling on because it solves a problem that is easy to underestimate. During the 3 seconds following a network blip that disconnects 500K clients, those 500K clients all reconnect and issue catch-up requests for the same high-traffic channels. Without Flannel, those requests would all hit MySQL simultaneously. A channel with 500K members would generate 500K identical queries within a 3-second window. Flannel collapses those queries into cache reads. The database sees one update per message written to Flannel's cache, not one query per reconnecting client.

Flannel also handles the case where a reconnecting client has been away for a while. If the client's last-known sequence number is recent enough that Flannel's cache covers it, the catch-up is served from memory. If the client was disconnected for hours (say, a laptop that was closed overnight), Flannel's cache may not reach that far back. In that case the client falls through to MySQL via the Message Service REST API. This fall-through is handled transparently: the client does not need to know whether the response came from Flannel or MySQL, because both return the same format indexed by ts.

Deep Dive 3: Connection Resilience

WebSocket connections break constantly. The user switches from WiFi to cellular, closes their laptop lid, or walks into a dead zone. Slack treats disconnection as a normal operating state, not an error condition, and builds the entire recovery path around that assumption.

Every connection goes through three phases: establishment, active use, and recovery. When a client connects, it upgrades from HTTP to WSS, authenticates, and registers its user-to-gateway mapping in Memcached. This registration allows the Fan-out Service to route messages to the correct Gateway instance. When the connection drops, the Gateway removes the registration.

The reconnection window is actually the most interesting operational challenge in this system. A "brief" disconnect of 3-10 seconds is extremely common (mobile network handoff, WiFi dead zone, laptop sleep). During that window, 10 messages might arrive in an active channel. The reconnecting client needs all 10, in order, delivered instantly on reconnect. The catch-up must be faster than the user's perception of delay. If catch-up takes 2 seconds, the user sees a 2-second frozen moment followed by an instant scroll of 10 messages. If catch-up takes 100ms, the reconnected experience feels seamless. Flannel's sub-millisecond in-memory response is what makes the seamless experience possible.

The backoff strategy is also non-trivial. If 500K clients all disconnect at the same time (network partition), and they all retry at 1 second, you get a thundering herd. Slack adds jitter to the backoff interval: instead of exactly 1s/2s/4s, each client waits 1s +/- 100ms, 2s +/- 200ms, and so on. This spreads the reconnection traffic over time and prevents the Gateway fleet from being overwhelmed by a synchronized reconnection storm.

The catch-up request sends per-channel sequence cursors, not wall-clock timestamps. This makes the catch-up bounded and fast for short disconnects. If Flannel's cache does not cover the gap (the disconnect was too long), the request falls through to the Message Service which reads from MySQL directly.

At-least-once delivery and deduplication

Slack guarantees at-least-once delivery, not exactly-once. Exactly-once would require distributed transactions across the write, fan-out, and acknowledgement layers, adding latency to every message send.

When you press Enter, the client generates a random nonce (UUID4) and includes it with the message. If the WebSocket drops before the server ACKs the delivery, the client does not know whether the write succeeded. On reconnect, the client retries with the same nonce. The server checks the nonce before persisting: if it already exists, return the existing message; if not, persist the new one.

The at-least-once guarantee also handles a subtlety in cross-datacenter delivery. When a message is written in datacenter A, it is replicated asynchronously to datacenter B. If a client in datacenter B is connected to a Gateway in datacenter B, that Gateway's Fan-out Service might not see the message until replication completes (50-100ms). The client might therefore receive the message slightly after a client in datacenter A. This is expected behavior, not a bug. The sequence number ensures both clients show the messages in the correct order even if one received them slightly later.

The deduplication store for nonces needs to be per-channel, not global. A nonce "abc123" in #engineering and a nonce "abc123" in #general are different operations (though a UUID4 nonce will never collide in practice). More importantly, the nonce dedup store needs to be durable enough to survive a short server restart. Slack stores nonces in the same MySQL shard as the channel, so a successful write to the channel table is automatically paired with a durable nonce record.

Graceful degradation: long-poll fallback

Some enterprise networks block WebSocket upgrades at the proxy layer. Slack falls back to HTTP long polling in these environments. The client issues an HTTP request that the server holds open until a message is ready, then responds and the client immediately re-issues the next request. Long-poll delivery latency increases to 100-500ms versus sub-50ms for WebSocket push. The Fan-out and Message Service layers are identical for both transports.

Typing indicators are completely separate from the message pipeline. They are ephemeral, never persisted, never assigned sequence numbers, and never included in catch-up responses. If you miss a typing indicator because your connection dropped for 2 seconds, nothing happens. Showing a stale "is typing" from 5 seconds ago would be worse than showing nothing.

The catch-up strategy is lazily scoped by channel. The client sends its per-channel cursor for a bounded set of recently active channels, not all 200 channels it belongs to. If the user is in 200 channels but only 15 had activity during a 3-second disconnect, the catch-up response covers those 15.

The Tricky Parts

Nine tricky parts might seem like a lot, but that density reflects how mature the Slack architecture is. Each item exists because Slack encountered a real production failure or scaling wall and made a deliberate engineering choice to solve it. Telling the interviewer that you can name nine distinct edge cases signals that you understand the system as it operates in production, not just as it appears in a whiteboard overview. When you mention these trade-offs unprompted, the interviewer does not need to prompt with "but what about...?" questions, which is exactly the signal a senior-level candidate should project.

1. Thundering herd on reconnection: When Slack's infrastructure has a brief blip, millions of clients disconnect simultaneously and reconnect within seconds. Flannel absorbs the catch-up query spike by serving cached results, and the Gateway rate-limits reconnections to accept them in controlled waves rather than all at once.

2. Hot channels: A channel like #general in a 50,000-person company generates massive fan-out for every message. Slack throttles delivery for very large channels, accepting slightly higher latency (500ms instead of 200ms) in exchange for not overloading the fan-out tier.

3. Message deduplication across retries: Network retries can cause the same message to be sent twice. The client includes a nonce with each message. The server deduplicates on this nonce before persisting. Without this, a flaky connection could cause duplicate messages to appear in the channel.

4. Cross-datacenter delivery: Slack runs across multiple datacenters. A message sent by a user connected to datacenter A needs to reach users connected to datacenter B. The message is persisted in the primary database, then replicated asynchronously. Fan-out in each datacenter reads from its local replica, so cross-DC delivery has slightly higher latency (50-100ms more).

5. Workspace sharding and hot shards: Each workspace is assigned to a database shard. All channels and messages for a workspace live in the same shard. A single extremely active workspace can become a hot shard. Slack mitigates this with Vitess, which supports dynamic shard splitting without downtime.

6. Message editing and deletion: When a user edits or deletes a message, the system must push an update event to all recipients who already received the original. This reuses the same fan-out pipeline with an "update" or "delete" event type rather than a "message" event type.

7. Gateway rolling deploys: Deploying a new version of the Gateway fleet without dropping connections requires graceful drain mode. Before an instance is taken out of service, it stops accepting new connections but keeps existing ones alive. Over a 30-60 second window, clients naturally reconnect via heartbeat timeouts and land on healthy instances.

8. Channel subscription scope on the Gateway: For the pub/sub model to work, each Gateway needs to know which of its locally connected users belong to which channels. When a user connects, the Gateway loads their active channel subscriptions into a local in-memory map and only subscribes to pub/sub topics for channels where it has connected users.

9. Message delivery receipts and read state: Slack shows unread message counts but does not show per-user read receipts (unlike iMessage). If Slack tracked read state per user per message at 10K-member channel scale, the read-receipt update storm for a single message would be enormous (10K read events per message). The product decision to not show per-user read receipts is directly informed by this engineering constraint.

What Most People Get Wrong

These are the six failure patterns I see most often when candidates answer this question. None of them are obvious mistakes, which is why they are worth memorizing before your interview.

The two most important things to get right in this question are: fan-out cost is O(channel members) and requires architectural mitigation above a certain channel size; and ordering is determined server-side by the database, not client-side by clocks or arrival sequence.

A mistake with specific consequence: candidates sometimes say "Slack uses exactly-once delivery because messages can't appear twice." This reveals a gap in distributed systems understanding. Exactly-once delivery requires two-phase commits or equivalent distributed transaction protocol across the write, fan-out, and acknowledgement steps. The added latency and complexity would destroy Slack's 200ms delivery target. The real answer is: Slack accepts duplicate delivery at the infrastructure level and makes the duplicate behavior invisible to users via nonce deduplication. The practical result is exactly-once from the user's perspective, achieved via at-least-once plus dedup.

How I Would Communicate This in an Interview

Structure is as important as content in a live interview. A candidate who knows everything but answers in a rambling stream of consciousness is harder to evaluate than one who signals their structure up front, covers three logical layers, and explicitly offers to go deeper on any one.

Here is how I would say this in a 90-second structured answer:

"When you send a message in Slack, it travels over your existing WebSocket connection to the Gateway Service, which is the WebSocket termination layer. The Gateway routes it to the Message Service, which validates permissions and persists the message to MySQL with a per-channel sequence number. This sequence number is the source of truth for ordering.

Then the Fan-out Service kicks in. It looks up channel membership, groups the online members by which Gateway instance holds their connection, and sends batch push messages to each Gateway. Each Gateway pushes the message to its local clients. Total delivery time is under 200ms for most channels.

For a 10K-member channel, the key insight is that fan-out is batched per Gateway instance, not per user. If 3,000 online users are spread across 45 Gateways, that is 45 batch pushes, not 3,000 individual ones. For really large channels, Slack moves to a pub/sub model where Gateways subscribe to channel topics and self-serve.

When a WebSocket drops, the client reconnects with exponential backoff and sends its last-known sequence numbers. The server responds with missed messages from Flannel, Slack's edge cache built specifically for this query pattern. Typing indicators skip this entire pipeline since they are ephemeral and best-effort."

The key phrases to hit: "per-channel sequence numbers for ordering," "fan-out grouped by Gateway," "Flannel as edge cache for catch-up," and "ephemeral vs durable events."

One follow-up I get asked frequently: "How does Slack handle a user sending a message while briefly offline?" The answer is: Slack requires an active connection to send. Once the client reconnects, it immediately allows the user to send, and nonce-based deduplication handles any retry uncertainty transparently.

On pacing: spend 60-70 seconds on the core path, then offer to go deeper on whichever layer the interviewer finds most interesting. The fan-out problem, the ordering guarantee, and the reconnection strategy are each worth 2-3 minutes of depth. Start high-level, hit the key terms, then wait for the follow-up question to guide your depth.

Another question I get in this thread: "What capacity numbers would you give for the Gateway fleet?" My answer: assuming 200K connections per Gateway server (comfortably under the memory ceiling) and 10 million concurrent peak connections, you need at least 50 Gateway servers. With redundancy and headroom, plan for 100-150. Each server needs roughly 10-20GB RAM dedicated to connection state. At that scale, the registry in Memcached becomes the coordination layer that makes the fleet work without any server-to-server communication.

A third follow-up: "How does Slack prevent a single bad actor from flooding a channel with messages?" Rate limiting happens at the Gateway before messages reach the Message Service. The Gateway tracks message rates per user per channel and drops requests that exceed the threshold (typically a few messages per second). This protects the message pipeline from abuse without adding overhead to the normal case.

If the interview is running long and the interviewer says "let's wrap up," my 30-second summary is: "Slack is WebSockets for push, MySQL for sequence-ordered persistence, Flannel for catch-up, Fan-out grouped by Gateway for scale, and at-least-once delivery with nonces for reliability. The architecture is driven by the fan-out cost: one write triggers N deliveries, and N can be in the thousands." That sentence covers all the key terms and if the interviewer remembers one thing from the answer, it will be the 1-to-N relationship between writes and deliveries.

Interview Cheat Sheet

• Connection model: Each client maintains one persistent WebSocket to the Gateway Service. Gateway is horizontally scaled, each instance holding 100K-500K connections, with the registry in Memcached.
• Message persistence: Messages are written to MySQL/Vitess with a per-channel monotonically increasing sequence number. This is the ordering guarantee.
• Fan-out strategy: Group recipients by Gateway instance. For large channels (1K+ members), use pub/sub topics instead of enumerating all members.
• Edge caching: Flannel caches recent messages and channel metadata. It absorbs reconnection storms and serves catch-up queries in sub-millisecond time.
• Offline handling: Offline users get push notifications via an async job queue. On reconnect, clients send last-known sequence numbers and catch up via Flannel.
• Message ordering: Always server-assigned, never client-determined. Clients sort by the ts field (a decimal string encoding sequence and timestamp), not WebSocket arrival time.
• Typing indicators: Ephemeral, best-effort delivery. No persistence, no sequence numbers, no catch-up. Fire-and-forget through the Gateway.
• Deduplication: Clients send a nonce with each message. Server deduplicates on nonce before persisting. At-least-once delivery with idempotent retry.
• Reconnection: Exponential backoff (1s, 2s, 4s, 8s, max 30s). Gateway rate-limits incoming reconnections to prevent thundering herd.
• Sharding boundary: Workspaces are sharded across Vitess. All channels in a workspace share one shard. Workspace isolation is the fan-out isolation boundary.
• Long-poll fallback: When WebSocket upgrade fails (enterprise proxy), the client falls back to HTTP long polling with the same sequence-number catch-up semantics.
• Gateway deploys: Graceful drain mode prevents forced disconnections during rolling Gateway deploys. Draining instances stop accepting new connections while existing ones migrate naturally.

Test Your Understanding

These questions test the depth of your understanding, not just recall. Each one is a variation on the core scenario that requires you to apply the architecture to a new constraint or failure mode.

Quick Recap

1. Slack clients maintain a persistent WebSocket connection to the Gateway Service, which handles authentication, rate limiting, and connection lifecycle for millions of concurrent users.
2. Every message is persisted to MySQL/Vitess with a per-channel monotonically increasing sequence number that serves as the definitive ordering guarantee.
3. Channel fan-out groups recipients by Gateway instance, reducing cross-service calls from O(members) to O(gateways) for large channels.
4. Flannel is Slack's purpose-built edge cache that serves reconnecting clients with missed messages, absorbing thundering herd problems that would otherwise overload the database.
5. Typing indicators are ephemeral, best-effort signals that bypass the persistence and sequencing pipeline entirely, because stale typing state is worse than missing it.
6. On WebSocket disconnect, clients reconnect with exponential backoff and send their last-known sequence numbers to receive a minimal catch-up response.
7. Message deduplication uses client-generated nonces to prevent duplicates from network retries during connection instability, giving at-least-once delivery with idempotent behavior.
8. Large channels (1K+ members) use pub/sub topic-based fan-out instead of direct member enumeration, letting Gateways self-serve from subscriptions rather than receiving individual pushes.
9. The workspace is the tenant isolation boundary: all channels in a workspace share one Vitess shard, keeping fan-out routing and data access workspace-local.
10. Long-poll fallback provides messaging continuity for enterprise environments where WebSocket upgrades are blocked at the proxy layer, with identical sequence-number catch-up semantics.
11. Subscription joins are sequenced to prevent missed messages between "join channel" and "subscription active."
12. Flannel's in-memory index answers catch-up queries in under a millisecond, keeping reconnects invisible to users.

Related Concepts

Studying these adjacent topics deepens the intuition built above, because every pattern Slack uses appears in other systems under different names.

• WebSocket protocol: The transport layer that makes Slack's push delivery possible. Understanding the upgrade handshake, frame format, and heartbeat mechanism explains why Slack chose WebSockets over Server-Sent Events or long polling as the primary transport.
• Fan-out patterns: The same fan-out challenge appears in Twitter's home timeline, Facebook's news feed, and any system where one write triggers reads for many users. The trade-offs between fan-out-on-write and fan-out-on-read apply directly here.
• Event-driven architecture: Slack's pipeline (persist, then fan-out asynchronously) is a textbook example of event-driven design where the write path and the delivery path are decoupled.
• Vitess and database sharding: Understanding how Vitess provides horizontal scaling for MySQL explains why Slack can handle workspace-level sharding while maintaining per-channel sequence guarantees.
• Idempotency and at-least-once delivery: The nonce-based deduplication pattern Slack uses for messages is the same pattern that prevents duplicate payments, double submissions, and replay attacks across distributed systems. At-least-once delivery with idempotent retries is a general reliability pattern worth recognizing wherever you see network unreliability combined with write operations.
• Thundering herd and jitter: When Slack Gateway pods restart and thousands of clients reconnect simultaneously, the reconnect storm is mitigated by exponential backoff with jitter. This pattern applies anywhere a shared resource (a cache, a database, a lock server) comes back online after an outage. Without jitter, correlated clients retry at the same instant and immediately overload the recovering service. Slack's use of Flannel to absorb catch-up reads is the caching half of the same solution; random jitter is the timing half.