How TikTok avoids showing you the same video twice

The Problem Statement

Interviewer: "You scroll TikTok every day. How does it avoid showing you the same video twice? Walk me through the system that tracks which videos you have already seen and filters them from the For You Page."

This question sounds deceptively simple. At first it seems like a database lookup: store a list of seen videos per user, check against it before showing a new one. But the moment you think about the scale, that naive model falls apart completely.

TikTok has over 1 billion active users. A typical user watches 100+ videos per day. That is 100 billion impression events per day across the platform. The question the interviewer is actually asking is: how do you track 100B events per day efficiently enough that a deduplication check can happen inside a recommendation pipeline that needs to complete in under 200ms?

A strong answer covers three layers: probabilistic filtering (Bloom filters), a persistent exclusion window (Redis sorted sets), and a session-scoped in-memory buffer that merges to the persistent layer at the end of each session. A weak answer says "store video IDs in a database per user." I have seen that answer from senior engineers who have not thought through the write amplification.

Clarifying the Scenario

You: "Before I start, a few questions to make sure I understand the scope."

You: "When you say seen, do you mean the user actually watched the video, or any video that appeared on screen, even briefly?"

Interviewer: "Good question. What do you think is the right definition?"

You: "I would define seen as a meaningful impression: the user watched at least a few seconds, or more than 50% of a short video. Brief flickers as the user swipes past should not count. If we track every frame that appeared on screen, the impression volume doubles and the dedup list grows faster than it needs to."

Interviewer: "Reasonable. Use that definition."

You: "Is the deduplication window global (never show the same video twice ever) or time-windowed? I ask because if a user saw a video 18 months ago, showing it again is probably fine."

Interviewer: "What would you recommend?"

You: "A 30-day rolling window. Videos older than 30 days become eligible again. This keeps the dedup data structure bounded per user and reflects how memory actually works. It also means if a meme resurfaces two months later, TikTok can re-serve it."

Interviewer: "Makes sense. Go ahead."

You: "Last one: when I say not showing the same video twice, I am talking about the For You Page feed. The search results, following feed, and trending tabs are separate surfaces and probably have their own dedup logic."

Interviewer: "Focus on the For You Page. That is the primary feed."

You: "Got it. I will cover four parts: the scale math that makes naive approaches fail, Bloom filters for probabilistic seen-video tracking, the Redis exclusion set for recent impressions, and the session-scoped buffer that makes the in-session experience seamless."

Asking about the time window is the detail most candidates skip. Without it, you end up designing an unbounded data structure that grows forever per user. Scoping the window to 30 days is the pragmatic design choice, and mentioning it shows you think about data lifecycle.

My Approach

I organize my thinking around three layers of deduplication, each optimized for a different trade-off:

1. Bloom filter layer: Probabilistic, O(1) per check, handles the bulk of the 30-day window cheaply. Tolerates rare false positives (occasionally withholds a video the user has not seen), but never false negatives (never re-serves a video the user definitely has seen).
2. Redis sorted set layer: Exact deduplication for the last 500 seen videos per user. Fast, bounded, and consistent across devices.
3. Session buffer layer: In-memory set on the application server for the current session. Prevents duplicates within a single browsing session before the data is persisted.

The candidate pipeline starts with roughly 1,000 videos from the recommendation model, runs them through all three dedup layers, and delivers 50 clean candidates to the feed ranking stage. The dedup filters have to complete in under 50ms because the full pipeline budget is around 200ms.

Think of it like the spam filter stack your email provider runs. A fast hash-based blocklist knocks out 95% of spam at the edge in a millisecond. A slower pattern-matching engine handles the remaining 5%. And a per-recipient rule set handles the last edge cases. Each layer is tuned for speed vs precision, and only the most suspicious emails make it to the expensive layer.

The Architecture

The key insight in this architecture is that impressions flow one way (from app to Kafka) and dedup reads flow the other (from dedup service to Bloom filter and Redis). The write path is async and the read path is sync inside the recommendation latency budget.

The Kafka pipeline does three jobs: updating the Bloom filter, updating the Redis sorted set, and archiving to the columnar store for ML training. These three consumers each do different things with the same event, which is exactly why the fan-out model is right here. Adding a fourth consumer (analytics, abuse detection, etc.) costs nothing on the producer side.

The recommendation engine generates 1,000 candidates. The dedup filter knocks that down to roughly 50 clean videos. The final ranker scores those 50 and returns the top 20 to the client. That funnel, 1000 to 50 to 20, is a common pattern in production recommendation systems and worth naming explicitly in an interview.

The request flow for a single For You Page fetch works like this:

1. The client sends a feed request with its session ID.
2. The feed service loads the session impression buffer from the session store. For a new session, this buffer starts empty.
3. The recommendation engine runs retrieval and generates 1,000 candidate video IDs using Approximate Nearest Neighbor search over user and video embeddings.
4. The dedup filter checks each candidate across three layers in priority order: session buffer first (in-memory hash set, no network call), then the 30-day Bloom filter (30 Redis BITFIELD GETs in one pipelined round trip), then the Redis sorted set (ZSCORE on the last 500 entries).
5. Any candidate that hits in any layer is removed. Roughly 950 candidates are filtered, leaving about 50 clean candidates.
6. The final ranker scores the 50 clean candidates and returns the top 20 to the client.
7. The 20 served video IDs are immediately added to the session impression buffer, before any watch events are confirmed by the client. This prevents the same video appearing twice in consecutive feed requests even if no impression event has been received yet.

Here is how the storage footprint breaks down at the stated scale:

The Bloom filter at 162 TB is sharded across a Redis cluster and entirely in DRAM. The Redis sorted set at 4 TB is also in DRAM. The columnar store is the only component that grows unboundedly, but it lives in cheap object storage and is accessed only by offline ML training jobs. The read path (feed generation) never touches the columnar store.

Deep Dive 1: Bloom Filters for Seen-Video Tracking

A Bloom filter is a probabilistic data structure that answers the question "have I seen this ID before?" in O(1) time and constant space. It can produce false positives (says "seen" when the video is actually new) but never false negatives (it never says "not seen" when the user actually watched it).

For TikTok's dedup problem, the error direction matters enormously. A false negative means re-serving a video the user has already watched. That is a bad experience and exactly what we are trying to prevent. A false positive means occasionally withholding a video the user has not seen. That is a minor loss of one video from a feed of thousands. The acceptable false negative rate is zero. A false positive rate of 0.1% is completely fine.

I will tune the Bloom filter to target a 0.1% false positive rate. At that rate, roughly 1 in 1,000 candidate videos gets incorrectly filtered. The user never notices.

The sizing math: if a user watches 100 videos per day for 30 days, that is 3,000 video IDs in the filter. To hit a 0.1% false positive rate with 3,000 elements, the optimal filter needs roughly 43,000 bits (about 5.4 KB) with 10 hash functions. Per user, that is one tiny bitfield. Across 1 billion users, that is approximately 5.4 TB of Bloom filter state, which fits comfortably in a Redis cluster.

The time-windowing trick is important. Rather than one monolithic Bloom filter per user, I keep 30 daily filter windows. Each day gets its own bitfield. When a day expires (older than 30 days), I delete that bitfield. This makes the rolling window automatic: you never need to "remove" individual video IDs from the filter (which Bloom filters cannot do). You just expire the whole day bucket.

Deep Dive 2: Impression Log Pipeline

The Bloom filter handles the dedup reads. The impression log pipeline handles the writes. Every time a user watches a video, that event needs to: update the Bloom filter, update the Redis sorted set, and land in long-term columnar storage for ML training. Three consumers. One producer. This is a textbook Kafka fan-out.

Partitioning Kafka by userID is the critical design choice here. All events for the same user land on the same partition, which means the Bloom Filter Writer processes a user's events in order. If you partitioned by videoID instead, events for the same user would scatter across 200 partitions, and the Bloom filter updates would be processed by different consumers concurrently, causing write races.

The Redis sorted set writer has a gating condition: only record impressions where watch_pct >= 0.5. Skips and brief flickers do not go into the sorted set. This keeps the set clean and aligned with the definition of "meaningfully seen." The Bloom filter is more aggressive, recording any view above a lower threshold, because false positives are cheap there.

Deep Dive 3: Session vs Lifetime Deduplication

The trickiest part of the dedup architecture is the gap between what the user has seen in the current session and what the persistent store knows about. The Bloom filter and Redis sorted set update asynchronously, minutes after the actual impression. But the user expects deduplication to be instantaneous.

If the user opens TikTok and watches video A at 2:00 PM, then closes and reopens at 2:01 PM before the impression has been persisted, the Bloom filter does not yet know about video A. Without session-scoped deduplication, the feed could serve video A immediately after reopening.

The session buffer lives on the application server for the duration of the request session, not on the client. When the recommendation pipeline processes a feed request, it passes the current-session impression set as part of the request context. The dedup service checks this in-memory set first (O(1) hash lookup), then the Redis sorted set, then the Bloom filters.

On session end, the application server receives the session-end event and merges the session buffer into the Redis sorted set. Only at that point do the persistent layers know about the current session's impressions. Until then, the session buffer is the source of truth for that user's active session.

The Tricky Parts

1. The write amplification problem: Every impression event needs to update multiple stores: Bloom filter bits, Redis sorted set entries, and the columnar archive. At 100B events per day, even a 1-second delay in any consumer can cause the consumer to fall minutes behind. Kafka consumer lag monitoring is critical. If the Bloom filter writer falls 5 minutes behind, the dedup accuracy degrades for those 5 minutes.

2. Cold start for new users: A new user has an empty Bloom filter and an empty Redis sorted set. Every video is a candidate. The dedup layer passes all 1,000 candidates through to ranking. That is fine for dedup (no false positives), but it also means the recommendation model has no negative signal (seen history) to work with, which affects the quality of embeddings used for retrieval. Cold start in recommendation systems always has this dual problem: no positive history and no negative history.

3. Video ID reuse and content recycling: TikTok occasionally re-uploads popular videos under new IDs (due to copyright takedowns, re-encodings, or creator re-posts). The same content appears under a new ID, so the dedup system treats it as a new video. This is actually correct behavior: the dedup system tracks impressions by ID, not by content hash. Content-level deduplication (detecting visually identical videos) is a separate, harder problem involving perceptual hashing or video embedding comparison.

4. Cross-device session state: Two active sessions on different devices for the same account share the persistent Bloom filter and Redis sorted set but have separate session buffers. A video watched on device A will not be excluded from device B's feed until the session buffer from A merges to the persistent store. This is a known best-effort situation. Building real-time cross-device session coordination is not worth the complexity for the marginal improvement.

5. The 30-day boundary effect: When videos exit the 30-day window, they become eligible for the feed again. For a prolific user who watches 100 videos/day, roughly 100 videos cycle back into eligibility every day. This is fine and intentional. But it creates a subtle anomaly where a user might see a video they watched exactly 30 days ago resurface. A tiered confidence decay (reducing exclusion probability as the window approaches expiry) makes this transition less jarring.

6. Impression event schema evolution: The impression payload (userID, videoID, watch_pct, timestamp) will grow over time. Adding a rewind_count field or an engagement_type enum without backward-compatible schema management breaks old consumers. Use a schema registry (Avro or Protobuf with schema evolution rules) rather than plain JSON so new fields are optional and old consumers handle them gracefully.

Failure Modes and Monitoring

This system fails silently. When the dedup layer degrades, it does not throw errors -- it just stops filtering correctly. Users start seeing already-watched videos. Understanding the specific failure signatures helps you both operate the system and discuss it in an interview.

Kafka consumer lag is the primary metric to watch. The three consumers (Bloom filter writer, Redis writer, columnar store writer) are independent. Bloom filter writer lag is the most user-visible: if it falls 5 minutes behind, users see videos they watched in the last 5 minutes reappear in the feed. Alert at 30 seconds of lag on the Bloom filter writer, 60 seconds for the Redis writer. The columnar store writer tolerates longer lag (up to 5 minutes) since it only affects ML training data freshness.

Bloom filter saturation is a quiet failure. A daily bitfield with more than 80% of its bits set has a dramatically elevated false positive rate. A user who watched 2,000 videos in a single day (a binge session or a bot) will overflow a filter sized for 100 videos/day. Track the fill ratio of each daily bitfield and cap writes at a reasonable maximum (say, 500 per day) to prevent a single anomalous session from corrupting the filter.

Redis sorted set cardinality anomalies indicate the ZREMRANGEBYRANK trim step is broken. If trimming stops, sorted sets grow unboundedly. Alert on any user's sorted set exceeding 600 entries and trigger a manual trim.

Recommended alert thresholds for the dedup monitoring dashboard:

What Most People Get Wrong

How I Would Communicate This in an Interview

Here is how I would actually say this in a 90-second verbal summary:

"TikTok has around 1 billion users watching 100+ videos each per day. That is 100 billion impression events daily. A naive approach of storing all seen video IDs per user in a database is immediately out: the write throughput and storage requirements are unmanageable.

I would use three layers. First, a Bloom filter per user, time-windowed into daily buckets with a 30-day rolling window. This is probabilistic and O(1) per check. It catches the vast majority of seen videos with a tuned false positive rate of about 0.1%. Second, a Redis sorted set storing the last 500 exact seen-video IDs per user. This gives exact deduplication for recent content without any probabilistic error. Third, a session-scoped in-memory set that captures what the user has watched in the current session, before those impressions have had time to propagate to the persistent layers.

The recommendation pipeline generates 1,000 candidates. The dedup filter checks all three layers and passes roughly 50 clean candidates to the final ranker. The ranker returns the top 20 to the app.

Impressions flow through Kafka, partitioned by user ID, to three consumers: the Bloom filter writer, the Redis writer, and a columnar store for ML training data. The write path is entirely async. It does not touch the read path at all.

The interesting edge cases are cold start (new users have no history, so all 1,000 candidates pass through), cross-device sessions (best-effort dedup between simultaneous sessions), and the 30-day boundary (videos cycle back in gracefully with tiered confidence decay)."

This hits the core architecture, the scaling-driven choices, and three nuanced edge cases. At senior level, you want to name specific data structures (Bloom filter, Redis sorted set) and give concrete numbers (1,000 candidates, 50 after dedup, 30-day window, 0.1% false positive rate).

Structure your verbal answer in three acts

Act 1 (30 seconds): State the scale problem and why naive approaches fail. "1B users, 100 videos/day, 100B daily impressions. A user-video history table in a relational database would need to handle 1.2M inserts per second and grow to trillions of rows. That is a write throughput problem, not a query problem."

Act 2 (60 seconds): Name the three-layer solution with data structures precise. "Bloom filter per user with daily time buckets and a 30-day rolling window. Redis sorted set for the last 500 exact IDs. Session impression buffer for same-session recency. Each layer handles a different time horizon: session (minutes), recent (days), historical (weeks)."

Act 3 (30 seconds): Cover the async pipeline and key edge cases. "Impressions fan out through Kafka, partitioned by userID, to three consumers: Bloom filter writer, Redis writer, columnar store for ML. Key edge cases: cold start (no history, all candidates pass), cross-device best-effort, 30-day boundary decay."

Vocabulary to use precisely

Use these terms exactly. Interviewers who know distributed systems register them as depth signals:

• "Bloom filter" (not "approximate set" or "probabilistic filter")
• "False positive rate, tuned to 0.1%" (not "some errors are acceptable")
• "Time-windowed daily buckets with Redis TTL" (you understand the no-deletion problem)
• "Kafka fan-out, partitioned by userID" (you know about write race conditions)
• "ZADD with ZREMRANGEBYRANK" (you can name the Redis operations)
• "Candidate funnel: 1,000 to 50 to 20" (you know two-stage recommendation architecture)
• "Session buffer as the first check" (you understand async lag)

Common follow-up questions and how to answer them

"What if the Bloom filter gives a wrong answer?" -- "The only wrong answer it can give is a false positive. It never produces a false negative. A false positive means withholding a new video the user has not seen, which is invisible to the user. A false negative (re-serving a seen video) is architecturally impossible from the filter layer."

"What is the latency impact of the dedup check?" -- "The 30 Bloom filter bitfield GETs are pipelined into a single Redis round trip. The ZSCORE check on the sorted set is one command. The session buffer lookup is in-memory. The entire dedup check adds roughly 1-2 ms to feed generation, inside a 150 ms total request budget."

"What would you change at 10x scale (10B users)?" -- "The approach does not change. The Redis cluster adds shards, and the Kafka topic adds partitions. The key architectural invariant (async write path decoupled from sync read path) holds all the way up. The Bloom filter math changes slightly: at 10B users the total Bloom filter storage is 1.6 PB, still manageable across a large Redis cluster."

Interview Cheat Sheet

• Scale anchor: 1B users x 100 videos/day = 100B impression events/day. This single number rules out any synchronous, per-user DB write approach.
• Bloom filter: Probabilistic set membership. O(1) check. False positives OK (1 in 1000 filtered incorrectly). False negatives never OK (re-serving seen video is the failure mode).
• Time window: 30-day rolling window, not global. Daily buckets with 31-day Redis TTLs. No sweeper job needed.
• Redis sorted set: Last 500 exact seen-video IDs per user. ZADD + ZREMRANGEBYRANK. Score = impression timestamp. Exact for recent history.
• Session buffer: In-memory set on the app server, keyed by session ID. First layer checked. Handles lag between impression and persistent store update.
• Kafka partitioning: Always partition by userID when the consumer updates per-user state. Prevents write races on shared structures like the Bloom filter.
• Candidate funnel: 1,000 candidates from recommendation, ~950 filtered by dedup, 50 to final ranker, top 20 served to app.
• Write path is async: Impression events fire-and-forget to Kafka. Feed generation never waits for impression writes to complete.
• Cross-device dedup: Best-effort. Persistent layers are shared. Session buffers are per-device. Simultaneous cross-device duplicates are a known, accepted trade-off.
• Cold start: New users have empty Bloom filters. All 1,000 candidates pass through dedup. Feed quality is driven entirely by the recommendation model's fallback strategy (trending, popular content).
• False positive math: For n=3,000 seen videos and p=0.001 (0.1% FPR), optimal bit array size m = ceil(n * log(1/p) / ln(2)^2) = ~43,000 bits = 5.3 KB per daily bucket. Knowing this formula at an interview for senior+ is a strong differentiator.
• Exactly-once semantics not required: The Bloom filter writer at-least-once delivery is fine. Duplicate impression events just re-set bits that are already 1, which is idempotent. The Redis ZADD is also idempotent for the same (userID, videoID) entry. No distributed transaction needed.
• Monitoring metric that matters: Track the re-serve rate (fraction of served videos that users report as already watched) as the dedup SLO, not Kafka consumer lag alone. Both are needed but the product metric is the source of truth.

Test Your Understanding

Quick Recap

1. TikTok's dedup problem is a write-at-scale problem: 100 billion impression events per day makes per-user database rows impractical.
2. Bloom filters let you track 30 days of seen-video history per user in roughly 160 KB of space, with a tunable false positive rate of 0.1%.
3. Daily time-windowed Bloom filter buckets with Redis TTLs eliminate the need for any active cleanup job and implement the rolling 30-day window automatically.
4. The Redis sorted set provides exact deduplication for the most recent 500 video IDs, covering the high-frequency re-serve window.
5. A session-scoped in-memory buffer handles the async lag between impression and persistent-store update, preventing same-session re-serves.
6. Kafka impression events are partitioned by user ID so all per-user state updates are serialized through a single consumer instance, preventing write races.
7. The candidate funnel runs 1,000 generated candidates through three dedup layers to produce 50 clean candidates for final ranking.
8. Cross-device session dedup is best-effort. Periodic session flushes reduce the exposure window without requiring real-time cross-device coordination.
9. False positives in the Bloom filter are acceptable and tunable. The failure mode to prevent is false negatives (re-serving seen videos), not false positives (occasionally withholding a new video).
10. The Kafka partition key (userID) is not incidental. It guarantees all impression events for a given user are processed in sequence by a single consumer, preventing write races on the per-user Bloom filter and Redis sorted set.

Related Concepts

• Bloom Filters: The probabilistic data structure at the core of the dedup layer. Understanding bit array sizing, hash function selection, and false positive rate math is essential background for this system.
• Kafka Fan-Out Pattern: The impression pipeline uses a single Kafka topic with multiple consumers, each updating different state. This pattern appears in activity feeds, notification systems, and any real-time event pipeline.
• Redis Data Structures: Sorted sets with ZADD and ZREMRANGEBYRANK are the key Redis primitives for the exact-recent impression store. Understanding sorted set complexity and eviction is foundational.
• Recommendation Systems Architecture: The dedup filter is downstream from the candidate retrieval and upstream from the final ranker. Understanding where dedup fits in the two-stage retrieval pipeline puts this article in context.
• Probabilistic Data Structures: Bloom filters are the canonical example, but the broader category includes Count-Min Sketch (frequency estimation), HyperLogLog (cardinality estimation), and Cuckoo filters (a Bloom variant that supports deletion). Each trades some accuracy for dramatic space savings at scale.
• Write-Ahead and Async Pipelines: The impression pipeline uses a fire-and-forget write model via Kafka, with the read path (dedup checks) fully decoupled from the write path (impression logging). This async separation is a recurring pattern in large-scale systems wherever write throughput would otherwise block read performance.
• Time-Windowed State with TTL: Using Redis TTL for automatic expiry instead of active deletion is a broadly applicable pattern. It appears in rate-limiting windows, session expiry, fraud detection lookback windows, and any system that needs bounded history without a background sweeper job.
• Two-Stage Ranking Architecture: Most large-scale recommendation systems use retrieval (fast, approximate, returns 1,000 candidates) followed by scoring (slower, precise, returns the top 20). The dedup filter sits between these two stages. Understanding this pipeline positions you for any feed or recommendation design question.