How Twitter computes trending topics in real time

The Problem Statement

Interviewer: "Twitter shows trending topics within minutes of a major event breaking. With 500 million tweets per day, how does the system decide what is trending right now? Walk me through the architecture."

This question tests three things: whether you understand real-time stream processing at massive throughput, whether you know the difference between "popular" and "trending" (velocity vs volume), and whether you can reason about data quality problems like spam, bot manipulation, and geographic localization.

Most candidates describe a word counter. Strong candidates explain why raw counting is insufficient, how time-decay scoring separates a genuine spike from a permanently popular keyword, and how geographic segmentation surfaces local events that would be invisible in a global aggregate.

Clarifying the Scenario

You: "Before I start, I want to make sure I scope this correctly."

You: "When you say 'trending,' do you mean globally trending, or also location-specific trends like 'trending in New York' vs 'trending worldwide'?"

Interviewer: "Both. Start with global, then explain how you add geographic awareness."

You: "Got it. Should I focus on hashtags only, or also keywords and phrases that trend organically without a hashtag? For example, during an earthquake people tweet 'earthquake' without any hashtag."

Interviewer: "Both hashtags and organic keywords."

You: "One more: how fast does a trend need to surface? Seconds or minutes?"

Interviewer: "Minutes. A topic should appear in the trending list within 5 to 10 minutes of starting to spike."

You: "OK. I will structure my answer in four parts: the tweet ingestion pipeline, the count-min sketch layer for approximate frequency counting, the time-decay scoring model that prioritizes velocity over volume, and the geographic plus personalized trend ranking."

My Approach

I break trending detection into five stages:

1. Ingestion: Consuming the tweet firehose (500M tweets/day, roughly 6,000 tweets/sec average, bursting to 50,000+ during events like the Super Bowl)
2. Extraction: Pulling hashtags, keywords, and named entities from each tweet
3. Approximate counting: Using a Count-Min Sketch to track frequency for millions of unique terms in sub-linear memory
4. Time-decay scoring: Applying exponential decay so recent mentions weigh more than older ones, making velocity the dominant signal
5. Ranking: Generating per-region and per-user trend lists using geographic segmentation and interest-based personalization

The fundamental insight is that "trending" does not mean "most mentioned." The word "the" appears in millions of tweets per day but never trends. "Good morning" spikes every day at 8am but is not interesting. Trending means the rate of mentions for a topic is significantly higher than its historical baseline right now. It is a velocity anomaly, not an absolute count.

This is the mistake I see most often in interviews: candidates design a system that surfaces the most popular topics, not the most accelerating ones. Justin Bieber would be permanently trending in that system. The correct system surfaces topics that are growing unusually fast compared to their own baseline.

The Architecture

The trending pipeline is a multi-stage stream processing system. Tweets flow from ingestion through extraction, approximate counting, scoring, filtering, and ranking before reaching the trending list.

Here is how a tweet becomes a trend:

Step 1: Ingestion. A user tweets "Just felt an earthquake in LA!" The tweet enters Kafka, partitioned for parallel processing across hundreds of stream workers.

Step 2: Extraction. The NLP pipeline extracts keywords: "earthquake," "LA." It detects the language (English), normalizes tokens (lowercase, stemming), and extracts any hashtags like #earthquake directly.

Step 3: Approximate counting. The Count-Min Sketch increments the approximate frequency for "earthquake" in the current time window. The Space-Saving algorithm checks whether "earthquake" qualifies as a top-K candidate based on its sketch count.

Step 4: Time-decay scoring. The decay function applies an exponential weight to recent mentions. The velocity score compares the decayed count against the historical baseline for "earthquake" at this hour and day of week. If the z-score exceeds 3.0, it qualifies as a trending candidate.

Step 5: Filtering. The spam filter checks whether the spike is driven by bots or coordinated accounts. The safety filter checks editorial policies. If the topic passes both, it enters the trending list.

Step 6: Ranking. The system generates separate trending lists for global, per-city, per-country, and personalized views. "Earthquake" trends in Los Angeles first (concentrated signal) before it trends globally (diluted by worldwide traffic).

Count-Min Sketch for Approximate Frequency Counting

This is the core data structure that makes trending detection possible at scale. You cannot maintain an exact counter for every keyword. With millions of unique tokens flowing through the pipeline, exact counting would require gigabytes of memory per worker. The Count-Min Sketch gives you approximate counts with fixed memory and O(1) operations.

A Count-Min Sketch is a 2D array of counters with d rows and w columns. Each row uses a different hash function. To increment a keyword, hash it with each of the d hash functions, and increment the counter at each resulting position. To query a keyword's count, hash it with all d functions and return the minimum value across all rows.

The key property: it can overestimate (due to hash collisions) but never underestimate. Taking the minimum across d rows minimizes the collision noise.

The memory math is important for interviews. A CMS with width w = 2^20 (about 1 million columns) and depth d = 5 uses 5 * 1M * 4 bytes = 20MB of memory. The error bound is ε = e/w ≈ 2.7 / 1M = 0.00027% of total count. For 6,000 tweets/sec over a 5-minute window (1.8M events), the maximum overestimation for any keyword is about 486 counts. For trending detection (where trending topics have tens of thousands of mentions), this error is negligible.

I combine CMS with the Space-Saving algorithm. CMS tracks approximate frequency for ALL keywords. Space-Saving maintains an exact top-K list (say, top 1,000) by evicting the least frequent candidate when a new one qualifies. This two-layer approach gives you bounded memory for global counting and precise tracking for the trending candidates.

For the interview: say "Count-Min Sketch for approximate counting in fixed memory, Space-Saving for top-K candidate tracking, and the combination gives us bounded resources with good accuracy." That single sentence shows you know your data structures.

Time-Decay Scoring: Why Velocity Matters More Than Volume

Raw counts are not enough. "Good morning" gets millions of mentions every day at 8am. "Justin Bieber" has a consistently high mention count. Neither is "trending" because they are always popular. A trending topic is one experiencing an unusual spike relative to its own historical baseline, and that spike must be recent.

Time-decay scoring solves both problems. Instead of counting raw mentions, each mention is weighted by how recent it is. A mention from 1 minute ago counts almost fully. A mention from 30 minutes ago counts half as much. A mention from 2 hours ago barely registers.

The exponential decay function:

decayed_count = Σ e^(-λ * (t_now - t_mention))

Where λ controls the half-life. With a half-life of 30 minutes (λ = ln(2) / 1800 ≈ 0.000385), a mention from 30 minutes ago contributes 0.5 to the count. A mention from 60 minutes ago contributes 0.25. This naturally makes the score velocity-sensitive: a burst of 10,000 mentions in the last 5 minutes produces a much higher decayed count than 10,000 mentions spread evenly across 2 hours.

I then compare the decayed count to a time-aware historical baseline. The baseline is computed from weeks of data, broken down by hour of day and day of week. "Good morning" has a high baseline at 8am on weekdays but a low baseline at 2am on Sundays. The anomaly detector uses the correct baseline for the current slot.

The z-score formula: z = (decayed_count - historical_mean) / historical_stddev

For "earthquake": z = (14,820 - 180) / 320 = 45.7. Massively anomalous. Trending.

For "good morning": z = (12,100 - 11,500) / 850 = 0.7. Normal daily variance. Not trending.

For "justin bieber": z = (3,200 - 2,900) / 320 = 0.94. Slightly above average. Not trending.

I also enforce a minimum absolute count threshold (500 mentions in the current window). This prevents obscure topics from trending just because their baseline is near zero. If a keyword normally gets 0 mentions and suddenly gets 5, the z-score is technically infinite, but 5 mentions do not constitute a meaningful trend.

The production approach uses hopping windows (a practical compromise). Divide time into small buckets (1 minute each), then sum the last 5 buckets to approximate a 5-minute sliding window. Apply time-decay weighting during the summation. The approximation error is at most 1 minute (the granularity of one bucket), which is fully acceptable for trending detection.

Geographic and Personalized Trend Ranking

A single global trending list misses local events entirely. An earthquake in LA generates massive tweet volume in Los Angeles but gets diluted in the worldwide aggregate. A political rally in London is meaningful to UK users but noise to users in Tokyo. Geographic segmentation is what makes trending useful.

The architecture maintains separate counting pipelines per region. Every tweet with location data (GPS, user profile city, IP geolocation) feeds into both the global pipeline and the appropriate regional pipeline. The anomaly detector runs independently at each level.

For a user in Los Angeles, the trending tab shows a blended list: local trends first (trending in LA), then national (trending in US), then global (trending worldwide). Each level runs its own anomaly detection with its own baselines, so a topic can trend at one level without trending at others.

Personalized trending adds another dimension. Instead of showing the same list to every user in LA, the system overlays the user's interest graph. Topics trending among accounts you follow are weighted higher in your personal view. If you follow earthquake monitoring accounts and LA news, "earthquake" surfaces with higher priority. If you follow gaming accounts, a trending game release might rank ahead of the earthquake in your personalized tab.

The personalization signal is lightweight: compute the fraction of your followed accounts contributing to the trend. If 20% of accounts you follow are tweeting about #WorldCup, that trend gets a personalization boost. Crucially, this is computed at read time (when you load the trending tab), not write time (when each tweet is processed). This keeps the write path simple with no per-user counting overhead.

About 30% of tweets have precise location data (GPS or tagged location). For the remaining 70%, the system infers location from the user's profile city, IP geolocation, and language signals. A user tweeting in Marathi about Mumbai traffic is almost certainly in Mumbai, even without GPS. The inference is imperfect but sufficient for regional trending.

The Tricky Parts

1. Cold start for new keywords. When a completely new term appears (a coined hashtag, a newly famous name), it has no historical baseline. The z-score formula fails (division by zero). The fix: maintain a default baseline computed as the average stats across all keywords of similar frequency class. For brand-new keywords, use this default. Switch to the keyword-specific baseline once 24 to 48 hours of data accumulates. The default is kept deliberately low so genuinely viral new terms still produce high z-scores.

2. The "always popular" masking problem. "Good morning" has such a high baseline at 8am that even a genuine viral meme involving those words only produces a z-score of 2.5. The fix: add a velocity z-score measuring acceleration (second derivative of the count). A viral meme causes acceleration that the normal daily ramp-up does not. Combine static z-score and velocity z-score with an OR gate.

3. Trending decay and removal. Once a topic stops accelerating, it should stop trending. But users expect trends to persist for a few hours. The system uses a "trending half-life": once the z-score drops below threshold, the topic remains with decaying priority for 2 to 4 hours. Topics that peaked at #1 globally get a longer half-life than those that barely crossed the threshold.

4. Keyword ambiguity. "Apple" could mean the company, the fruit, or a person. Without disambiguation, unrelated spikes conflate into one trend. The mitigation: co-occurrence analysis clusters tweets by context words, and hashtags serve as disambiguation anchors. The UI can present subtopics when users click through.

5. Retweets vs original mentions. A single tweet retweeted 50,000 times is one opinion amplified, not 50,000 independent signals. Weight original tweets at 1.0 and retweets at 0.1 to 0.3 to measure genuine breadth of conversation rather than amplification depth.

6. Coordinated manipulation. Bot networks tweet the same hashtag from thousands of low-reputation accounts. The fix: each tweet is weighted by account reputation, and a coordination detector flags synchronized bursts from socially unconnected accounts. This degrades gracefully even if some bots are misclassified.

What Most People Get Wrong

How I Would Communicate This in an Interview

Here is how I would actually say this:

"Trending detection is fundamentally an anomaly detection problem, not a counting problem. The word 'the' appears in millions of tweets but never trends. Trending means the mention rate for a topic is significantly above its own historical baseline right now.

The pipeline has four stages. First, tweets enter Kafka partitioned for parallel processing. An NLP pipeline extracts hashtags, keywords, and named entities.

Second, I use a Count-Min Sketch for approximate frequency counting across millions of unique keywords in fixed memory, about 20MB per sketch. The sketch feeds a Space-Saving top-K algorithm that maintains the 1,000 most frequent candidates.

Third, I apply time-decay scoring with an exponential half-life of about 30 minutes, so recent mentions weigh much more. I compute a z-score for each candidate by comparing its decayed count against its historical baseline for this hour and day of week. A z-score above 3.0 with a minimum absolute count qualifies as trending.

Fourth, I run this entire pipeline independently at city, country, and global levels. An earthquake in LA trends locally within minutes even if it takes longer to register globally. Personalization overlays the user's follow graph to re-rank each user's blended geographic list.

Spam filtering weights each tweet by account reputation. Bot campaigns with thousands of low-reputation accounts produce minimal signal that rarely triggers trending."

Interview Cheat Sheet

• Trigger: "How does Twitter detect trends?" say "Anomaly detection over time-decayed counts, not raw popularity. Z-score against historical baselines. Z above 3.0 with minimum absolute count equals trending."
• Count-Min Sketch: "Probabilistic data structure for approximate frequency counting. Fixed memory (20MB), O(1) update and query. Overestimates but never underestimates. Pair with Space-Saving for top-K candidates."
• Time-decay: "Exponential decay with 30-minute half-life. A mention from 1 minute ago counts almost fully. A mention from 2 hours ago barely registers. Makes the score velocity-sensitive by design."
• Velocity vs volume: "Trending means 'growing unusually fast' not 'frequently mentioned.' Justin Bieber would permanently trend in a volume-based system. Time-decay plus z-score scoring fixes this."
• Sliding windows: "Hopping windows: 1-minute buckets summed over 5-minute spans. Avoids the boundary spike problem of tumbling windows where a spike straddling two buckets is undercounted in both."
• Spam filtering: "Weight tweets by account reputation score (0 to 1). Bot accounts contribute minimally. Coordination detection flags synchronized bursts from socially unconnected accounts."
• Geographic trending: "Separate counting pipelines per city, country, and global. Independent anomaly detection at each level. A topic can trend locally without trending globally."
• Personalization: "At read time, re-rank the geographic trend list using the user's follow graph. Topics trending among accounts you follow rank higher. No per-user write-time computation."
• Stream processing: "Apache Flink or Storm for real-time counting. Kafka for ingestion buffering. Never batch MapReduce for trending detection."
• New keywords: "No historical baseline for new terms. Use a default baseline from similar-frequency keywords until 24 to 48 hours of keyword-specific data accumulates."

Test Your Understanding

Quick Recap

1. Trending is an anomaly detection problem, not a counting problem. "Popular" and "trending" are fundamentally different: trending means growing unusually fast relative to a topic's own baseline.
2. Count-Min Sketch provides approximate frequency counting with fixed memory and O(1) operations. Space-Saving maintains the top-K candidates for anomaly evaluation.
3. Time-decay scoring with exponential decay (30-minute half-life) makes the count velocity-sensitive: recent mentions dominate, old mentions fade naturally.
4. Z-score anomaly detection compares decayed counts against time-aware baselines (hour of day, day of week) to surface genuinely unusual spikes.
5. Spam filtering weights each tweet by account reputation (0 to 1) and uses coordination detection to suppress synchronized bot campaigns.
6. Geographic trending runs independent counting pipelines per city, country, and global with separate anomaly detection at each level.
7. Personalized trending re-ranks the geographic list at read time using the user's follow graph, requiring no per-user write-time computation.
8. Stream processing (Flink, Storm, Kafka Streams) is mandatory for sub-minute latency. Batch processing is never acceptable for trending detection.

Related Concepts

• Stream processing architectures covers how Apache Flink, Storm, and Kafka Streams distribute stateful computation across workers, handle exactly-once semantics, and manage checkpointing for fault tolerance.
• Probabilistic data structures explains the mathematical foundations of Count-Min Sketch, Bloom filters, and HyperLogLog, including error bounds, hash function selection, and conservative update optimization.
• Time-series anomaly detection covers the broader field of detecting outliers in temporal data, including EWMA, Holt-Winters, and seasonal decomposition methods beyond simple z-scores.
• Content moderation and trust and safety explains how platforms detect coordinated inauthentic behavior, bot networks, and manipulation campaigns at scale.
• Real-time personalization covers recommendation systems that overlay global signals with per-user interest graphs to produce individualized feeds and trend lists.

title: "How Twitter detects trending topics in real time" description: "How Twitter uses Apache Storm, sliding time windows, and anomaly detection to identify trending topics within minutes of emergence." tags:

• "situational"
• "twitter"
• "trending"
• "stream-processing" difficulty: "medium" category: "situational/architecture" order: 60 publishedAt: "2026-04-12" relatedArticles: []

The Problem Statement

Interviewer: "Every time a major event happens, Twitter shows trending topics within minutes. How does Twitter detect that something is trending from a firehose of 500 million tweets per day? Walk me through the architecture."

This question tests three things: whether you can design a real-time stream processing pipeline that handles massive throughput, whether you understand anomaly detection (trending is not just "popular," it is "unusually popular right now"), and whether you can reason about the data quality challenges of spam, manipulation, and geographic localization.

Most candidates describe a word counter. Strong candidates explain why raw counting is insufficient (popular does not equal trending), how sliding windows differ from fixed windows, how anomaly detection separates signal from noise, and how spam filtering prevents gaming the trending list. The best candidates also address personalization and geographic scoping.

Clarifying the Scenario

You: "Before I start, I want to scope this correctly."

You: "When you say 'trending,' do you mean globally trending, or do you also want location-based trending (trending in New York vs. trending worldwide)?"

Interviewer: "Both. Start with global, then explain how you add location awareness."

You: "Got it. And should I focus on hashtags only, or also keywords and phrases that trend without a hashtag? For example, during an earthquake, people tweet 'earthquake' without a hashtag."

Interviewer: "Both hashtags and organic keywords."

You: "One more question: how fast does 'trending' need to surface? Are we talking seconds or minutes?"

Interviewer: "Minutes. A topic should appear in the trending list within 5-10 minutes of starting to spike."

You: "I will structure my answer in four parts: the tweet ingestion and keyword extraction pipeline, the sliding window counting mechanism, the anomaly detection logic that separates 'trending' from 'always popular,' and the spam and manipulation filtering layer."

My Approach

I break the trending detection system into five stages:

1. Ingestion: Consuming the tweet firehose (500M tweets/day, roughly 6000 tweets/second average, bursting to 50,000+ during major events)
2. Extraction: Pulling hashtags, keywords, and named entities from each tweet
3. Counting: Maintaining real-time counts using sliding time windows in a stream processing framework
4. Anomaly detection: Comparing current counts to historical baselines to find topics with unusual velocity
5. Filtering: Removing spam, bot-driven manipulation, and sensitive content from the trending list

The fundamental insight is that "trending" does not mean "most mentioned." The word "the" appears in millions of tweets per day but never trends. "Good morning" spikes every day at 8am but is not interesting. Trending means the rate of mentions for a topic is significantly higher than its historical baseline right now. It is a velocity anomaly, not an absolute count.

This is the mistake I see most often in interviews: candidates design a system that surfaces the most popular topics, not the most accelerating topics. Justin Bieber would be permanently trending in that system. The correct system surfaces topics that are growing unusually fast compared to their own baseline.

The Architecture

The trending detection pipeline is a multi-stage stream processing system. Tweets flow from ingestion through extraction, counting, anomaly detection, and filtering before reaching the trending list that users see.

Here is how a tweet becomes a trend:

Step 1: A tweet enters the firehose. A user tweets "Just felt an earthquake in LA!" The tweet enters Kafka, partitioned for parallel processing.

Step 2: Keyword extraction. The NLP pipeline extracts keywords: "earthquake," "LA." It also detects the language (English) and normalizes the tokens (lowercase, stemming). Hashtags like #earthquake are extracted directly.

Step 3: Windowed counting. The stream processor increments the count for "earthquake" in three sliding windows: 1-minute, 5-minute, and 1-hour. The Count-Min Sketch data structure tracks approximate counts for millions of unique keywords simultaneously with constant memory.

Step 4: Anomaly detection. The detector compares the current 5-minute count for "earthquake" against its historical baseline (average 5-minute count for this time of day and day of week). If "earthquake" normally gets 200 mentions per 5 minutes but currently has 15,000, the z-score is well above 3, and it qualifies as trending.

Step 5: Filtering. The spam filter checks whether the spike is driven by bots or coordinated accounts. The safety filter checks editorial policies. If the topic passes both filters, it enters the trending list.

Step 6: Geographic scoping. The system also tracks location-tagged counts. "Earthquake" is trending in Los Angeles (where tweets are concentrated) before it trends globally (where the signal is diluted by worldwide tweet volume).

Sliding Window Counting with Stream Processing

The counting layer is the computational heart of the system. You need to count how many times each keyword appears in the last N minutes, and you need to do this for millions of unique keywords simultaneously, updating in real time as new tweets arrive and old ones expire.

The challenge is choosing between tumbling windows and sliding windows:

Tumbling windows divide time into fixed, non-overlapping intervals. A 5-minute tumbling window counts from 0:00-5:00, then 5:00-10:00. The problem: a spike that starts at 4:30 and ends at 5:30 is split across two windows. Neither window shows the full magnitude of the spike.

Sliding windows continuously evaluate the last N minutes. At any point in time, the window covers "now minus 5 minutes." This catches spikes regardless of when they start. The cost: you need to track when each event entered the window so you can expire it when it slides out.

The production approach uses hopping windows (a compromise): divide time into small buckets (1 minute each), then sum the last 5 buckets to approximate a 5-minute sliding window. The approximation error is at most 1 minute (the granularity of one bucket), which is acceptable for trending detection.

For the counting data structure, you cannot maintain an exact counter for every possible keyword. With millions of unique tokens, exact counting would require gigabytes of memory per worker. Instead, use probabilistic data structures:

Count-Min Sketch maintains approximate frequency counts using a matrix of hash-based counters. It uses O(1) memory per update and O(1) memory per query, regardless of the number of unique items. The tradeoff: it can overestimate counts (never underestimate), but the error is bounded and configurable.

Space-Saving algorithm maintains an exact top-K list by only tracking the K most frequent items. When a new item arrives that is not in the top-K, it replaces the item with the smallest count. This guarantees finding the top-K items if they exceed a minimum frequency threshold.

In practice, the system uses both: Count-Min Sketch for approximate counting of all keywords (cheap and fast), plus Space-Saving for maintaining an exact top-K candidate list of the most frequent keywords (the trending candidates).

Anomaly Detection for Trend Emergence

Raw counts are not enough. "Good morning" gets millions of mentions every day at 8am. "Justin Bieber" has a consistently high mention count. Neither is "trending" because they are always popular. A trending topic is one that is experiencing an unusual spike relative to its own historical baseline.

The anomaly detection layer compares the current count rate to the expected count rate and flags topics where the deviation is statistically significant.

The z-score formula is the core of the anomaly detector:

z = (current_count - historical_mean) / historical_stddev

For "earthquake": z = (15,230 - 200) / 320 = 47.2. This is massively anomalous. It is trending.

For "good morning": z = (45,000 - 42,000) / 3,500 = 0.86. This is normal daily variance. Not trending.

For "justin bieber": z = (8,500 - 7,800) / 650 = 1.08. This is slightly above average but within normal variance. Not trending.

The threshold (z > 3.0) means a topic needs to be more than 3 standard deviations above its own baseline to qualify as trending. This is deliberately conservative to avoid false positives.

The historical baseline is computed from weeks of data, broken down by hour of day and day of week. "Good morning" has a high baseline at 8am on weekdays but a low baseline at 2am on Sundays. The anomaly detector uses the right baseline for the current time slot.

I also add a minimum absolute count threshold (e.g., 500 mentions in the current window). This prevents obscure topics with very low baselines from trending. If a keyword normally gets 0 mentions and suddenly gets 5, the z-score is infinite, but 5 mentions are not enough to constitute a meaningful trend.

Spam and Manipulation Filtering

Without spam filtering, the trending list becomes a playground for manipulation. Bot networks can generate hundreds of thousands of coordinated tweets to push a hashtag into trending. Spam accounts can flood a keyword to promote products or political agendas. The filtering layer is essential for the integrity of the trending list.

I categorize manipulation patterns into three types:

Bot amplification. Automated accounts generate thousands of tweets with the same hashtag. These accounts often have common characteristics: recently created, few followers, no profile picture, tweets at inhuman speed (dozens per minute), and identical or template-based tweet content.

Coordinated inauthentic behavior. Real-looking accounts tweet the same hashtag at roughly the same time. Unlike bots, each individual account looks legitimate. The signal is the coordination: hundreds of accounts that do not normally interact all tweeting the same thing within a 10-minute window.

Hashtag hijacking. Injecting a commercial or political message into a trending hashtag by including it in unrelated tweets. This does not create a fake trend, but it pollutes the tweet stream for a real trend.

The filtering pipeline runs before the anomaly detector:

Each tweet gets an "organic score" based on the account's reputation (account age, follower count, engagement history), the content's uniqueness (not a near-duplicate of other tweets about the same topic), and the engagement pattern (organic trends generate replies and quote tweets, not just original tweets with the same hashtag).

The trending detector uses weighted counts instead of raw counts. A tweet from a 10-year-old account with 10,000 followers counts more than a tweet from a 3-day-old account with 0 followers. A unique tweet counts more than the 500th copy of the same text. This weighting naturally suppresses bot-driven campaigns even without explicitly identifying bots.

The Tricky Parts

1. Geographic scoping vs. global dilution. An earthquake in LA generates massive tweet volume, but if you only measure globally, the signal gets diluted by worldwide traffic. A topic that is trending locally (and should be shown to LA users) might not register globally. The fix: maintain separate counting windows per geographic region (city, country, worldwide) and run anomaly detection independently at each level. A topic can be "trending in Los Angeles" without being "trending globally."

2. Language and ambiguity. The keyword "apple" could refer to Apple the company, apple the fruit, or a person named Apple. Without disambiguation, the system conflates unrelated spikes. Named entity recognition (NER) helps, but it is imperfect and adds latency. The production compromise: treat ambiguous keywords as a single topic and let the trend surface. If "apple" is trending because of an iPhone launch, the tweet content under the trend will make the context clear.

3. The cold start for new keywords. When a completely new term appears (a new product name, a coined phrase, a newly famous person), it has no historical baseline. The z-score cannot be computed because the mean and standard deviation are zero. The fix: use a default baseline for unknown keywords (average across all keywords of similar frequency), and switch to the keyword-specific baseline once enough history accumulates (24-48 hours).

4. Trending decay and removal. Once a topic stops accelerating (the count rate returns to a new, higher baseline), it should stop trending. But users expect trending topics to persist for at least a few hours. The system uses a "trending half-life": once the z-score drops below the threshold, the topic remains in the trending list with decaying priority for 2-4 hours before being removed. This prevents topics from flickering in and out of the list.

5. Coordinated attempts to suppress trends. Attackers can try to suppress a trending topic by flooding it with spam, triggering the spam filter to suppress the entire hashtag. The fix: the spam filter should suppress individual tweets (reduce their weight), not suppress the topic itself. Even if 80% of tweets about a topic are spam, the remaining 20% of organic tweets still represent a real trend.

What Most People Get Wrong

How I Would Communicate This in an Interview

Here is how I would actually say this:

"Trending detection is fundamentally an anomaly detection problem, not a counting problem. The word 'the' appears in millions of tweets but never trends because it is always popular. A trending topic is one whose mention rate is significantly above its own historical baseline right now.

The pipeline has four stages. First, tweets enter a Kafka topic partitioned for parallel processing. An NLP pipeline extracts hashtags, keywords, and named entities from each tweet.

Second, a stream processor like Apache Flink maintains sliding window counts using hopping windows: 1-minute buckets summed over 5-minute spans. For memory efficiency, I use a Count-Min Sketch for approximate counting of all keywords, plus the Space-Saving algorithm to maintain a top-K candidate list of the most frequent items.

Third, the anomaly detector compares each candidate's current 5-minute count against its historical baseline (average count for this time of day and day of week). If the z-score exceeds 3.0 and the absolute count exceeds a minimum threshold, the topic qualifies as trending.

Fourth, a spam filter weights each tweet by the authoring account's reputation score. Bot-driven campaigns with thousands of low-reputation accounts produce much less signal than the same number of organic tweets. An additional coordination detector flags sudden bursts from accounts that have no social connection to each other.

For geographic trending, I run the same pipeline with separate counting windows per region. A topic can trend in New York without trending globally, which is important for localized events like weather, sports, and local news."

Interview Cheat Sheet

• Trigger: "How does Twitter detect trends?" → "Anomaly detection, not raw counting. Compare current mention rate to historical baseline using z-scores. Z > 3.0 with minimum absolute count = trending."
• Trigger: "What data structure for counting?" → "Count-Min Sketch for approximate frequency of all keywords (fixed memory, O(1)). Space-Saving algorithm for exact top-K heavy hitters."
• Trigger: "Sliding vs tumbling windows?" → "Hopping windows: 1-minute buckets summed to approximate a 5-minute sliding window. Avoids the boundary spike problem of tumbling windows."
• Trigger: "How do you handle spam?" → "Weight tweets by account reputation score (0-1). Bot accounts with low reputation contribute minimally. Coordination detection flags synchronized bursts from unrelated accounts."
• Trigger: "What about geographic trending?" → "Separate counting windows per region (city, country, global). Run anomaly detection independently at each level. A topic can trend locally without trending globally."
• Trigger: "What stream processor?" → "Apache Flink or Storm for real-time processing. Kafka for ingestion buffering. Not MapReduce, which has batch latency."
• Trigger: "Why not just count hashtags?" → "Trending topics often emerge organically without hashtags. 'Earthquake' trends without #earthquake. NLP keyword extraction captures both."
• Trigger: "How fast?" → "Target: topic surfaces in trending list within 5 minutes of spike onset. Stream processing with sub-second event processing latency makes this achievable."
• Trigger: "What about always-popular terms?" → "Historical baselines are time-aware (hour of day, day of week). 'Good morning' has a high baseline at 8am, so its daily spike does not register as anomalous."
• Trigger: "Personalized trending?" → "Overlay the global trending signal with the user's interest graph. Topics trending among accounts they follow are weighted higher. This surfaces niche trends relevant to the individual."

Test Your Understanding

Quick Recap

1. Trending is an anomaly detection problem, not a counting problem. "Popular" and "trending" are fundamentally different: trending means growing unusually fast relative to a topic's own baseline.
2. The pipeline flows through ingestion (Kafka), extraction (NLP), counting (stream processing with sliding windows), anomaly detection (z-scores against historical baselines), and filtering (spam and manipulation removal).
3. Count-Min Sketch provides approximate counting with fixed memory and O(1) operations. Space-Saving maintains the top-K heavy hitters for anomaly detection candidates.
4. Historical baselines must be time-aware (hour of day, day of week) to prevent periodic patterns like "good morning" from triggering false trends.
5. Spam filtering uses weighted counts (account reputation score 0-1) and coordination detection (flagging synchronized bursts from socially unconnected accounts).
6. Geographic trending requires separate counting windows per region with independent anomaly detection at each level.
7. Hopping windows (small buckets summed over a larger span) approximate true sliding windows with bounded error and are simpler to implement.
8. New keywords without historical baselines use a default baseline until they accumulate enough history for keyword-specific anomaly detection.

Related Concepts

• Stream processing architectures covers how Apache Flink, Storm, and Kafka Streams distribute stateful computation across worker nodes, handle exactly-once semantics, and manage checkpointing for fault tolerance.
• Count-Min Sketch and probabilistic data structures explains the mathematical foundations of approximate frequency counting, including error bounds, hash function selection, and the conservative update optimization.
• Time-series anomaly detection covers the broader field of detecting outliers in temporal data, including EWMA, Holt-Winters, and seasonal decomposition methods that go beyond simple z-scores.
• Content moderation and trust and safety explains how platforms detect and mitigate coordinated inauthentic behavior, bot networks, and manipulation campaigns at scale.
• Real-time personalization covers how recommendation systems overlay global signals (trending topics) with per-user interest graphs to produce personalized feeds and trend lists.