How Twitter detects and hides spam replies in real time

The Problem Statement

Interviewer: "You open a viral tweet and see hundreds of replies. Some are genuine, but many are bots, crypto scams, and self-promotion spam. Twitter hides the spam replies before you scroll. How does the system decide which replies to show and which to hide, in real time, at the scale of 500 million tweets per day?"

This question tests three things: whether you can design a real-time classification pipeline on the write path (classifying content as it is created, not after), whether you understand the feature engineering behind spam detection (user reputation, text similarity, behavioral patterns), and whether you can reason about the adversarial nature of spam (spammers constantly adapt).

Most candidates describe a keyword filter. Strong candidates explain why keywords fail (spammers misspell on purpose), how user-level reputation signals are more robust than content-level signals, why the classification has multiple outcomes (not just spam/not-spam), and how the feedback loop keeps the model current. The best candidates also discuss the tension between aggressive spam removal and accidentally suppressing legitimate minority voices.

Clarifying the Scenario

You: "Before I start, I want to scope this. When you say 'hides spam replies,' are we talking about the full reply ranking pipeline, or specifically the spam classification step?"

Interviewer: "Specifically spam classification. Assume ranking is a separate system. Your job is: given a new reply, decide if it is spam, and if so, what to do with it."

You: "Got it. And by 'real time,' do you mean the classification happens before the reply is visible to other users? Or is there a window where spam is briefly visible before being hidden?"

Interviewer: "Ideally before it is visible. But if that adds too much latency to the write path, a brief window is acceptable."

You: "One more question: what counts as spam here? Just bot-generated content, or also things like self-promotion, crypto scams, harassment, and low-quality replies?"

Interviewer: "All of those. Think of it as a quality classifier. Anything that degrades the reply experience should be caught."

You: "I will structure my answer in four parts: how features are extracted from the reply and the user who wrote it, how the ML model scores the reply, the verdict cascade that decides what action to take, and the feedback loop that keeps the model adapting as spammers evolve."

My Approach

I break the spam detection pipeline into five stages:

1. User reputation scoring (pre-computed): Every account has a reputation score based on account age, follower/following ratio, past content quality, and how often their content gets reported. This score is computed offline and cached, not calculated per-reply.
2. Feature extraction (on write path, 10-30ms): When a reply is created, extract features from the text (embeddings, known-spam-pattern similarity), the user (reputation score, account creation recency), and the behavior (reply velocity, repetition patterns).
3. ML inference (5-15ms): Run the feature vector through a classifier (typically a neural network or gradient-boosted ensemble) to produce a spam probability.
4. Verdict cascade (1-5ms): Based on the spam probability and the user's reputation tier, decide: pass, reduce distribution, hide from non-followers, or block entirely.
5. Feedback loop (async): User reports, moderator reviews, and engagement signals feed back into the training pipeline to keep the model current.

The fundamental insight is that user reputation is the most powerful feature. Text-based features can be gamed (spammers change their text constantly), but reputation is slow and expensive to build. A 3-day-old account with 0 followers replying to a viral tweet with a link is almost certainly spam, regardless of what the text says.

I always tell candidates this: start with the user, not the content. Content is cheap to change. Reputation is not.

The Architecture

The spam detection pipeline sits on the write path, between the moment a user taps "Reply" and the moment that reply becomes visible in other users' timelines. Every reply passes through feature extraction and ML scoring before it is stored and distributed.

Here is the walkthrough. When a user submits a reply, it hits the Tweet API with standard authentication and rate limiting. The feature extractor runs three parallel lookups: the user's pre-computed reputation score from Redis, a text embedding from a Sentence-BERT model, and a pattern match against known spam URLs and phrase fingerprints.

These features combine into a vector that feeds the spam classifier. The classifier outputs a spam probability (0.0 to 1.0), which the verdict engine maps to an action: pass (show normally), reduce distribution (show lower in reply thread), hide (only visible if the viewer clicks "show more replies"), or block (rejected entirely, the user sees an error).

The reply is stored with its spam label as metadata. The timeline service respects this label during fanout. When someone opens the reply thread, the reply ranker filters out hidden replies and orders the visible ones by quality.

For your interview: emphasize that classification happens on the write path, not the read path. This means every user who views the reply thread benefits from the classification done once at write time. If you classified on the read path, you would pay the inference cost for every viewer (millions of times for a viral tweet).

Real-Time Feature Extraction on the Write Path

The quality of the spam classifier depends entirely on the features it sees. Text content alone is insufficient because spammers adapt their text constantly. The strongest signals come from who the user is and how they behave, not what they write.

I want to walk through the most important feature categories, because this is where interviewers drill in:

User reputation (highest signal): A user's reputation score aggregates their entire history on the platform. Account age, follower/following ratio, percentage of past content that was reported or removed, whether they have been suspended before, and how they signed up (email-verified with a real domain vs. temporary email service). This score is recomputed hourly and cached in Redis. At inference time, it is a single cache lookup, essentially free.

Reply velocity and targeting: A human might reply to 5-10 tweets in an hour. A spammer replies to 200 tweets in 5 minutes, specifically targeting viral tweets with high visibility. The velocity counter (replies per 5-minute window) and the targeting pattern (what percentage of replies go to tweets with 10K+ likes) are both strong behavioral signals that are hard for spammers to game without reducing their reach.

Text embedding similarity: Rather than matching exact keywords (which spammers evade with misspellings and Unicode tricks), the system computes a dense sentence embedding and measures cosine similarity to a cluster of known-spam embeddings. "FREE BITCOIN! DM me now!" and "Fr33 B1tc0in!! DM m3 n0w" produce nearly identical embeddings even though they share few exact characters.

URL reputation: Spammers almost always include links. The domain reputation of those links (using a frequently-updated blocklist plus a domain age check) is a strong signal. Links through URL shorteners get extra scrutiny because they obscure the true destination.

The Spam Verdict Cascade

Not all spam is equal, and not all actions are equal. A binary spam/not-spam decision is too crude. The verdict cascade maps the classifier's probability into a range of actions, from full visibility to complete rejection.

The four verdicts serve different purposes:

PASS (score < 0.2, trusted user): The reply is legitimate with high confidence. It shows normally in the reply thread, appears in notifications, and gets full distribution. No action needed.

REDUCE (score 0.2-0.5, or new account): The reply might be fine, but there is some uncertainty. It shows in the reply thread but ranked lower than PASS replies. The original poster still gets notified. This is the "benefit of the doubt" tier, used for new accounts posting their first replies.

HIDE (score 0.5-0.85, or suspicious account): The reply is likely spam. It is stored but hidden from the default reply view. Users can see it by clicking "Show more replies" at the bottom of the thread. The original poster does not get notified. This is the "soft suppression" tier. If the user's future replies improve, their reputation recovers and future replies get better treatment.

BLOCK (score > 0.85, or hard rule match like a known malware URL): The reply is almost certainly spam or harmful. It is rejected entirely. The user sees an error message. A strike is added to their account. Enough strikes trigger a suspension review.

The cascade is important because it handles uncertainty gracefully. A binary spam/not-spam decision forces you to choose between aggressive blocking (high false positives) and permissive passing (spam gets through). The four-tier cascade lets you suppress uncertain content softly (REDUCE/HIDE) without permanently punishing users who might be legitimate.

Adversarial Adaptation and Model Retraining Loop

Spam detection is an arms race, not a deploy-once system. Spammers constantly adapt their techniques to evade the current model. The retraining loop is what keeps the system effective over time. Without it, any model degrades within weeks.

Here is the adversarial feedback loop:

1. Stripe deploys a model that catches 95% of spam
2. Spammers who get caught change their tactics (new text patterns, aged accounts, slower velocity)
3. The 5% of spam that gets through grows as spammers adapt
4. User reports and engagement signals identify the new spam patterns
5. The retraining pipeline incorporates these signals and produces an updated model
6. The new model catches the adapted spam, and the cycle repeats

The signal sources for retraining have different strengths:

User reports are high-signal but low-volume. When someone clicks "Report spam," they are usually right. But only a tiny fraction of spam viewers bother to report. You cannot rely on reports alone.

Engagement signals are implicit labels derived from content performance. A reply that gets 1000 impressions but 0 likes, 0 retweets, and 0 replies is suspicious. A reply that gets engagement, even negative engagement (quote tweets mocking it), is probably not spam. These signals are noisy but abundant.

Moderator labels are the gold standard but expensive. A team of human moderators reviews a sample of borderline cases (replies in the HIDE tier) and provides ground-truth labels. These labels anchor the training data and prevent model drift.

User appeals are critical false-positive signals. When a legitimate user's reply is hidden and they appeal, that appeal (if upheld) teaches the model where its boundary is too aggressive. Without appeal data, the model gradually becomes more aggressive over time because it only learns from confirmed spam, not from accidental suppressions.

The Tricky Parts

1. Adversarial account farming: Sophisticated spammers do not use fresh accounts. They buy or create accounts months in advance, build a small follower base with automated engagement, post a few legitimate-looking tweets, and then activate them for a spam campaign. These "aged" accounts have reasonable reputation scores, which defeats user-reputation-based detection. The behavioral features (sudden reply velocity spike on an account that posted once a month for 6 months) are the key signal for catching farmed accounts.

2. Gray area content: Not all unwanted replies are clear spam. Self-promotion ("check out my new app!"), controversial opinions, persistent but non-abusive criticism, and cultural differences in communication style all live in a gray zone. The model must suppress crypto scams without suppressing a user who genuinely wants to share their project. The REDUCE tier exists specifically for this gray area: lower the visibility without hiding it entirely.

3. Coordinated inauthentic behavior: Some spam campaigns use networks of hundreds of accounts that retweet and reply to each other to boost visibility. Detecting individual accounts is insufficient; you need graph-based detection that identifies clusters of accounts acting in coordination. This requires a separate system (network analysis on the social graph) that feeds signals into the per-reply classifier.

4. Latency on the write path: Adding a classifier to the write path means every reply takes 15-30ms longer to post. For a platform where responsiveness is a core UX metric, this is a real concern. The system must be optimized so that the classifier rarely exceeds 30ms. If the classifier is slow (model server overloaded during a traffic spike), the system should degrade to a fast-path fallback (reputation-only scoring, no text analysis) rather than making the user wait.

5. Appeal volume at scale: If 0.1% of replies are incorrectly hidden, and there are 100 million replies per day, that is 100,000 false positives per day. Even if 1% of those users appeal, that is 1,000 appeals per day that need human review. The appeal system must triage efficiently, prioritizing appeals from users with good reputation scores (more likely to be genuine false positives) over appeals from accounts with prior spam strikes.

What Most People Get Wrong

How I Would Communicate This in an Interview

Here is how I would actually say this in 90 seconds:

"Twitter's spam detection runs on the write path, classifying every reply before it becomes visible. The classifier uses three categories of features: user reputation (account age, follower ratio, past content quality), text signals (sentence embeddings compared against known spam clusters, URL reputation), and behavioral signals (reply velocity, targeting pattern of which tweets they reply to).

User reputation is by far the strongest signal, catching 70-80% of spam. It is pre-computed hourly and served from a Redis cache, so the lookup is fast. Text embeddings handle the remaining cases where reputation is ambiguous.

The key design decision is the verdict cascade. Instead of binary spam/not-spam, the system outputs four tiers: PASS, REDUCE (shown but lower ranked), HIDE (only visible via 'show more replies'), and BLOCK (rejected with account strike). The HIDE tier is the workhorse. It catches the bulk of spam without the false-positive risk of outright blocking.

The system adapts through a feedback loop: user spam reports, engagement signals (zero-engagement replies are suspicious), moderator labels on sampled borderline cases, and appeal outcomes from falsely-suppressed users. The model retrains daily, with a bias audit before each deployment to check for disproportionate false positives across account demographics.

The adversarial challenge is that spammers adapt constantly. When the model catches velocity-based spam, they switch to aged accounts that post slowly. The behavioral features, especially the sudden activation of a dormant account, are what catch these sophisticated campaigns."

Interview Cheat Sheet

• Trigger: "How does Twitter detect spam?" → "Multi-signal classifier on the write path: user reputation (70-80% of signal), text embeddings, and behavioral features. Four-tier verdict cascade: PASS, REDUCE, HIDE, BLOCK."
• Trigger: "Why not just use keyword filtering?" → "Spammers evade keywords within hours using Unicode tricks and misspellings. Sentence embeddings capture semantic meaning regardless of exact characters. But text is secondary to user reputation."
• Trigger: "What is the most important feature?" → "User reputation. Account age, follower ratio, content history, and creation pattern. A 3-day-old account with 0 followers replying to a viral tweet with a link is almost certainly spam, regardless of text content."
• Trigger: "How do you handle uncertain cases?" → "The HIDE tier. Reply is stored but not shown in the default view. Users can access it via 'show more replies.' Lower risk than blocking, and the user's reputation can recover if future replies are legitimate."
• Trigger: "How do spammers adapt?" → "When the model catches one pattern, they switch tactics: aged account farming, slower posting velocity, text variation. The daily retraining loop with multi-source labels is what keeps the model current."
• Trigger: "What about false positives?" → "Bias audit before every model deployment. Check false positive rates across account age, geography, and language. The REDUCE tier provides soft suppression for gray-area cases rather than hard blocking."
• Trigger: "Why classify on write path, not read path?" → "A viral tweet gets millions of views. Classifying each reply once at write time costs N inferences. Classifying at read time costs N times M inferences (M = number of viewers). Write-path classification is orders of magnitude more efficient."
• Trigger: "How do you handle coordinated spam networks?" → "Graph-based detection that identifies clusters of accounts acting in coordination (mutual follows, synchronized posting, retweeting each other). This feeds as a feature into the per-reply classifier."
• Trigger: "What if the classifier is slow during a traffic spike?" → "Degrade to reputation-only fast path. Skip text embedding computation, use cached reputation score only. Less accurate but keeps write latency under 10ms."
• Trigger: "How do you retrain the model?" → "Daily retraining with weighted labels from four sources: user reports (high precision), engagement signals (high volume), moderator reviews (ground truth), and upheld appeals (false positive corrections)."

Test Your Understanding

---

---

---

---

---

---

---

Quick Recap

1. Twitter's spam detection runs on the write path, classifying every reply once at creation time rather than at read time, avoiding millions of redundant inference calls per viral tweet.
2. User reputation is the strongest signal (70-80% of detection power), pre-computed hourly and cached in Redis. Text and behavioral features fill the remaining gap.
3. The verdict cascade has four tiers (PASS, REDUCE, HIDE, BLOCK), handling uncertainty with soft suppression rather than forcing binary decisions.
4. Text embeddings (Sentence-BERT) catch spam regardless of Unicode tricks and misspellings, unlike keyword blocklists that spammers evade within hours.
5. Behavioral features like reply velocity, targeting patterns (replying only to viral tweets), and sudden account activation detect sophisticated spammers who have built aged accounts.
6. The retraining loop runs daily, combining user reports, engagement signals, moderator labels, and upheld appeals to keep the model current against adversarial adaptation.
7. Bias audits before every model deployment check false positive rates across account age, geography, and language to prevent disproportionate suppression of legitimate minority voices.
8. Write-path latency is kept under 30ms through parallel feature extraction, cached reputation scores, and a reputation-only fast path that activates during traffic spikes.

Related Concepts

• Content Moderation Pipeline: The four-tier verdict cascade is the same pattern used by every major platform (Facebook, YouTube, TikTok) for content moderation. The tiers may have different names, but the structure (pass, reduce, suppress, remove) is universal.
• Recommendation Ranking Systems: The reply ranker that orders visible replies by quality uses the same infrastructure as the main timeline ranking system. Spam labels are one input to the ranking model alongside engagement predictions and relevance signals.
• Anomaly Detection and Rate Limiting: The velocity-based behavioral features (replies per time window) use the same sliding-window counters as API rate limiters. The difference is that rate limiters block above a hard threshold, while spam detection uses velocity as one input to a probabilistic model.
• Adversarial Machine Learning: Spam detection is one of the purest examples of adversarial ML in production. Every model improvement triggers spammer adaptation. The retraining loop, bias audits, and multi-source labeling are standard defensive techniques.
• Stream Processing for Real-Time Features: The behavioral feature pipeline (computing reply velocity and targeting patterns in real time) uses the same stream processing infrastructure (Kafka, Flink, or equivalent) as trending topic detection and real-time analytics.