How feature flag percentage rollouts work

The Problem Statement

Interviewer: "Your team is shipping a new checkout flow. Product wants to roll it out to 5% of users first, then 25%, then 100%, with the ability to kill it instantly if conversion drops. How does the feature flag system decide which users see the new flow, and how do you make sure the same user always sees the same version?"

This question tests three things: your understanding of consistent hashing for deterministic user assignment, your knowledge of how feature flag SDKs evaluate rules without adding latency, and whether you can design a system that supports gradual rollout with instant rollback.

Most candidates say "randomly pick 5% of users." That answer misses the critical requirement: consistency. If a user sees the new checkout today, refreshes the page, and sees the old checkout, the experience is broken. Strong candidates explain how hashing produces deterministic, sticky assignments without storing per-user state.

Clarifying the Scenario

You: "Before I jump in, I want to make sure I understand the requirements."

You: "When you say 5% rollout, do you mean 5% of all users globally, or 5% within specific segments like mobile users or users in a particular country?"

Interviewer: "Start with a global 5%, but I would like to hear how targeting by segment works too."

You: "Got it. And does the team need to run an A/B test alongside the rollout, or is this purely a safety mechanism for gradual release?"

Interviewer: "Primarily a safety mechanism, but the analytics team will compare metrics between the two groups."

You: "One more: is the feature flag evaluated on the client side (in the browser or mobile app), or on the server side?"

Interviewer: "Server side. The checkout logic runs on the backend."

You: "OK. I will structure this in three parts. First, how the flag evaluation works using consistent hashing to assign users to buckets. Second, how the rollout percentage increases over time with sticky bucketing. Third, how the kill switch works for instant rollback."

My Approach

I break feature flag percentage rollouts into four layers:

1. Hashing and bucketing: How a user ID maps to a deterministic percentage bucket (0-99) so the same user always gets the same assignment.
2. Rollout progression: How increasing from 5% to 25% to 100% works without reassigning existing users.
3. Targeting rules: How to layer segment-based rules (country, plan, cohort) on top of percentage rollouts.
4. Evaluation performance: How the SDK evaluates flags in under 1ms using local caches and background sync.

The mental model I use: think of it like a lottery where each user has a permanent ticket number between 0 and 99. At 5% rollout, ticket numbers 0-4 see the new feature. At 25% rollout, ticket numbers 0-24 see it. The key insight is that the ticket number never changes for a given user, and increasing the percentage always includes people who were already included.

The Architecture

Here is the flow when a user hits the checkout page:

1. The application code calls flag.enabled('new_checkout', user) on the SDK. This is a local call, no network request. The SDK has the flag configuration cached in memory.

2. The SDK runs the evaluation pipeline: first check the kill switch (is the flag force-disabled?), then check targeting rules (is this user in a targeted segment?), then evaluate the percentage rollout.

3. For the percentage rollout, the SDK hashes the combination of the flag key and user ID to get a number between 0 and 99. If that number is less than the rollout percentage, the user sees the new feature.

4. The evaluation result is logged as an event for analytics. This allows the team to compare conversion rates between the control and treatment groups.

5. Flag configuration updates (like increasing rollout from 5% to 25%) propagate from the admin dashboard through the API to all SDK instances within 30 seconds via background sync.

Consistent Hashing: The Core of Deterministic Assignment

This is the part that makes percentage rollouts work. The hash function must be deterministic, uniformly distributed, and stable across rollout changes.

The implementation is straightforward:

import mmh3  # MurmurHash3

def get_user_bucket(flag_key: str, user_id: str) -> int:
    hash_input = f"{flag_key}:{user_id}"
    hash_value = mmh3.hash(hash_input, signed=False)
    return hash_value % 100

def is_flag_enabled(flag_key: str, user_id: str, rollout_pct: int) -> bool:
    bucket = get_user_bucket(flag_key, user_id)
    return bucket < rollout_pct

Why this works:

1. Deterministic. The same user with the same flag always gets the same bucket. User abc_123 on the new_checkout flag always hashes to bucket 37. No database lookup needed.

2. Uniformly distributed. MurmurHash distributes inputs evenly across the 0-99 range. With 100,000 users, roughly 1,000 users land in each bucket.

3. Sticky across rollout changes. When rollout increases from 5% to 25%, users in buckets 0-4 (who were already in the 5% group) stay in the treatment group. Users in buckets 5-24 join the treatment group. Nobody gets moved from treatment back to control.

4. Independent per flag. The flag key is part of the hash input, so the same user gets a different bucket for each flag. This prevents correlation: users who are in the 5% for new_checkout are not necessarily the same 5% for new_search.

Why the flag key is part of the hash

New engineers sometimes hash only the user ID: hash(user_id) % 100. This creates a hidden correlation problem.

If user abc_123 hashes to bucket 7 using just the user ID, that user is in the bottom 10% for every flag in the system. When you roll out new_checkout to 10% and new_search to 10%, the exact same users see both. This is not random experimentation. It is systematically exposing the same users to all new features while other users never see any.

Including the flag key in the hash input means each flag produces an independent assignment. User abc_123 might be in bucket 7 for new_checkout but bucket 73 for new_search.

Gradual Rollout and Sticky Bucketing

The second hard problem is increasing the rollout percentage without reassigning existing users.

How rollout progression works

When the rollout goes from 5% to 25%, the system needs to guarantee:

1. All users who were in the 5% treatment group stay in treatment
2. An additional 20% of users join the treatment group
3. No user moves from treatment back to control

The 0-based modulo approach handles this naturally. At 5% rollout, buckets 0-4 are in treatment. At 25%, buckets 0-24 are in treatment. Since 0-4 is a subset of 0-24, all original treatment users are still included.

This is not true if you use random assignment or if you change the hash function between rollout stages.

Edge case: rollout decrease

What happens if the rollout goes from 25% back to 10%? Users in buckets 10-24, who were seeing the new feature, now see the old one. This is intentional: a rollout decrease is an emergency response to a bad metric. You want those users to stop seeing the broken feature immediately.

The kill switch is just a special case: setting rollout to 0% instantly removes all users from the treatment group.

Sticky bucketing across identity changes

The hash-based approach depends on having a stable user ID. What happens when:

• A user logs in on a new device (different anonymous ID)
• A user was anonymous (cookie-based ID) and then creates an account (permanent ID)
• A user clears cookies and gets a new anonymous ID

This is where sticky bucketing becomes important. The SDK stores the user's bucket assignment in a cookie or local storage. If the user's identity changes (anonymous to logged in), the SDK can look up the old assignment and maintain consistency.

The typical implementation:

1. Evaluate the flag using the current user ID's hash
2. If the user has a stored override (from a previous identity), use that instead
3. When identities merge (anonymous to logged in), carry the flag assignments forward

A/B Test Integration

Feature flag rollouts naturally create treatment and control groups, which makes them a foundation for experimentation.

How it works

Every flag evaluation is logged as an event: {user_id, flag_key, variation, timestamp}. The experimentation platform consumes these events and joins them with outcome metrics (conversion, revenue, engagement).

The treatment group is all users whose bucket falls below the rollout percentage. The control group is everyone else. Because the assignment is deterministic and sticky, each user is consistently in one group for the entire experiment duration.

The metrics pipeline

I send flag evaluation events to a data warehouse (the same one the analytics team uses). An experiment analysis job runs nightly:

1. Join flag events with conversion events on user ID
2. Compute conversion rate for treatment vs control
3. Calculate statistical significance (typically using a chi-squared test or Bayesian method)
4. Alert the team when significance is reached or when treatment is significantly worse

Guardrail metrics

Beyond the primary metric (conversion rate), I define guardrail metrics that must not degrade:

• Page load time (the new checkout should not be slower)
• Error rate (the new checkout should not throw more errors)
• Revenue per user (even if conversion improves, revenue should not drop)

If any guardrail metric degrades beyond a threshold, the system auto-alerts even if the primary metric looks fine.

Targeting Rules: Beyond Simple Percentages

Real feature flag systems layer targeting rules on top of percentage rollouts. The evaluation pipeline runs rules in priority order:

The evaluation order matters:

1. Kill switch: If the flag is globally disabled, return OFF immediately. This is the emergency stop button. No rules, no percentage, just OFF.
2. Allow-list: Specific users who always see the feature (internal team, beta testers). Useful for testing in production.
3. Deny-list: Specific users who never see the feature (VIP accounts, compliance-sensitive users).
4. Targeting rules: Segment-based rules like "roll out to 50% of US users on the Pro plan." The percentage rollout within a segment uses the same hash-based bucketing, scoped to the segment.
5. Default rollout: The global percentage for all remaining users.

Combining targeting with percentages

A common pattern: roll out to 100% of internal users (allow-list), 50% of US Pro plan users (targeting rule), and 5% of everyone else (default rollout).

The SDK evaluates rules top-to-bottom and returns the first match. This means an internal user in the US on the Pro plan gets the flag from the allow-list, not from the targeting rule. This deterministic priority prevents conflicting assignments.

Flag Evaluation Performance

Every feature flag check must complete in under 1ms. A typical page load evaluates 20-50 flags. If each flag required a network request, you would add 200-500ms of latency to every page load. That is unacceptable.

Local evaluation with background sync

The SDK maintains an in-memory cache of all flag configurations. Flag evaluation is pure local computation: read the flag config from memory, run the rules, hash the user, return the result.

The background sync process updates the cache:

1. On SDK initialization, fetch all flag configs from the API (or CDN)
2. Every 30 seconds, poll for updates (or subscribe to an SSE stream for push-based updates)
3. When an update arrives, atomically swap the in-memory config

This means flag changes propagate within 30 seconds (polling) or 1-2 seconds (SSE push). The tradeoff: a flag change is not instant. If you flip the kill switch at 2:00:00 PM, servers receive the update between 2:00:01 and 2:00:30 PM (with SSE) or 2:00:00 and 2:00:30 PM (with polling).

Handling SDK initialization failure

What if the SDK cannot reach the flag API when the application starts? I configure two fallbacks:

1. Bootstrap from a local file. Ship a flags.json file with the deployment that contains the last known flag config. The SDK loads this file if the API is unreachable.
2. Default values in code. Every flag.enabled() call has a default value: flag.enabled('new_checkout', user, default=False). If the SDK has no config at all, it returns the safe default.

The Flag Lifecycle: From Creation to Cleanup

Feature flags are temporary by design. Treating them as permanent creates a codebase that is impossible to reason about. I enforce a strict lifecycle for every flag.

The four stages

1. Created: Flag is defined with an owner, description, and expected removal date. Default state is OFF. No code references exist yet.
2. Active rollout: Flag is in code, rolling out from 0% to 100%. The owner monitors metrics and adjusts the percentage.
3. Fully rolled out: Flag is at 100% for all users. The feature is live. The flag exists only as a safety net for instant rollback.
4. Cleanup: After the confidence period (typically 2 weeks at 100%), the flag is removed from code. Only the "on" branch is kept. The flag definition is archived.

I set the expected removal date at creation time. An automated job scans for flags past their removal date and creates cleanup tickets. If a flag is 90+ days past its expected removal, it escalates to the engineering manager.

The cost of stale flags

Every stale flag is a code fork. The developer reading the code must understand both branches. Tests must cover both paths. New engineers ask "what does this flag do? Can I remove it?" and nobody knows the answer. I have seen production incidents caused by someone accidentally toggling a 2-year-old flag that controlled a critical code path nobody remembered.

The Tricky Parts

1. Flag key collision in experiments. If two flags use similar keys (checkout_v2 and checkout_v2_alt), their hashes might correlate. Solution: use fully qualified names like team.project.feature and include a salt in the hash.

2. Cross-platform consistency. The mobile app, web client, and backend all need to evaluate the same flag for the same user and get the same result. This requires using the exact same hash algorithm across SDKs in different languages. MurmurHash3 has well-tested implementations in every major language.

3. The "thundering herd" on flag change. When a kill switch flips, all server SDKs fetch the new config simultaneously. If 500 servers all request the config at the same moment, the flag service gets a spike. Solution: jittered polling intervals so servers sync at different times.

4. Stale flags become tech debt. A 6-month-old flag that has been at 100% for 4 months is dead code. Nobody removes it. After a year, you have 200 stale flags adding complexity to every code path. I enforce a flag lifecycle: every flag has an expiration date, and the system alerts when a flag is past its expected removal date.

5. Testing flag combinations. With 50 active flags, a user might have any combination of flags enabled. Testing every combination is impossible (2^50 states). I focus testing on flags that modify the same user flow and establish "compatibility groups" that are tested together.

6. Percentage math with small populations. At 1% rollout with 500 daily active users, only 5 users see the new feature. That is not enough to detect a conversion drop. I always calculate the minimum sample size needed for statistical significance before choosing the initial rollout percentage. If 5% is too few users to detect a 2% conversion change, start higher or accept longer measurement periods.

What Most People Get Wrong

How I Would Communicate This in an Interview

Here is how I would actually say this:

"Feature flag percentage rollouts use deterministic hashing to assign each user to a fixed bucket. The system takes the flag key and user ID, hashes them with MurmurHash to get a number between 0 and 99, and compares that to the rollout percentage. If the hash result is less than the percentage, the user sees the new feature.

The critical property is that this is sticky without any storage. The same user always gets the same bucket for the same flag. When we increase the rollout from 5% to 25%, users in buckets 0 through 4 were already seeing the new feature, and now users in buckets 5 through 24 join them. Nobody gets moved backward.

The flag SDK runs entirely in local memory. It caches the flag configuration and syncs updates in the background every 30 seconds. Flag evaluation is a hash computation and a comparison, which takes about 50 microseconds. There is no network call per flag check.

For the kill switch, flipping a flag to 0% propagates to all servers within 30 seconds via background sync, or within 1-2 seconds if we use server-sent events for push-based updates. The tradeoff is that it is not truly instant, but 1-2 seconds is fast enough for most incidents. For sub-second kill switches, I would use a circuit breaker at the load balancer level that does not depend on SDK sync."

Interview Cheat Sheet

• "How do you decide which users see the feature?" leads to hashing flag_key + user_id with MurmurHash, modulo 100, compare to rollout percentage.
• "How is it sticky?" leads to deterministic hashing produces the same output for the same input, no storage needed, same user always gets the same bucket.
• "What happens when you increase the percentage?" leads to monotonic inclusion: increasing the threshold only adds users, never removes them, because 0-4 is a subset of 0-24.
• "How fast is flag evaluation?" leads to in-memory SDK, no network call, ~50 microseconds per evaluation using local cache with background sync.
• "How fast does the kill switch take effect?" leads to 1-30 seconds depending on sync mechanism (SSE vs polling), not truly instant, circuit breaker for sub-second kills.
• "How do you handle targeting rules?" leads to priority-ordered evaluation: kill switch, allow-list, deny-list, segment rules, default percentage.
• "How do you prevent flag tech debt?" leads to every flag has an owner and expiration date, automated alerts on stale flags past their removal date.
• "What about A/B testing?" leads to flag evaluation events are logged, treatment vs control groups emerge naturally from bucket assignment, analyzed by the experimentation platform.
• "Can the user see the flag config?" leads to server-side evaluation is opaque to the user, client-side evaluation exposes the config (treat it as visible), never put secrets in flag configs.

Test Your Understanding

Quick Recap

1. Feature flag percentage rollouts use deterministic hashing (hash(flag_key + user_id) % 100) to assign users to stable buckets without any storage.
2. Including the flag key in the hash ensures independent bucket assignments per flag, preventing systematic exposure bias.
3. Increasing the rollout percentage only adds users to the treatment group, never removes existing users, because the comparison is a simple threshold check.
4. The SDK evaluates flags locally in memory (~50 microseconds), syncing configuration from the flag service in the background every 30 seconds.
5. Targeting rules are evaluated in priority order: kill switch, allow-list, deny-list, segment rules, default percentage.
6. Sticky bucketing is only needed when user identities change (anonymous to authenticated), not for general flag evaluation.
7. Every feature flag should have an owner, an expiration date, and an automated cleanup process to prevent stale flag tech debt.
8. The kill switch propagates within the sync interval (1-30 seconds); for sub-second kills, use infrastructure-level circuit breakers.

Related Concepts

• Consistent hashing is the same principle used in distributed caching and database sharding, applied here to assign users to rollout buckets.
• A/B testing platforms integrate with feature flags to compare metrics between treatment and control groups that emerge from the bucketing system.
• Canary deployments are the infrastructure equivalent of feature flag rollouts: gradually shifting traffic to new code versions using load balancer weights instead of application-level flags.
• Circuit breakers provide an infrastructure-level kill switch that complements the application-level feature flag kill switch for sub-second response.
• Blue-green deployments are the all-or-nothing alternative to gradual rollouts, useful when feature flags are not feasible (database migrations, schema changes).