How API rate limiting works end to end

The Problem Statement

Interviewer: "Your API is getting hammered. Some clients are making 10,000 requests per second. Walk me through how you would implement rate limiting end to end, from the algorithm choice to the HTTP headers the client sees, to how it works when you have multiple API gateway instances."

This question tests three things: your understanding of rate limiting algorithms and their tradeoffs, your knowledge of HTTP protocol conventions for communicating rate limits to clients, and whether you can reason about the distributed systems challenges of enforcing limits across multiple gateway nodes.

Most candidates describe a token bucket and stop there. Strong candidates cover the full stack: algorithm selection, header semantics, distributed counter synchronization, graceful degradation for well-behaved clients, and tiered limits per user/plan.

Clarifying the Scenario

You: "Before I dive in, I want to clarify a few things."

You: "When you say 'rate limiting,' are we talking about per-user limits (authenticated API keys), per-IP limits (anonymous traffic), or both?"

Interviewer: "Both. Assume you have authenticated clients with API keys and also anonymous traffic you want to throttle."

You: "Got it. And should I assume we are running multiple API gateway instances behind a load balancer, so the rate limit state needs to be shared?"

Interviewer: "Yes, distributed rate limiting is part of the problem."

You: "OK. Should I also cover burst handling? For example, a client that sends 100 requests in 1 second but normally averages 10 per second?"

Interviewer: "Yes, that is an important edge case."

You: "I will structure my answer in four parts: the rate limiting algorithms and when to use each one, the HTTP headers that communicate limits to clients, the distributed implementation using Redis, and the policy layer that defines tiered limits and burst allowances."

My Approach

I break this into five layers:

1. Algorithm selection: Token bucket, leaky bucket, fixed window, sliding window log, sliding window counter, and when each is appropriate
2. HTTP header contract: How the server communicates rate limit state to clients using standard headers
3. Distributed enforcement: How multiple API gateway instances share counters using Redis with atomic operations
4. Policy and tiering: Per-user, per-IP, per-API-key limits with different tiers (free, pro, enterprise)
5. Client-side behavior: How well-behaved clients use the headers to self-throttle and implement exponential backoff

The mental model I use: rate limiting is a shared resource allocation problem. The "resource" is your API's request processing capacity. The "allocator" is the rate limiter. The "contract" between server and client is the HTTP headers. Without the headers, the client is flying blind and will keep hammering your API even after being limited. With the headers, the client can self-regulate, reducing load on both sides.

Most engineers think of rate limiting as a defense mechanism: "protect the server." But the real value is fairness and predictability. A well-designed rate limiter ensures every client gets their fair share of capacity, noisy neighbors cannot degrade the experience for everyone, and clients can predict and adapt to their quota.

The implementation spans three levels of the stack. At the top, the policy layer defines who gets what limits. In the middle, the algorithm layer enforces those limits with specific accuracy and fairness guarantees. At the bottom, the infrastructure layer (Redis, gateways) makes it work at scale across distributed nodes.

The Architecture

The rate limiting architecture has three tiers: the API gateway (where enforcement happens), the Redis cluster (where counters live), and the policy service (where rules are defined). Understanding how these three layers interact is essential because the most common mistakes come from getting the layer boundaries wrong.

The gateway must make a rate limit decision on every request. This means the rate limit check is on the critical path for every API call. Any latency added by the rate limiter is latency added to every response. This constraint drives the design: the check must be fast (sub-5ms), reliable (if it fails, what happens?), and consistent across gateway instances (no per-instance divergence).

Here is the full architecture:

Here is the request lifecycle through the rate limiting stack:

Step 1: Request arrives at load balancer. The client sends a request with an API key in the Authorization header (or no key for anonymous traffic). The load balancer routes to any available gateway instance.

Step 2: Gateway extracts the rate limit key. The gateway identifies the client. For authenticated requests, the key is the API key or user ID. For anonymous requests, the key is the client IP address. For some APIs, the key might be a combination (user ID + endpoint).

Step 3: Gateway checks the rate limit. The gateway sends an atomic increment-and-check operation to Redis. If the counter is under the limit, the request proceeds. If over, the gateway returns a 429 response immediately without forwarding to the backend.

Step 4: Gateway injects rate limit headers. Whether the request is allowed or rejected, the gateway adds rate limit headers to the response. These tell the client: how many requests they are allowed per window, how many remain, and when the window resets.

Step 5: Client reads headers. A well-behaved client reads X-RateLimit-Remaining and slows down as it approaches zero. A misbehaving client ignores the headers and gets 429 responses with Retry-After telling it when to try again.

Rate Limiting Algorithms: When to Use Each One

This is the core algorithmic decision, and it is the section interviewers probe most deeply. Each algorithm has different properties around burst handling, memory usage, fairness, and accuracy. Most candidates can describe one or two algorithms. Strong candidates explain when each is appropriate and why.

The tradeoff space has four dimensions: accuracy (does it enforce the exact limit or approximate it?), memory (how much state per client?), burst handling (does it allow temporary bursts above the average rate?), and simplicity (how hard is it to implement and debug?). No single algorithm wins on all four dimensions.

Token bucket

The two parameters (bucket size and refill rate) map to intuitive concepts. Bucket size is the maximum burst (how many rapid-fire requests are allowed). Refill rate is the sustained throughput limit. For example, a bucket of 100 with a refill rate of 10/s allows bursts of up to 100 requests but limits sustained traffic to 10 requests per second.

The challenge with token bucket is expressing it as "X requests per minute" for client-facing documentation. A bucket size of 100 with a 16.7/s refill rate translates to approximately 1000 requests per minute sustained, but the burst capacity adds complexity to the client's mental model.

The token bucket is the most widely used algorithm. A bucket starts full of tokens (say, 100). Each request consumes one token. Tokens refill at a fixed rate (say, 10 per second). If the bucket is empty, the request is rejected.

Why it works well: It naturally allows bursts. A client that has been idle accumulates tokens, so it can make a burst of requests (up to the bucket size) without being rejected. But sustained traffic above the refill rate is throttled.

Implementation in Redis:

-- Lua script for token bucket
local tokens = tonumber(redis.call('GET', KEYS[1]) or ARGV[1])  -- bucket_size
local last_refill = tonumber(redis.call('GET', KEYS[2]) or ARGV[4])
local now = tonumber(ARGV[4])
local elapsed = now - last_refill
local refill = math.floor(elapsed * tonumber(ARGV[2]))  -- refill_rate
tokens = math.min(tonumber(ARGV[1]), tokens + refill)    -- cap at bucket_size

if tokens >= 1 then
  tokens = tokens - 1
  redis.call('SET', KEYS[1], tokens, 'EX', ARGV[3])
  redis.call('SET', KEYS[2], now, 'EX', ARGV[3])
  return {1, tokens}  -- allowed, remaining
else
  return {0, 0}  -- rejected, no tokens
end

Fixed window

The simplest algorithm. Divide time into fixed windows (for example, 1-minute intervals). Maintain one counter per client per window. If the counter exceeds the limit, reject the request. Reset the counter when the window expires.

The boundary problem: A client can send 100 requests at 11:59:59 and 100 more at 12:00:01, effectively making 200 requests in 2 seconds while still staying within "100 requests per minute" across two adjacent windows. Fixed windows are easy to implement, but they allow burstiness at the boundary.

Sliding window counter

The production-grade compromise. It combines the current window's count with a weighted portion of the previous window's count. If you are 30% into the current window, the effective count is: current_count + previous_count * 0.70.

This eliminates the boundary spike problem with minimal additional complexity. Most API gateways (Kong, Envoy, AWS API Gateway) use some variant of this.

Sliding window log

Stores the timestamp of every request in a sorted set. To check the rate, count all timestamps within the last N seconds. Perfectly accurate, but memory grows linearly with request volume. At 10,000 requests per second, you are storing 600,000 timestamps per minute per client. Only use this when exact counting matters (billing, compliance).

Leaky bucket

Processes requests at a constant rate, like water leaking from a bucket. Incoming requests queue up; if the queue (bucket) overflows, new requests are dropped. Unlike token bucket, there is no burst capacity. Every request exits at the same rate.

Best for scenarios where the downstream system truly cannot handle bursts: database write queues, message broker ingestion, or webhook delivery.

Algorithm comparison summary

HTTP Headers: The Rate Limit Contract

The HTTP headers are the communication protocol between server and client. Without them, rate limiting is a black box that frustrates developers.

The standard headers

The 429 response

When a client exceeds the rate limit, the server returns HTTP 429 Too Many Requests:

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1681234567
Retry-After: 30

{
  "error": "rate_limit_exceeded",
  "message": "Rate limit of 1000 requests per minute exceeded. Try again in 30 seconds.",
  "retry_after": 30
}

Critical details:

1. Always include Retry-After on 429 responses. Without it, the client has no idea when to retry. Some clients will retry immediately, creating a thundering herd.
2. Include the limit info in the JSON body too, not just headers. Some HTTP client libraries do not expose response headers easily.
3. Use seconds, not a date, for Retry-After when the wait is short. Clients do not need to parse dates for a 30-second wait.
4. Return rate limit headers on every response, not just 429s. This lets clients proactively throttle before hitting the limit.

Client-side behavior with headers

A well-behaved client implements this logic:

on_response(response):
  remaining = response.headers['X-RateLimit-Remaining']
  reset = response.headers['X-RateLimit-Reset']

  if response.status == 429:
    retry_after = response.headers['Retry-After']
    sleep(retry_after)
    retry()
  elif remaining < threshold:
    # Slow down proactively
    delay = calculate_delay(remaining, reset)
    sleep(delay)

The best clients never see a 429 because they read X-RateLimit-Remaining and slow down before hitting zero.

Multi-resource rate limits

Many APIs enforce multiple rate limits simultaneously. For example, the GitHub API has:

• Primary limit: 5000 requests per hour per authenticated user
• Secondary limit: 100 concurrent requests
• Search API limit: 30 requests per minute

Real-world rate limit patterns from major APIs

Different APIs implement rate limiting differently, and studying their patterns reveals practical design decisions:

GitHub API: Uses a primary rate limit (5,000/hour for authenticated, 60/hour for unauthenticated) plus secondary limits on compute-intensive operations (search: 30/min, code search: 10/min). Returns X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, and X-RateLimit-Used. Their GraphQL API uses a cost-based system where each query consumes "points" based on complexity.

Stripe API: Uses a sliding window rate limit of 100 reads/sec and 100 writes/sec in live mode, 25 each in test mode. Returns RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset. Stripe's 429 response includes a recommendation to use exponential backoff with jitter. They also throttle based on account activity, not just request volume.

Twitter/X API: Uses per-endpoint, per-user, and per-app limits. Free tier is extremely limited (1,500 tweets/month). The API returns rate limit info on every response. Twitter was one of the first major APIs to standardize on X-RateLimit headers, which is why the X- prefix convention became widespread.

Slack API: Uses tiered rate limits per method type. Web API methods are limited by "tier" (Tier 1: 1 per minute, Tier 4: 100+ per minute). Slack returns Retry-After in seconds for 429 responses and recommends a specific backoff algorithm in their docs.

The pattern across all major APIs: rate limit headers on every response, clear documentation of the limit tiers, and explicit guidance on how clients should handle 429 responses. When multiple limits apply, include all of them in the response. Some APIs use X-RateLimit-Limit as a JSON object or multiple header instances. The emerging standard (draft-ietf-httpapi-ratelimit-headers) supports multiple rate limit policies:

RateLimit-Policy: primary;l=5000;w=3600, search;l=30;w=60
RateLimit: primary;r=4523;t=1800, search;r=28;t=45

For simpler APIs, pick the most relevant limit (usually the one closest to being exceeded) and include that in the standard headers.

What happens when clients ignore 429

Not all clients are well-behaved. Some retry immediately, some retry in a tight loop, and some are buggy scrapers that never check status codes. For these clients:

1. Exponential backoff enforcement: If a client sends requests while rate-limited, increase the Retry-After duration exponentially with each rejected request. First 429 says "retry in 30s." Fifth consecutive 429 says "retry in 480s."
2. Temporary IP ban: After N consecutive 429s with no backoff, block the IP for a longer duration (15-60 minutes). This is different from rate limiting: it is an abuse prevention measure.
3. Response body shrinking: Return minimal 429 responses (empty body, no JSON) to reduce bandwidth consumed by rejected requests. At 10,000 rejected requests per second, even a small JSON body adds up.

Distributed Rate Limiting with Redis

The hardest part of rate limiting is making it work across multiple gateway instances. If you have 5 gateway instances and a limit of 100 requests per minute, you cannot give each instance a local limit of 20 (because traffic is not evenly distributed). You need a shared counter.

At high throughput (10,000+ requests per second to the same rate limit key), this race condition is not theoretical. It happens on every burst. Without atomicity, your 1000-request limit effectively becomes ~1010-1050, depending on contention.

Redis Lua scripts execute atomically on the Redis server. The entire script runs as a single operation, with no interleaving from other clients. This is how you get distributed atomicity without distributed locks. The Lua script is uploaded once (via SCRIPT LOAD) and called by hash (via EVALSHA), so the overhead is minimal.

Why Lua scripts in Redis?

The rate limit check must be atomic: read the counter, compare to the limit, and increment, all in one operation. If these are separate Redis commands, two gateway instances could both read "999 remaining," both allow the request, and the actual count becomes 1001 (exceeding the limit).

-- Sliding window counter Lua script
local current_key = KEYS[1]           -- "ratelimit:user123:current"
local previous_key = KEYS[2]          -- "ratelimit:user123:previous"
local limit = tonumber(ARGV[1])       -- 1000
local window_size = tonumber(ARGV[2]) -- 60 (seconds)
local now = tonumber(ARGV[3])         -- current timestamp

local current_window = math.floor(now / window_size)
local previous_window = current_window - 1
local window_elapsed = (now % window_size) / window_size

local previous_count = tonumber(redis.call('GET', previous_key) or '0')
local current_count = tonumber(redis.call('GET', current_key) or '0')

-- Weighted count: current + (1 - elapsed_fraction) * previous
local weighted_count = current_count + previous_count * (1 - window_elapsed)

if weighted_count >= limit then
  local reset_at = (current_window + 1) * window_size
  return {0, 0, reset_at}  -- rejected
end

redis.call('INCR', current_key)
redis.call('EXPIRE', current_key, window_size * 2)
local remaining = limit - weighted_count - 1
local reset_at = (current_window + 1) * window_size
return {1, math.floor(remaining), reset_at}  -- allowed

Redis deployment considerations for rate limiting

The Redis instance backing your rate limiter has specific requirements that differ from a typical application cache:

Persistence is optional. If Redis restarts, all rate limit counters reset to zero. Clients briefly get a fresh quota, which might cause a small burst. This is acceptable for most APIs. Do not enable Redis persistence (RDB/AOF) just for rate limiting; the write amplification is not worth it.

Memory sizing. Each rate-limited client needs 2-3 Redis keys (current window, previous window, optionally last-refill timestamp). With 1 million API keys and 100 bytes per key, you need roughly 200-300MB of Redis memory for rate limit data. This is modest even for a single Redis instance.

Latency budget. The rate limit check adds latency to every request. With Redis on the same network, expect 0.5-2ms per check. This is acceptable for most APIs (adding 1ms to a 200ms API call is negligible). If you need sub-millisecond rate limiting, use local in-memory counters with periodic Redis sync.

Cluster vs single instance. For most APIs, a single Redis instance (with a standby replica for failover) handles millions of rate limit checks per second. Redis Cluster is only needed when you have millions of distinct rate limit keys and the single instance's memory or throughput is insufficient. Use hash tags ({user123}:current, {user123}:previous) to ensure both keys for the same client land on the same shard.

Monitoring rate limiter health

The rate limiter itself needs monitoring. Key metrics to track:

• Allow vs reject rate: A sudden jump in 429s usually means a real traffic spike, a misconfigured client, or a deployment that changed retry behavior.
• Redis latency: If the central Redis check goes from sub-millisecond to 10ms+, the rate limiter becomes part of your API latency budget.
• Hot keys: A single API key, tenant, or IP generating massive traffic can create skew even when overall throughput looks normal.
• Fallback path activation: If Redis is unavailable and the gateway falls back to local limits or fail-open behavior, alert immediately.
• Header correctness: Sample responses and verify X-RateLimit-Remaining, X-RateLimit-Reset, and Retry-After stay internally consistent.

Client-side rate limit handling best practices

The server side gets most of the attention, but client-side implementation matters just as much. A poorly designed client can turn rate limiting into a denial-of-service attack on itself.

Exponential backoff with jitter. When a client receives a 429, the retry strategy should use exponential backoff: wait 1s, 2s, 4s, 8s, with a random jitter of up to 50% added to each wait. Without jitter, synchronized clients create thundering herds.

retry_delay = min(base_delay * 2^attempt + random(0, base_delay), max_delay)

Proactive throttling. Do not wait for a 429 to slow down. Read X-RateLimit-Remaining on every response. When remaining drops below 20% of the limit, start adding proportional delays between requests. This keeps the client under the limit and avoids 429s entirely.

Queue and batch. Instead of making API calls individually, queue them and send in batches at a controlled rate. A token bucket on the client side mirrors the server's rate limit, ensuring the client never exceeds its quota. Libraries like Bottleneck (Node.js) and ratelimiter (Python) implement this pattern.

Per-endpoint tracking. If the API has different limits per endpoint, track rate limit state per endpoint. A client that exhausts its search quota should not also stop making read requests that have a separate, higher limit.

Circuit breaker integration. If a client receives multiple consecutive 429s, trip a circuit breaker that stops all requests for a configurable period (e.g., 60 seconds). This prevents persistent hammering when the client is fundamentally exceeding its allocation and needs to be redesigned, not just retried.

Monitoring signals worth alerting on

What happens when Redis is down?

This is the question that separates good answers from great ones. If Redis is unreachable, you have three options:

1. Fail open: Allow all requests through without rate limiting. The API is unprotected but still serving traffic. This is the common choice for non-critical APIs.
2. Fail closed: Reject all requests with a 503 Service Unavailable. The API is down, but you are not risking overload. This is the choice for APIs that protect critical resources.
3. Fall back to local rate limiting: Each gateway instance enforces a local limit of total_limit / num_instances. Not perfectly fair, but provides approximate protection. This is the best production choice.

How production API gateways implement this

Real-world API gateways have battle-tested rate limiting implementations:

Kong uses the rate-limiting plugin with sliding window counter backed by Redis. It supports multiple rate limit windows (e.g., 10 requests per second AND 1000 per hour), per-consumer and per-route limits, and returns standard X-RateLimit headers. Kong also supports a "local" strategy for single-node deployments and a "cluster" strategy using its own PostgreSQL/Cassandra backend.

Envoy implements rate limiting via the ratelimit service, a standalone gRPC service that Envoy calls before forwarding each request. The ratelimit service uses a sliding window counter with Redis. This architecture decouples rate limiting from the proxy, allowing independent scaling. The downside is an extra network hop per request.

The Tricky Parts

1. Clock skew across gateway instances. The sliding window counter uses timestamps. If gateway instances have clocks that are 2 seconds apart, the "current window" calculation disagrees, causing inconsistent rate limiting. Use NTP synchronization and compute windows on the Redis server (using Redis TIME command), not on the gateway.

2. Hot keys in Redis. A single popular API key might generate 50,000 rate limit checks per second, all hitting the same Redis key. This creates a hot key that bottlenecks the Redis shard. Solutions: use Redis Cluster with hash tags to distribute across shards, or batch rate limit checks (check every 10th request against Redis, estimate locally in between).

3. Rate limit by what? The identity key matters enormously. Rate limiting by IP address punishes users behind corporate NATs (thousands of employees sharing one IP). Rate limiting by API key can be bypassed by creating multiple keys. Rate limiting by user ID requires authentication. Most production systems use API key as the primary key, with IP-based limits as a secondary defense for unauthenticated traffic.

4. Distributed rate limits across regions. If your API runs in US-East, EU-West, and AP-Southeast, a global rate limit of 1000/min requires cross-region coordination. Options: global Redis (adds latency), per-region limits (1000/min per region, effectively 3x the global limit), or periodic sync (each region tracks locally and exchanges counts every 5 seconds, accepting some over-limit leakage).

5. Retry storms after 429. If 10,000 clients all hit the rate limit simultaneously and all retry after the same Retry-After duration, you get a thundering herd at reset time. Add jitter to Retry-After: instead of "retry in 30 seconds," return "retry in 25-35 seconds" (randomized per client).

6. WebSocket and streaming connections. Rate limiting HTTP requests is straightforward (one request, one counter increment). But WebSocket connections persist, and a single connection can send thousands of messages. Rate limiting WebSocket traffic requires per-connection message counting or per-connection bandwidth tracking, which is a different problem from request-level rate limiting. Similarly, streaming APIs (SSE, gRPC streams) need per-stream rate limits rather than per-request limits.

7. Internal service-to-service rate limiting. Public API rate limiting protects your service from external clients. But internal microservices can also overwhelm each other. Service A might make 100,000 requests per second to Service B during a batch job, starving Service B's other consumers. Internal rate limiting uses the same algorithms but with service-identity keys and typically higher limits. Service mesh tools like Istio and Linkerd provide this natively.

Rate Limit Policy Design

The rate limiting algorithm is only half the problem. The other half is defining the policies: who gets what limits, on which endpoints, and what happens when they exceed them.

Tiered limits by plan

Most APIs define limits per pricing tier:

The policy is stored in the Policy Service (a simple key-value store, often backed by a database with aggressive caching). When a gateway instance receives a request, it looks up the API key's tier and retrieves the corresponding limits. This lookup is cached locally on each gateway with a 60-second TTL, so tier changes propagate within a minute.

Per-endpoint limits

Some endpoints are more expensive than others. A search endpoint might hit multiple databases and an ML model, while a simple GET endpoint reads from cache. It makes sense to have different limits:

/api/users/{id}        → 5,000/min (simple read)
/api/search            → 100/min   (expensive query)
/api/export            → 10/min    (heavy batch operation)
/api/webhooks          → 1,000/min (moderate)

The rate limiter key becomes a combination of identity + endpoint: ratelimit:user123:/api/search:current. This gives granular control but increases the number of Redis keys. For APIs with many endpoints, group them into "cost tiers" (light, medium, heavy) rather than defining per-endpoint limits.

Graceful degradation strategies

When a client approaches their rate limit, there are several response strategies beyond hard rejection:

1. Hard limit: Return 429 immediately. Simple and clear. Best for most APIs.
2. Soft limit with degradation: Allow the request but return degraded results. For example, a search API might return cached results instead of running a fresh query. The response includes a header indicating degraded mode.
3. Priority queuing: Put over-limit requests in a low-priority queue instead of rejecting them. They are processed when capacity is available, with higher latency. Good for batch/async APIs.
4. Cost-based limiting: Instead of counting requests, count "cost units." A simple GET costs 1 unit, a search costs 10, an export costs 100. The client gets 10,000 units per minute. This is what GitHub uses for their GraphQL API.

What Most People Get Wrong

How I Would Communicate This in an Interview

Here is how I would actually say this:

"Rate limiting has three layers: the algorithm, the HTTP contract, and the distributed enforcement.

For the algorithm, I would use a sliding window counter. It avoids the boundary-spike problem of fixed windows, uses minimal memory (two counters per client), and maps cleanly to 'X requests per minute' semantics. For burst handling, I allow up to 2x the per-second average for short bursts using a burst multiplier.

For the HTTP contract, every response includes X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset headers. When a client exceeds the limit, they get a 429 response with a Retry-After header. This lets well-behaved clients self-throttle before hitting the limit, which reduces load on both sides.

For distributed enforcement, all gateway instances share a Redis counter. The rate limit check runs as a Lua script in Redis for atomicity: increment and check happen as a single atomic operation, so two concurrent requests cannot both see '999 remaining' and both succeed.

The important tradeoff is what happens when Redis is down. I would fail open with a fallback to local in-memory rate limiting at a reduced limit. This is better than either allowing unlimited traffic or taking the API offline entirely.

For the identity key, I use API key for authenticated traffic and IP address for anonymous traffic. I define tiered limits per plan: free tier gets 100 requests per minute, pro gets 1000, enterprise gets 10,000. Each tier has a different burst multiplier.

For monitoring, I track 429 rate as a percentage of total traffic, Redis P99 latency, and the top rate-limited clients. High 429 rates on specific endpoints might indicate the limit is too low or that the endpoint is being abused. High Redis latency means the rate limiter itself is becoming a bottleneck."

Interview Cheat Sheet

• "Which algorithm?" → Sliding window counter for most cases; token bucket if burst handling is critical
• "What headers?" → X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset on every response; Retry-After on 429s
• "How to distribute?" → Redis with Lua scripts for atomic increment-and-check across gateway instances
• "What if Redis is down?" → Fall back to local in-memory rate limiting with reduced limits; fail open, not closed
• "Rate limit by what?" → API key for authenticated clients, IP for anonymous, user ID + endpoint for granular control
• "How to handle bursts?" → Token bucket allows accumulated burst; sliding window counter with burst multiplier
• "What about retry storms?" → Add jitter to Retry-After values to spread retries over time
• "Fixed vs sliding window?" → Fixed has 2x boundary spike; sliding window eliminates it with weighted previous-window count
• "How does the client know?" → Headers on every response let clients self-throttle before hitting 429
• "What about multi-region?" → Per-region limits with periodic cross-region sync, or global Redis with added latency

Test Your Understanding

Quick Recap

1. Use sliding window counters as the default rate limiting algorithm; they eliminate the 2x boundary spike problem of fixed windows with minimal complexity.
2. Return rate limit headers (X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset) on every response, not just 429s, so clients can self-throttle.
3. Enforce rate limits at the API gateway layer, not the application layer, to protect all upstream resources.
4. Use Redis with Lua scripts for distributed rate limiting; the atomic script ensures consistency across multiple gateway instances.
5. When Redis is unavailable, fall back to local in-memory rate limiting with reduced limits rather than failing open or closed.
6. Rate limit by API key for authenticated traffic and by IP address as a secondary defense for anonymous traffic.
7. Add jitter to Retry-After values to prevent thundering herd retry storms.
8. Define tiered limits per pricing plan (free, pro, enterprise) and include the client's specific limit in the response headers.

Related Concepts

• API gateway design: Rate limiting is one of several cross-cutting concerns handled by API gateways alongside auth, routing, and logging.
• Backpressure propagation: Rate limiting is one form of backpressure. In distributed systems, that pressure often interacts with queues, circuit breakers, and load shedding.
• Token bucket for traffic shaping: Token bucket is also used at the network layer (Linux tc, AWS Network Firewall) for bandwidth throttling, using the same fill-rate and bucket-size parameters.
• Circuit breaker pattern: Rate limiting protects the server from clients; circuit breakers protect the client from failing servers. They are complementary patterns often deployed together.
• Load shedding vs rate limiting: Rate limiting is per-client fairness; load shedding is per-server survival. When the server is overloaded regardless of per-client limits, load shedding drops low-priority requests to preserve capacity for high-priority ones.
• Distributed consensus for global counters: The multi-region rate limiting problem is a simplified version of distributed consensus; understanding Raft or gossip protocols helps reason about the sync tradeoffs.