How idempotency keys prevent duplicate operations

The Problem Statement

Interviewer: "Your API accepts a POST request to create an order. The client sends the request, the server creates the order, but the response is lost due to a network timeout. The client retries. Now there are two identical orders in the database. How do you prevent this?"

This question tests whether you understand the fundamental unreliability of network communication, how idempotency works as a protocol-level guarantee, and whether you can implement deduplication that handles race conditions, expiry windows, and different HTTP methods correctly.

I find this is one of the most practical interview questions because every production API faces this problem. It is not theoretical. Every time a mobile app retries a failed request, every time a webhook delivery system resends a payload, every time a payment processor resubmits a charge, the system must decide: is this a new request, or a retry of something I already processed?

The answer is idempotency keys, and the details of how they work are what distinguish a junior answer from a senior one.

Here is the scope of the problem. Any API that accepts POST requests is vulnerable. Every mobile app that retries after a timeout. Every webhook system that redelivers on failure. Every payment processor that resubmits on network errors. Every message queue consumer that re-processes after a crash. The pattern is universal, and the solution is the same everywhere: give each logical operation a unique identifier, and use that identifier to detect and suppress duplicates.

To put numbers on it: a typical production API with 10,000 requests per second experiences network timeouts at a rate of 0.1-0.5%. That is 10-50 potentially duplicated operations every second. Over a day, that is 864,000-4,320,000 potential duplicates. Without idempotency keys, even a small fraction of those becoming actual duplicates means thousands of duplicate orders, double charges, or repeated side effects daily.

Clarifying the Scenario

You: "Before I walk through the solution, let me clarify a few things."

You: "Are we talking about a general-purpose API (like a REST API for order creation), or a specific domain like payments where the consequences of duplicates are financial?"

Interviewer: "Start general, then I want you to touch on how it applies to payments."

You: "Got it. Should I assume the client is under our control (like our own mobile app), or could it be a third-party integration sending webhooks?"

Interviewer: "Both. I want to see how the design changes."

You: "And for the storage layer, should I design for a single database, or do we need to consider a distributed setup?"

Interviewer: "Single database is fine for the core design. Mention what changes at scale."

You: "OK. I will structure my answer in three parts: the idempotency key lifecycle from client generation through server deduplication and response caching, the race condition handling when two concurrent requests arrive with the same key, and the deduplication window design including TTL and cleanup."

My Approach

I break this into five parts:

1. Why retries cause duplicates: Network communication has no built-in exactly-once guarantee. A timeout tells you nothing about whether the server processed the request.
2. The idempotency key as a contract: The client generates a unique identifier for each logical operation. The server uses this identifier to detect and deduplicate retries.
3. The server-side lifecycle: Check key, execute operation, store result, return cached result on retry.
4. Race conditions: What happens when two requests with the same key arrive simultaneously. This is the part that separates a production implementation from a whiteboard sketch.
5. Deduplication windows and cleanup: How long to keep keys, when to expire them, and what happens when a key is used after expiry.

The mental model I always come back to is this: an idempotent operation produces the same result whether you execute it once or ten times. The idempotency key is how the server knows that request #2 is the same logical operation as request #1, not a different operation that happens to look the same.

The Architecture

Here is the full flow of how an idempotency key travels through the system. The client generates the key, the server stores it with the result, and retries get the cached response without re-executing the operation.

The lookup flow has exactly three outcomes:

1. Key not found: This is a new request. Execute the operation, store the key with the response, return the response.
2. Key found, status = completed: This is a retry of a successfully completed request. Return the cached response without re-executing.
3. Key found, status = in_progress: This is a concurrent retry while the first request is still processing. Either wait and poll, or return a 409 Conflict.

The third case is the tricky one, and I will cover it in the race conditions deep dive.

For your interview: always mention all three lookup outcomes. Most candidates only think about cases 1 and 2. The interviewer is specifically testing whether you think about the concurrent case.

How Different APIs Implement It

Before diving into the lifecycle, let me show how real production APIs expose idempotency:

The pattern is identical across all of them: client sends a unique key, server stores it, retries return the cached response. The only differences are the header name and TTL. When you explain this in an interview, naming 2-3 real implementations shows you have production experience, not just textbook knowledge.

The Idempotency Record Schema

Here is what the idempotency store looks like at the database level:

CREATE TABLE idempotency_keys (
    key         VARCHAR(255) PRIMARY KEY,
    status      VARCHAR(20) NOT NULL DEFAULT 'in_progress',
    -- 'in_progress', 'completed', 'failed'
    request_hash VARCHAR(64),
    -- SHA-256 of critical request params, for mismatch detection
    response_code INTEGER,
    response_body JSONB,
    created_at  TIMESTAMP NOT NULL DEFAULT NOW(),
    completed_at TIMESTAMP,
    CONSTRAINT unique_key UNIQUE (key)
);

CREATE INDEX idx_idempotency_created ON idempotency_keys (created_at);
-- For TTL cleanup queries

The request_hash column is important. It stores a hash of the critical request parameters (amount, user_id, product_id). When a retry arrives, the server compares the hash of the retry's parameters against the stored hash. If they do not match, the client is reusing a key for a different operation, and the server rejects the request.

The response_code and response_body together form the cached response. When a retry hits a completed key, the server returns these values directly, reconstructing the exact HTTP response the client would have received on the original request.

The Idempotency Key Lifecycle

This deep dive walks through every stage of the key's life, from generation to expiry. Understanding this lifecycle is what lets you reason about edge cases correctly.

Walk through the critical stages:

Stage 1: Key generation (client side). The client generates a UUID v4 before making the request. This is a random 128-bit identifier with a collision probability so low it is effectively zero (1 in $2^122$). The key must be generated before the network call, not after, so retries reuse the exact same key.

Stage 2: Key lookup (server side). When the request arrives, the server checks the idempotency store. This must happen before any business logic executes. If the key exists with a completed status, skip all processing and return the cached response.

Stage 3: Lock acquisition. If the key is new, the server inserts it with status = in_progress. This INSERT uses a UNIQUE constraint on the key column. If two concurrent requests arrive with the same key, only one INSERT succeeds. The other gets a constraint violation, which is the server's signal that a concurrent request is already processing.

Stage 4: Business logic execution. The winning request proceeds to execute the actual operation (create order, charge card, etc.).

Stage 5: Result caching. After the operation completes, the server updates the idempotency record with status = completed and the full response body. Future retries will return this cached response.

Stage 6: TTL expiry. A background job cleans up old idempotency records after 24-48 hours. This prevents the idempotency store from growing unbounded.

Handling Race Conditions on Concurrent Retries

This is the section that interviewers care about most, because it reveals whether you have actually implemented this pattern or just read about it. The race condition occurs when two identical requests (same idempotency key) arrive at the server within milliseconds of each other.

The scenario: a mobile user on a flaky connection taps "Submit Order." The app's retry logic fires after 500ms of no response. Both the original request and the retry hit your load balancer and get routed to different server instances. Both servers try to process the same order. Without proper concurrency handling, you get two orders.

This is not an edge case. In a system processing 10,000 requests per second with a 1% timeout rate, you get 100 potential concurrent duplicates per second. The race condition handling must be rock-solid.

The race condition has a simple resolution, but you need to pick a strategy and commit to it.

Choosing the Right Deduplication Window

The deduplication window (TTL) determines how long the server remembers an idempotency key. Set it too short and late retries create duplicates. Set it too long and your storage grows unbounded. This tradeoff is where real-world design decisions happen.

Here is how different systems handle it:

The key factors for choosing your TTL:

Retry frequency. Automated retries (with exponential backoff) typically complete within 5-15 minutes. A 24-hour window gives ample margin.

Human investigation. If a support agent might manually retry a failed operation hours later, you need a longer window. Stripe's 24-hour window covers the scenario where a developer investigates a failed charge the next morning.

Storage cost. Each idempotency record is small (key + status + response, typically 1-5KB). At 1 million requests per day with 24-hour TTL, you store roughly 5GB. At 48 hours, 10GB. These are trivial numbers for a modern database.

Cleanup strategy. Use a background job that runs every few minutes and deletes records older than the TTL. Alternatively, if your database supports it (like PostgreSQL with pg_cron or DynamoDB with TTL), let the database handle expiry natively.

A Visual Summary of TTL Tradeoffs

Pick your TTL based on who retries and when. For automated machine retries (webhooks, queue consumers), 5-10 minutes is sufficient. For APIs where humans are involved (checkout flows, support retry), 24 hours. For financial APIs with regulatory audit requirements, 48-72 hours.

The Tricky Parts

1. Idempotency for different HTTP methods. GET is naturally idempotent (reading does not change state). PUT is naturally idempotent (replacing the full resource with the same data is a no-op). DELETE is idempotent (deleting an already-deleted resource returns 404, not an error). POST is the only method that typically needs an explicit idempotency key, because POST creates new resources and there is no built-in dedup.

2. Idempotency vs. safety. A safe operation has no side effects (GET). An idempotent operation can have side effects, but repeating it does not change the outcome. DELETE is idempotent but not safe (the first call deletes the resource). Understanding this distinction helps you explain why POST needs special treatment.

3. Partial execution and cleanup. If the operation fails midway (after the idempotency key is inserted as in_progress but before the business logic completes), the key is stuck. You need a reaper job that detects stale in_progress records (e.g., older than 5 minutes) and either retries the operation or marks the key as failed so the client can retry with the same key without hitting the "in progress" block.

4. Response serialization. The cached response must be exactly the same format as the original. If your API returns timestamps, and the cached response has a different timestamp than what a fresh execution would produce, the client might be confused. Store the complete serialized response (headers and body) exactly as it was returned the first time.

5. Scope of idempotency. An idempotency key should cover one logical operation, not one HTTP request. If creating an order involves charging a card and sending a confirmation email, the key covers all of it. On retry, the server should not re-charge the card or re-send the email. This means your business logic needs to be checkpointed, so the retry can skip already-completed sub-steps.

6. Header naming conventions. There is no universal standard for the header name. Stripe uses Idempotency-Key. AWS uses x-amzn-client-token. Google Cloud uses request-id. Shopify uses X-Shopify-Idempotency-Key. If you are designing a new API, I recommend Idempotency-Key (Stripe's convention) because it is the most widely recognized and self-documenting.

7. Testing idempotency. In your integration tests, always include a test that sends the same request twice with the same idempotency key and verifies: (a) both responses are identical, (b) the side effect happened only once (one database row, one charge, one email). This is the most important test for any idempotent endpoint, and it catches the majority of implementation bugs.

Here is a concrete test structure that covers the critical paths:

def test_idempotency_deduplicates_order_creation():
    key = str(uuid4())
    # First request creates the order
    r1 = client.post("/orders", json=order_data,
        headers={"Idempotency-Key": key})
    assert r1.status_code == 201
    order_id = r1.json()["order_id"]

    # Second request (same key) returns cached response
    r2 = client.post("/orders", json=order_data,
        headers={"Idempotency-Key": key})
    assert r2.status_code == 201
    assert r2.json()["order_id"] == order_id  # Same order!

    # Only one order exists in the database
    assert db.query("SELECT COUNT(*) FROM orders").scalar() == 1

def test_idempotency_rejects_key_reuse_with_different_params():
    key = str(uuid4())
    client.post("/orders", json={"amount": 50},
        headers={"Idempotency-Key": key})
    r = client.post("/orders", json={"amount": 75},
        headers={"Idempotency-Key": key})
    assert r.status_code == 422  # Parameter mismatch

def test_concurrent_requests_same_key():
    key = str(uuid4())
    # Send two requests simultaneously
    import concurrent.futures
    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
        futures = [
            executor.submit(client.post, "/orders", json=order_data,
                headers={"Idempotency-Key": key})
            for _ in range(2)
        ]
        results = [f.result() for f in futures]
    # Both succeed with the same order ID
    assert results[0].json()["order_id"] == results[1].json()["order_id"]
    assert db.query("SELECT COUNT(*) FROM orders").scalar() == 1

These three tests cover the happy path (deduplication works), the error path (key reuse rejected), and the race condition (concurrent requests handled). If all three pass, your idempotency implementation is correct for the vast majority of production scenarios.

What Most People Get Wrong

How I Would Communicate This in an Interview

Here is how I would actually say this:

"The core problem is that network failures make retries necessary, but retries on non-idempotent operations (like POST) can create duplicates. The solution is idempotency keys.

The client generates a UUID before sending the request and includes it in the Idempotency-Key header. The server uses this key as a deduplication identifier.

When the request arrives, the server does an INSERT into an idempotency table with a UNIQUE constraint on the key column. If the INSERT succeeds, this is a new request, and the server executes the operation normally. After it completes, it stores the response alongside the key.

If the INSERT fails with a unique violation, this is a retry. The server looks up the existing record. If the operation already completed, it returns the cached response. If it is still in progress (concurrent retry), it either waits briefly or returns 409.

The critical design decisions are: the key is generated by the client (not the server) so retries across multiple server instances carry the same key. The UNIQUE constraint resolves race conditions atomically. And the records have a 24-hour TTL so the table does not grow unbounded.

This is exactly how Stripe's API works. They accept an Idempotency-Key header on every POST request, cache the response for 24 hours, and return the cached result on retries. It turns an at-least-once retry mechanism into effectively-once semantics."

Interview Cheat Sheet

• Trigger: "How do you prevent duplicate orders on retry?" → "Client-generated idempotency key in the request header, server-side storage with a UNIQUE constraint, and cached responses for retries."
• Trigger: "Where is the key generated?" → "Always on the client, before the network call. This ensures retries carry the same key even if they hit different server instances."
• Trigger: "What about race conditions?" → "The database UNIQUE constraint on the key column resolves concurrent inserts atomically. One succeeds, the other gets a constraint violation and falls back to the cached response."
• Trigger: "How long do you keep the keys?" → "24-hour TTL. Covers automated retries, manual investigation, and aligns with Stripe's production implementation. Cleanup via background job or database-native TTL."
• Trigger: "What about GET and PUT?" → "GET is naturally idempotent (no state change). PUT is naturally idempotent (full replace). Only POST and PATCH need explicit idempotency keys."
• Trigger: "What if the operation fails midway?" → "The key is stuck in 'in_progress' state. A reaper job detects stale records after 5 minutes and marks them 'failed,' allowing the client to retry."
• Trigger: "How is this different from exactly-once?" → "True exactly-once is impossible in distributed systems. Idempotency keys give you effectively-once: at-least-once retries combined with server-side dedup."
• Trigger: "What does the idempotency record look like?" → "Key (unique), status (in_progress/completed/failed), response body (serialized JSON), created_at (for TTL), and optionally a request fingerprint for parameter validation."
• Trigger: "Can two different users have the same key?" → "With UUID v4, the collision probability is negligible (1 in 2^122). In practice, scope the key to the user or API key for extra safety."
• Trigger: "Real-world example?" → "Stripe accepts Idempotency-Key on every POST. AWS SQS uses MessageDeduplicationId. Shopify uses X-Shopify-Idempotency-Key. Same pattern, different headers."

Test Your Understanding

Quick Recap

1. Network timeouts are ambiguous, so clients must retry, but retries on POST endpoints create duplicates without explicit deduplication.
2. The client generates a UUID v4 as the idempotency key before each request, ensuring retries always carry the same identifier regardless of server routing.
3. The server stores the key in a database table with a UNIQUE constraint, which resolves concurrent duplicate requests atomically at the database level.
4. On retry, the server checks the idempotency store: if the key exists with a completed status, it returns the cached response without re-executing the operation.
5. Race conditions (two concurrent requests with the same key) are resolved by the UNIQUE constraint, where one INSERT wins and the other catches the constraint violation and waits or returns 409.
6. Idempotency records have a TTL (typically 24 hours) and are cleaned up by a background job, database-native TTL, or time-partitioned table drops.
7. GET, PUT, and DELETE are naturally idempotent. POST (and sometimes PATCH) are the methods that need explicit idempotency keys, because they create new resources or trigger non-repeatable side effects.
8. The combination of at-least-once retries with server-side idempotency dedup gives you effectively-once semantics, which is the practical substitute for truly-impossible exactly-once delivery.

Related Concepts

• Payment retry handling covers how Stripe, PayPal, and bank APIs use idempotency keys as part of a broader retry strategy including state machines, reconciliation loops, and exponential backoff. Payments are the most critical use case for idempotency, where getting it wrong means double-charging customers.
• The outbox pattern complements idempotency keys by ensuring that local database writes and external API calls happen atomically, preventing the partial-success problem where the charge succeeds but the order record fails. The outbox event is processed by a background worker that uses its own idempotency key when calling external services.
• Exponential backoff with jitter is the retry timing strategy that prevents thundering herd problems when many clients retry simultaneously after an outage. Without jitter, retries synchronize and amplify the load on the recovering service. Full jitter (random delay from 0 to max) provides the best load distribution.
• Optimistic concurrency control (using version numbers or ETags) solves a related but different problem: preventing conflicting updates to the same resource, rather than preventing duplicate creation. Idempotency handles "I want to do this exactly once" while OCC handles "I want to update this without losing someone else's changes."
• The saga pattern handles multi-step distributed transactions where each step needs its own idempotency, and failed steps require compensating actions (like refunds) to unwind partial progress. Each saga step is an idempotent operation with its own key, and the saga coordinator tracks which steps have completed.