How payment systems handle retries without double-charging

The Problem Statement

Interviewer: "A customer clicks 'Pay Now' on your checkout page. Your server sends a charge request to Stripe. The network times out, but the charge actually went through on Stripe's side. Your server retries. Walk me through how you prevent the customer from being charged twice."

This question tests three things: whether you understand that network timeouts are ambiguous (you do not know if the operation succeeded or failed), whether you know how idempotency keys work at the protocol level, and whether you can design a payment state machine that is resilient to partial failures across multiple services.

I see candidates lose points on this one because they jump straight to "just use an idempotency key" without explaining the mechanics. The interviewer wants to see that you understand why the problem exists, how the solution works at every layer, and what breaks when you get the implementation wrong.

The double-charge problem is not hypothetical. It happens in production at scale. Stripe processes hundreds of millions of transactions, and network failures between your server and Stripe's API happen constantly. The entire payment industry's reliability model is built around making retries safe.

To frame why this matters: a double charge is one of the worst things that can happen in e-commerce. The customer sees two charges on their statement, calls their bank, files a chargeback, and you lose the revenue plus a $15-$30 chargeback fee. At scale, unhandled double charges lead to elevated chargeback rates, which can cause your payment processor to flag your account, increase your processing fees, or terminate your merchant agreement entirely. This is a business-critical problem, not just a technical one.

Clarifying the Scenario

You: "Before I dive in, let me make sure I am thinking about the right scope."

You: "Are we talking about a simple single-step charge (like Stripe's PaymentIntents API), or a multi-step payment flow where I need to authorize first and capture later?"

Interviewer: "Start with the single-step case, then I want to hear about multi-step flows and what changes."

You: "Got it. And should I assume we control both the client and the server, or are we also thinking about the bank's deduplication on their side?"

Interviewer: "Assume you control client and server. But I want you to mention where the bank's dedup comes in."

You: "One more thing: should I consider partial failures, like the charge succeeds but writing the order to our database fails?"

Interviewer: "Yes. That is where most people get tripped up."

You: "OK. I will structure my answer in three parts: the two-phase payment state machine that makes each charge trackable, the idempotency key flow that makes retries safe at the API level, and the handling of partial failures when one step in a multi-step payment succeeds but another fails."

My Approach

I break this into four parts:

1. Why retries cause double charges: The fundamental problem is that network timeouts are ambiguous. Your server does not know if Stripe received the request or not. Without a deduplication mechanism, retrying means potentially sending the same charge instruction twice.
2. The two-phase payment state machine: Every payment moves through explicit states (pending, authorized, captured, settled, failed). This state machine is the backbone that makes everything else work.
3. Idempotency keys as deduplication: The client generates a unique key for each payment intent. The server (and Stripe) use this key to detect and deduplicate retries.
4. Partial failure handling: What happens when the charge succeeds at Stripe but your database write fails. This is the hardest part, and the part most candidates skip.

The core insight is this: retries are not the problem. Retries are the solution. The problem is making retries safe by ensuring that executing the same operation twice produces the same result as executing it once. That is the definition of idempotency, and it is the foundation of every reliable payment system.

Let me give you the numbers that make this concrete. Stripe's API has a typical response time of 200-500ms. But network timeouts between your server and Stripe happen at a rate of roughly 0.1-0.5% of requests, depending on your infrastructure. At 100,000 payments per day, that is 100-500 ambiguous payments daily. Without idempotency, each of those could become a double charge. At an average order value of $50, that is $5,000-$25,000 in potential overcharges per day. The business case for getting this right is not theoretical.

The Architecture

Here is the full picture of how a payment moves from the customer's browser through your server to Stripe and back. The critical detail is that every retry carries the same idempotency key, so Stripe can detect and deduplicate at the API gateway level.

Walk through what happens on a retry. The customer clicks "Pay Now," the request times out, and the client retries with the same idempotency key:

1. First attempt: The client generates a UUID (e.g., pay_abc123) and sends it with the charge request. Your server creates a "pending" payment record with this key. Stripe receives the charge, processes it, and stores the result keyed to pay_abc123.
2. Network timeout: Your server never receives Stripe's response. From your perspective, the payment is in an unknown state.
3. Retry (same key): The client retries with the same pay_abc123. Your server sees the pending record and forwards to Stripe with the same key. Stripe's gateway looks up pay_abc123, finds the cached result, and returns it without processing a new charge.
4. Resolution: Your server receives the original result, updates the state to "captured," and confirms to the client.

The customer is charged exactly once. The retry is safe because both your server and Stripe use the idempotency key to detect the duplicate.

I want to call out something that trips up a lot of candidates: the idempotency key and the payment intent ID are different things. The idempotency key is a deduplication token that you generate before the request. The payment intent ID is Stripe's internal identifier for the charge, which you only get back in the response. During a timeout, you have the idempotency key but you might not have the payment intent ID. This is why the idempotency key is the primary lookup mechanism for retries, not the payment intent ID.

Retry Timing: Exponential Backoff with Jitter

When a payment request fails, you do not retry immediately. Immediate retries during an outage create a thundering herd that makes the outage worse. The standard approach is exponential backoff with jitter.

The formula: delay = min(base * 2^attempt + random_jitter, max_delay)

For payment retries specifically:

• Base delay: 1 second
• Max delay: 30 seconds
• Max attempts: 3 (for synchronous retries to the user) or 5 (for background reconciliation)
• Jitter: Random value between 0 and the current delay (full jitter)

import random
import time

def retry_with_backoff(func, max_attempts=3):
    for attempt in range(max_attempts):
        try:
            return func()
        except TimeoutError:
            if attempt == max_attempts - 1:
                raise  # Last attempt, propagate the error
            delay = min(1 * (2 ** attempt), 30)
            jitter = random.uniform(0, delay)
            time.sleep(delay + jitter)

The jitter is critical. Without it, if 1,000 requests fail at the same time (Stripe momentarily unavailable), all 1,000 retry at exactly 1 second, then exactly 2 seconds, then exactly 4 seconds. With jitter, the retries are spread across the entire window, reducing peak load by 50-80%.

For payment retries, I recommend a maximum of 3 synchronous retries (the user is waiting). If all 3 fail, create the payment in "pending_confirmation" state and let the background reconciliation job handle it. The user sees "Payment processing" instead of waiting for more retries.

The Two-Phase Payment State Machine

This is the backbone of reliable payment processing. Every payment must move through explicit states, and each transition must be atomic and auditable. Without a state machine, you cannot reason about what happened during a failure.

The key insight here is the distinction between authorization and capture. In a two-phase payment flow, the bank "holds" the funds during authorization but does not transfer them until capture. This separation gives you a safety window.

If the authorization succeeds but your server crashes before recording it, you can query Stripe's API using the idempotency key to recover the state. If the capture request fails, you can retry it safely because capture is idempotent by nature (you cannot capture the same authorization twice for different amounts).

Here is the practical difference between single-phase and two-phase flows:

I recommend two-phase flows for any payment above $100 or any multi-step checkout. The ability to void an authorization without the customer seeing a charge-and-refund on their statement is worth the added complexity.

The Reconciliation Loop in Detail

The reconciliation loop is the unsung hero of payment reliability. It is a background job that continuously sweeps for payments in ambiguous states and resolves them by querying the source of truth (Stripe).

Here is the exact logic:

# Runs every 30 seconds via cron or task scheduler
def reconcile_stuck_payments():
    # Find payments stuck in transitional states
    stuck = db.query("""
        SELECT * FROM payments
        WHERE status IN ('pending', 'pending_confirmation', 'authorized')
        AND updated_at < NOW() - INTERVAL '60 seconds'
        LIMIT 100
    """)

    for payment in stuck:
        try:
            # Query Stripe for the actual state
            intent = stripe.PaymentIntent.retrieve(
                payment.stripe_intent_id
            )
            # Update local state to match Stripe's truth
            if intent.status == 'succeeded':
                transition(payment, to='captured')
            elif intent.status == 'canceled':
                transition(payment, to='failed')
            elif intent.status == 'requires_payment_method':
                transition(payment, to='failed')
            # If still processing at Stripe, leave it and check next cycle
        except stripe.NotFoundError:
            # Stripe has no record. The charge was never created.
            transition(payment, to='failed')

The key design decisions in the reconciliation loop:

• Batch size limit (100): Prevents the job from overwhelming Stripe's API with queries during a mass failure event.
• 60-second threshold: Gives normal requests enough time to complete before flagging them as stuck. Too short (5 seconds) and you get false positives during slow network conditions.
• Idempotent transitions: Calling transition(payment, to='captured') when the payment is already captured is a no-op. This makes the reconciliation job itself safe to run multiple times.

For your interview: mention that the state machine is not just for tracking payments. It is the mechanism that makes reconciliation possible. Without explicit states and transitions, you cannot build a reconciliation loop that resolves ambiguous payments.

Idempotency Key Storage and Deduplication

The idempotency key is a client-generated UUID that travels with the payment request through every layer. Understanding where and how it is stored is what separates a surface-level answer from a production-level one.

The deduplication happens at two independent layers:

Layer 1: Your server. When a request arrives, your server checks the payment database for the idempotency key. If it finds a completed payment, it returns the cached result immediately without calling Stripe. If it finds a pending payment, it knows a previous attempt is in progress and can either retry the Stripe call (safe, because Stripe also deduplicates) or wait for the reconciliation job.

Layer 2: Stripe's API gateway. Stripe stores every idempotency key for 24 hours. If the same key arrives twice, Stripe returns the cached response from the first request without processing a new charge. This is the last line of defense.

Handling Partial Failures in Multi-Step Payments

This is where the real complexity lives, and where most candidates lose the thread. A payment is rarely a single API call. In production, a typical checkout involves: creating a payment intent, authorizing the card, capturing funds, creating an order record, sending a confirmation email, and updating inventory. When step 3 succeeds but step 4 fails, you have a consistency problem.

Let me be concrete about why this is hard. You have two databases: Stripe's (where the charge lives) and yours (where the order lives). There is no distributed transaction that spans both. You cannot wrap stripe.charge() and db.create_order() in the same ACID transaction. They are independent systems connected by a network that can fail at any point.

The failure modes are asymmetric. If your database write fails, you can retry it (idempotent INSERT with a unique order key). But if the Stripe charge fails, you cannot "un-fail" your database write, you have to roll it back or mark it as "payment_pending." The order of operations matters: charge first means you might take money with no order, and create order first means you might have an order with no payment.

Neither order is perfect, which is why the outbox pattern exists. It decouples the two operations by making the local database write the atomic commit point and handling the external API call asynchronously.

The fundamental tension: the payment processor (Stripe) and your application database are two separate systems. There is no distributed transaction that atomically commits to both. You must design for the case where one succeeds and the other fails.

Here is a table showing the failure permutations and how to handle each one:

The Tricky Parts

1. Clock skew and deduplication windows. Stripe's 24-hour idempotency window is measured by Stripe's clock, not yours. If your server's clock is ahead by 30 minutes, you might think you are within the window when Stripe has already expired the key. Always reconcile ambiguous payments within a few minutes, not hours.

2. Concurrent retries from the same client. A user double-clicks the "Pay Now" button. Two identical requests hit your server simultaneously with the same idempotency key. Your database's unique constraint handles this: one INSERT succeeds, the other fails. But you need to handle the constraint violation gracefully (return a 409 Conflict or wait and return the result of the first request), not crash.

3. Idempotency key reuse across different operations. If the client reuses the same idempotency key for a different amount or different customer, Stripe will return the cached response from the first request. The amounts will not match. Your server must validate that the cached response matches the current request parameters, or reject the retry.

4. Authorization holds and expiry. When you authorize a card, the bank places a hold on the funds. This hold expires after a period (typically 7 days). If you do not capture within that window, the authorization expires and you must re-authorize, which might fail if the customer's balance has changed. Your state machine must handle the authorized → expired transition.

5. Bank-level deduplication. Beyond your server and Stripe, the issuing bank has its own dedup logic based on authorization codes. If two charges arrive with the same amount, merchant, and card within a short window, some banks will flag or decline the second as a potential duplicate. This is a third layer of protection, but it is not reliable or consistent across banks.

6. Currency and amount precision. Payment amounts must be stored in the smallest currency unit (cents for USD, pence for GBP). Storing $50.00 as a floating-point number introduces rounding errors that can cause mismatches between your records and Stripe's records. Stripe uses integer cents: amount: 5000 means $50.00. Always mirror this representation in your database.

7. Webhook vs polling for status updates. Stripe sends webhooks for payment status changes, but webhooks can be delayed, duplicated, or lost. Your reconciliation job should not depend solely on webhooks. It should also poll Stripe's API for payments stuck in ambiguous states. Treat webhooks as an optimization (faster notification), not a guarantee.

What Most People Get Wrong

How I Would Communicate This in an Interview

Here is how I would actually say this:

"The core problem is that network timeouts are ambiguous. When my server sends a charge request to Stripe and the connection times out, I do not know if Stripe processed the charge or not. If I naively retry, I might charge the customer twice.

The solution has three layers. First, the client generates a UUID as an idempotency key before sending the payment request. This key travels with every retry.

Second, my server stores this key in the payment database with a unique constraint. When a request arrives, I check if the key already exists. If it does, I return the cached result. If not, I create a pending payment record and forward to Stripe with the key in the Idempotency-Key header.

Third, Stripe itself stores the key and caches the response for 24 hours. If the same key arrives twice, Stripe returns the original response without processing a new charge.

For the state machine: every payment moves through pending, authorized, captured, and settled states. If a payment gets stuck in pending for more than 60 seconds, a reconciliation job queries Stripe's API to find out what actually happened and transitions the payment to the correct state.

The hardest part is partial failures. If the charge succeeds but my database write fails, I have taken money with no order record. I solve this with the outbox pattern: in a single database transaction, I create the order and write outbox events for capture, email, and inventory. Background workers process these events with their own idempotency keys. Each step is independently retryable."

Interview Cheat Sheet

• Trigger: "How do you prevent double charges?" → "Idempotency keys generated on the client, stored on the server with a unique constraint, and forwarded to the payment processor. Three layers of dedup."
• Trigger: "What if the network times out?" → "Timeout is not failure, it is ambiguity. I query the payment processor to confirm state before retrying."
• Trigger: "Where do you store the idempotency key?" → "In the payment database with a UNIQUE constraint. Not in Redis, not in memory. Durable storage."
• Trigger: "What if two requests arrive simultaneously?" → "The database unique constraint ensures only one INSERT succeeds. The other gets a constraint violation and waits for the first to complete."
• Trigger: "What about multi-step payments?" → "Transactional outbox pattern. Commit the order and outbox events in one database transaction. Background workers handle capture, email, and inventory idempotently."
• Trigger: "How does Stripe handle this?" → "Stripe caches idempotency key to response mappings for 24 hours. Same key returns same response without reprocessing."
• Trigger: "What is the reconciliation loop?" → "A background job that runs every 30-60 seconds, finds payments stuck in pending, queries Stripe for the actual state, and resolves them."
• Trigger: "Authorization vs capture?" → "Authorization holds funds. Capture transfers them. This two-phase approach gives you a safety window to cancel without refunding."
• Trigger: "What if the idempotency key expires?" → "Stripe's 24-hour window. My reconciliation resolves all ambiguous payments within minutes, well before expiry."
• Trigger: "Can you retry a refund?" → "Yes. Refunds are also idempotent with their own key. You cannot refund the same charge twice for the same amount."

Test Your Understanding

Quick Recap

1. Network timeouts are ambiguous: you do not know if the payment processor received your charge request, so you must design every operation to be safely retryable.
2. The client generates the idempotency key (a UUID) before sending the request, ensuring retries always carry the same identifier regardless of which server handles them.
3. Your server stores the key in the payment database with a UNIQUE constraint, providing the first layer of deduplication that prevents concurrent duplicates.
4. Stripe (and other processors) store the key for 24 hours and return cached responses on duplicate requests, providing the second layer of deduplication.
5. Every payment moves through a state machine (pending, authorized, captured, settled, failed), and each transition is recorded so you can always determine the current state.
6. A reconciliation job runs every 30-60 seconds, finding stuck payments and querying the payment processor to resolve ambiguous states, so no payment stays unknown for long.
7. For multi-step flows (charge + order + email + inventory), the transactional outbox pattern ensures atomicity: commit the order and outbox events in one database transaction, and process downstream steps asynchronously with independent idempotency.
8. The fundamental principle: make one system the source of truth (your database for business logic, Stripe for charge state), and treat everything else as an eventually-consistent projection that converges through retries and reconciliation.

Related Concepts

• Idempotency keys (covered in detail in the companion article) are the mechanism that makes retries safe at the API level. Understanding the key lifecycle, storage, and race conditions is essential for implementing payment retries correctly. The idempotency article covers the general pattern applicable to any API, while this article focuses on the payment-specific nuances.
• The saga pattern coordinates multi-step transactions across services using compensating actions. Payment flows with authorization, capture, order creation, and fulfillment are classic saga candidates. If any step fails, the saga runs compensating transactions (void the authorization, cancel the order) to restore consistency.
• The outbox pattern ensures that local database writes and external side effects (like charging a card) happen reliably by writing events to an outbox table inside the same transaction. A background worker reads the outbox and processes the external calls with retry logic. This decouples your database commit from the Stripe API call.
• Exponential backoff with jitter is the retry strategy that prevents thundering herd problems when multiple clients retry simultaneously after a payment processor outage. The jitter component (random delay added to each retry) spreads the load and reduces the probability of synchronized retry storms.
• Circuit breakers protect your payment service from cascading failures when the upstream processor (Stripe, bank network) is degraded. When Stripe starts returning errors above a threshold, the circuit breaker opens and immediately returns errors to clients instead of queueing up requests that will fail, giving Stripe time to recover.
• Event sourcing is an alternative to the state machine approach where every payment state transition is stored as an immutable event. The current state is derived by replaying events. This gives you a complete audit trail and the ability to reconstruct the payment's history at any point, which is valuable for compliance and debugging.