How webhook delivery systems work

The Interview Question

Interviewer: "Your platform needs to notify thousands of external applications whenever a payment completes. Walk me through how you would design a webhook delivery system that handles retries, guarantees at-least-once delivery, and does not overwhelm the consumers. How do Stripe and GitHub do it?"

This question tests whether you understand the full lifecycle of outbound event notifications. The interviewer wants to hear about queue-based delivery, retry strategies with backoff, signature verification, idempotency, and what happens when a consumer is permanently down. Candidates who say "just send an HTTP POST" miss 90% of the real engineering.

What to Clarify Before Answering

You: "Let me clarify the scope before diving in..."

• "Are we building the webhook producer side (sending notifications) or the consumer side (receiving them)? I will cover both, but the design is different."
• "What scale are we targeting? Thousands of events per second (Stripe-scale) or hundreds per day?"
• "Do we need exactly-once delivery semantics, or is at-least-once with consumer-side idempotency acceptable?"
• "Should I cover fan-out scenarios where one event goes to multiple subscribers?"
• "Is the payload sensitive? That affects whether we send the full event or just a notification with an ID for the consumer to fetch."

Why this matters: A webhook system for 100 subscribers sending 10 events per day is a cron job. A webhook system for 100,000 subscribers sending millions of events per day is a distributed system with queues, worker pools, circuit breakers, and dead letter queues. Scoping the answer shows you understand the spectrum.

The 30-Second Answer

A webhook delivery system converts internal events into outbound HTTP POST requests sent to consumer-registered URLs. Events flow from the application into an event queue (SQS, Kafka, or RabbitMQ), where delivery workers pick them up and send HTTP requests to registered endpoints. Each delivery attempt is recorded. If a consumer returns a non-2xx response, the system retries with exponential backoff and jitter, typically up to 5-8 attempts over 24-72 hours. Every request includes an HMAC-SHA256 signature so consumers can verify the payload was not tampered with. A timestamp header prevents replay attacks. After all retries are exhausted, failed events move to a dead letter queue with a dashboard so developers can inspect and manually replay them. At scale, the system adds per-endpoint rate limiting and circuit breakers to avoid overwhelming slow consumers or wasting resources on permanently down endpoints.

The Architecture Overview

The architecture splits cleanly into four layers. The application layer generates events. The queue layer provides durability and decouples event production from delivery. The worker pool handles the actual HTTP delivery with rate limiting and circuit breaking. The observability layer records every attempt for debugging.

I find this separation critical. Without the queue, a spike in events would either overwhelm consumers or block the application from processing new transactions. The queue acts as a shock absorber.

Event Generation and Queueing

The first step in any webhook system is converting an internal state change into a deliverable event. When a payment completes, the service creates an event payload with a unique ID, event type, timestamp, and the relevant data.

The critical design decision is to persist the event to a database before enqueuing it. If the queue goes down momentarily, you can replay from the database. Stripe does this: every event is stored in a durable events table, and the queue is a delivery mechanism, not the source of truth.

Each event gets a unique ID (UUID or ULID). The payload typically includes:

{
  "id": "evt_1NqL7z2eZvKYlo2C",
  "type": "payment.completed",
  "created": 1681234567,
  "data": {
    "object": {
      "id": "pay_abc123",
      "amount": 2500,
      "currency": "usd",
      "status": "succeeded"
    }
  }
}

Fan-out happens at the subscription registry level. One internal event (payment completed) becomes N delivery tasks, one per registered endpoint. This is why the queue matters: if you have 10,000 subscribers, you need 10,000 independent delivery tasks, each with its own retry state.

Delivery Pipeline and Retry Strategy

This is the heart of a webhook system. The delivery worker dequeues a task, constructs the HTTP request, signs it, sends it, and handles the response.

Exponential backoff with jitter

The retry schedule uses exponential backoff to avoid thundering herds. The formula is:

delay = min(base_delay * 2^attempt + random_jitter, max_delay)

The jitter is critical. Without it, if 1,000 deliveries fail at the same time (because a consumer went down for 30 seconds), all 1,000 retries fire simultaneously when the consumer comes back. That is a self-inflicted DDoS. Jitter spreads the retries across a window.

Response code handling

Not all failures are the same:

Signature Verification and Security

Every webhook delivery must be signed so the consumer can verify it came from you and was not tampered with in transit. The standard approach is HMAC-SHA256.

Why include a timestamp?

The timestamp prevents replay attacks. Without it, an attacker who intercepts a valid webhook payload could replay it days later. The consumer checks that the timestamp is within a tolerance window (typically 5 minutes). Any delivery older than that is rejected.

Secret management

Each consumer gets a unique webhook secret when they register their endpoint. This means a compromised secret for one consumer does not affect any other consumer. Secrets should be at least 32 bytes of cryptographic randomness, displayed once at registration, and rotatable without downtime.

Stripe supports secret rotation by allowing two active secrets simultaneously. During rotation, the producer signs with both the old and new secret, sending two signatures in the header. The consumer can verify against either one. After a grace period, the old secret is revoked.

Idempotency: Handling Duplicate Deliveries

At-least-once delivery means consumers will sometimes receive the same event twice. Network timeouts are the most common cause: the consumer processes the event and returns 200, but the producer times out before receiving the response and retries.

Consumers must handle duplicates. The standard pattern is:

1. Extract the event ID from the payload (e.g., evt_1NqL7z2eZvKYlo2C)
2. Check if the event ID exists in a local processed-events table
3. If it exists, return 200 immediately without reprocessing
4. If it does not exist, process the event, then insert the ID into the table
5. Return 200

Dead Letter Queues and Debugging

When all retry attempts are exhausted, the event moves to a dead letter queue (DLQ). This is not a black hole. It is a debugging tool with a user-facing dashboard.

A good DLQ dashboard shows:

Stripe's approach is instructive. Their dashboard shows every webhook event with its delivery status, every attempt with the response code and latency, and a "Resend" button. When a developer is integrating, they can see exactly why their endpoint is failing (usually a 500 with a stack trace in the response body) and fix it without contacting support.

Fan-Out at Scale

At Stripe or GitHub scale, a single event can trigger thousands of deliveries. A popular repository with 5,000 installed GitHub Apps means one push event generates 5,000 independent webhook deliveries.

Architecture for high fan-out

The key optimization is separating event creation from delivery fan-out:

1. Event ingestion: Write the event once to the events table
2. Fan-out stage: Query the subscription registry, create one delivery task per subscriber, enqueue all tasks
3. Delivery stage: Workers process tasks independently

The fan-out stage can be parallelized. For 10,000 subscribers, you batch the enqueue operations (SQS supports SendMessageBatch with up to 10 messages per call, so 1,000 batch calls). Kafka handles this naturally with topic partitioning.

Per-endpoint rate limiting

You cannot send 1,000 events per second to a consumer running on a single $20/month VPS. Each endpoint gets a rate limit (typically 100-500 requests per minute). Deliveries that exceed the limit are delayed, not dropped. The rate limiter uses a token bucket per endpoint URL.

Circuit breaking

If an endpoint fails 5 consecutive deliveries, the circuit breaker trips. While the circuit is open, deliveries are queued but not attempted. Every 60 seconds, the circuit enters a half-open state and probes with a single delivery attempt. If it succeeds, the circuit closes and normal delivery resumes. If it fails, the circuit stays open.

This prevents the system from wasting millions of HTTP requests on endpoints that are permanently down. It also protects consumers from being hammered during an outage, which would make recovery harder.

What Happens When Things Break

Performance Characteristics

How This Compares to Alternatives

I reach for webhooks when I need to notify external applications about events in near real-time. Polling works for simple integrations where the consumer only needs periodic updates. Server-Sent Events are ideal for browser-based dashboards. WebSockets are for truly interactive, bidirectional communication like chat or collaborative editing.

Interview Cheat Sheet

• When asked about delivery guarantees: "Webhooks provide at-least-once delivery. Events are persisted to a database first, then enqueued for delivery. Consumers must be idempotent because duplicates are inevitable."
• When asked about retries: "Exponential backoff with jitter. Stripe retries 8 times over 3 days. Jitter prevents thundering herds when a consumer recovers from an outage."
• When asked about security: "HMAC-SHA256 signature over the raw payload with a per-consumer secret. A timestamp header prevents replay attacks. Consumers verify within a 5-minute tolerance window."
• When asked about scale: "Fan-out separates event creation from delivery. Each subscriber gets an independent delivery task in a queue. Per-endpoint rate limiting and circuit breakers protect both the producer and consumers."
• When asked about failures: "Dead letter queue captures events that exhaust all retries. A developer dashboard shows delivery logs, response codes, and a manual replay button. This is a product feature, not just an ops tool."
• When asked about idempotency: "The producer includes a unique event ID. Consumers store processed IDs in a database with a TTL matching the retry window. Duplicate deliveries are silently acknowledged."
• When asked about alternatives to webhooks: "Polling is simpler but higher latency. SSE and WebSockets require persistent connections. Webhooks are the standard for server-to-server event notification because they work across firewalls and scale independently."
• When asked about ordering: "Webhooks do not guarantee ordering. Events can arrive out of order due to retries and parallelism. Consumers should use the event timestamp, not arrival order, and handle out-of-order delivery gracefully."

Test Your Understanding

Quick Recap

1. Webhook systems convert internal events into outbound HTTP POST requests delivered to consumer-registered URLs.
2. Events are persisted to a durable store first, then enqueued for delivery, ensuring no data loss if the queue fails.
3. Delivery workers sign the payload with HMAC-SHA256 and send the HTTP request with a timestamp header for replay protection.
4. Failed deliveries are retried with exponential backoff and jitter, typically 5-8 attempts over 24-72 hours, with response-aware scheduling.
5. Consumers must be idempotent, using the event ID to deduplicate, because at-least-once delivery guarantees duplicates will occur.
6. Dead letter queues capture events that exhaust all retries, surfaced in a developer dashboard with manual replay buttons.
7. Per-endpoint rate limiting and circuit breakers protect both the producer and consumers at scale.
8. Webhooks are the standard for server-to-server event notification because they provide near real-time delivery and work across firewalls.

Related Concepts

• Message queues (SQS, RabbitMQ, Kafka): The backbone of webhook delivery infrastructure, providing durable event buffering and ordered processing.
• Circuit breaker pattern: The same pattern used in microservice communication applies to outbound webhook delivery, preventing resource exhaustion on failing endpoints.
• Idempotency in distributed systems: The consumer-side requirement for handling duplicate webhook deliveries is the same idempotency pattern used in payment systems and API design.
• Event-driven architecture: Webhooks are the external-facing manifestation of internal event-driven systems, bridging the gap between your event bus and third-party consumers.
• API rate limiting: The per-endpoint rate limiting in webhook systems uses the same token bucket and sliding window algorithms as inbound API rate limiting, applied in the outbound direction.