How connection draining prevents dropped requests

The Problem Statement

Interviewer: "Your team is deploying a new version of a backend service. During the rolling update, users start seeing 502 errors for about 10 seconds. The error rate is only 0.3%, but it happens every deployment. What is happening, and how do you fix it?"

This question tests three things: your understanding of how load balancers route traffic to backends, your knowledge of graceful shutdown mechanisms, and whether you can connect the deployment lifecycle to the request lifecycle.

Most candidates jump straight to "add a health check." Strong candidates walk through the exact sequence of events that causes a request to hit a dying server, and then describe connection draining as the systematic fix.

Clarifying the Scenario

You: "Before I dive in, I want to narrow down the setup."

You: "What type of load balancer are we using? An L4 (TCP) or L7 (HTTP)? And are we running on Kubernetes, or is this a traditional setup with an ALB or Nginx in front of EC2 instances?"

Interviewer: "Let's say Kubernetes with an ingress controller, but I want you to explain the general concept that applies to both setups."

You: "Got it. And when you say 502 errors, that tells me the load balancer is sending requests to a backend that is no longer accepting connections. The backend has already shut down, but the LB does not know yet."

Interviewer: "Exactly. Walk me through why that happens and how to fix it."

You: "I will structure my answer in three parts. First, the root cause: the race condition between deregistration and shutdown. Second, the solution: connection draining and its lifecycle. Third, the implementation specifics in both traditional load balancers and Kubernetes."

My Approach

I break this problem into four layers:

1. The race condition: Why naive server shutdown causes dropped requests
2. The draining lifecycle: Stop accepting new connections, finish in-flight requests, timeout, force close
3. Load balancer draining: How ALB deregistration delay, Nginx upstream, and HAProxy drain mode work
4. Kubernetes graceful shutdown: preStop hooks, SIGTERM handling, terminationGracePeriodSeconds, and readiness probe coordination

The mental model I use: think of connection draining like a restaurant closing for the night. You do not kick out customers who are in the middle of eating. You stop seating new tables (remove from LB rotation), let current diners finish their meals (complete in-flight requests), and then close the doors (shut down the process). The 502 errors happen when you lock the doors while people are still chewing.

The Architecture

Here is what happens during a healthy rolling update with connection draining:

1. The orchestrator decides to terminate Server v1 (OLD). Before killing it, the server is removed from the load balancer's rotation.
2. The load balancer stops sending new requests to Server v1. But the 12 requests that are already being processed continue to completion.
3. Meanwhile, Server v2 (NEW) starts up, passes health checks, and joins the LB rotation.
4. Once Server v1 finishes all in-flight requests (or the drain timeout expires), the process shuts down cleanly.

The problem happens when step 1 and step 4 are collapsed into a single "kill the process" command. The server dies while those 12 requests are mid-flight, and each one gets a 502.

The Race Condition That Causes 502s

This is the core of the problem, and I need to explain exactly what goes wrong before showing the fix.

The failure sequence is:

1. T=0s: The orchestrator sends SIGTERM to the old server process.
2. T=0.01s: The process exits immediately (no signal handler, default behavior).
3. T=0.5s: A new client request arrives at the load balancer.
4. T=0.5s: The LB's health check has not run yet (it runs every 5-10s), so the LB still thinks the old server is healthy.
5. T=0.5s: The LB forwards the request to a dead process. Connection refused. 502.

The root cause is a timing gap: the server is dead, but the load balancer does not know yet. Health checks are periodic, not instant. There is always a window where the LB's view of the world is stale.

The Draining Lifecycle

Connection draining solves this by inverting the sequence. Instead of "kill then deregister," you "deregister then drain then kill."

The four phases of connection draining:

Phase 1: Signal and deregister (0-2 seconds) The server receives a shutdown signal. It immediately starts failing health checks (returns 503 on the health endpoint). The load balancer detects the failure and removes the server from rotation. No more new requests arrive.

Phase 2: Drain in-flight requests (2-30 seconds) The server continues processing all requests that are already in progress. A typical HTTP request takes 50-200ms, so most requests finish within 1 second. But some requests are slow: large file uploads, long database queries, or SSE streams.

Phase 3: Timeout enforcement (at drain timeout) If requests are still running after the drain timeout (typically 30 seconds), the server force-closes them. This is a safety net. You do not want a single stuck request to block deployments indefinitely.

Phase 4: Process termination The server process exits with code 0. The orchestrator confirms the instance is gone and proceeds with the next server in the rolling update.

Deep Dive 1: Load Balancer Draining Mechanisms

Different load balancers implement draining differently, and the details matter.

How ALB deregistration delay actually works:

When you deregister a target from an ALB target group, the ALB enters a draining state for that target:

1. The ALB stops sending new requests to the target.
2. Existing connections continue until they complete naturally or the deregistration delay expires.
3. If the target has sticky sessions enabled, the ALB redirects sticky session requests to other healthy targets.
4. After the delay, any remaining connections are force-closed.

The key insight: the ALB tracks connection count at the target level. This is not a guess or a fixed sleep. It is active monitoring of real connections.

Deep Dive 2: Kubernetes Graceful Shutdown

Kubernetes has the most complex draining mechanism because it coordinates multiple systems: the kubelet, the API server, kube-proxy/iptables, and the application itself.

There are two critical race conditions in Kubernetes pod termination:

Race condition 1: Endpoint removal vs SIGTERM

When a pod is deleted, the API server sends two signals in parallel:

• It tells the kubelet to terminate the pod (which sends SIGTERM).
• It removes the pod from the Endpoints object (which tells kube-proxy to update iptables rules).

These happen asynchronously. The SIGTERM might arrive before kube-proxy has updated the iptables rules. If the app shuts down immediately on SIGTERM, traffic is still being routed to it via the old iptables rules.

The fix: add a preStop hook with a small sleep.

lifecycle:
  preStop:
    exec:
      command: ["sleep", "5"]

This 5-second sleep gives kube-proxy enough time to update iptables rules before the app starts its shutdown sequence.

Race condition 2: Readiness probe timing

Even with a preStop hook, external load balancers (like an ALB using target groups with pod IPs) have their own health check interval. If the ALB's health check runs every 10 seconds, it might keep routing traffic to the pod for up to 10 seconds after the pod stops accepting connections.

The fix: set the readiness probe to fail as soon as the pod starts shutting down.

terminationGracePeriodSeconds: 45
containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["sleep", "5"]
    readinessProbe:
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 2
      failureThreshold: 1

A properly instrumented application handles SIGTERM like this:

import signal
import sys

shutting_down = False

def handle_sigterm(signum, frame):
    global shutting_down
    shutting_down = True
    server.stop_accepting()
    server.wait_for_drain(timeout=30)
    sys.exit(0)

signal.signal(signal.SIGTERM, handle_sigterm)

@app.route('/healthz')
def health():
    if shutting_down:
        return '', 503
    return '', 200

Deep Dive 3: Long-Lived Connection Draining

HTTP request/response cycles are the easy case. Each request takes 50-500ms, and draining a server with only HTTP traffic is straightforward. The hard case is long-lived connections.

gRPC streaming connections have similar challenges. A gRPC server can send a GOAWAY frame to signal that clients should open new connections. The gRPC client library handles this automatically.

// gRPC graceful shutdown in Go
server.GracefulStop()  // sends GOAWAY, waits for streams to finish
// If GracefulStop takes too long, force it:
time.AfterFunc(30*time.Second, func() { server.Stop() })

Server-Sent Events (SSE) are the simplest long-lived case. The server can send a custom event telling the client to reconnect, and the EventSource API has built-in reconnection logic with configurable retry intervals.

The Tricky Parts

1. Health check propagation delay: Even after your server starts failing health checks, the LB needs at least one check cycle to detect it. With a 10-second interval and 2 failure threshold, that is 20 seconds of stale routing. Some teams lower the health check interval during deployments, but that adds load to the health check system.

2. Connection reuse and keep-alive: HTTP/2 multiplexes many requests over one TCP connection. "Draining connections" does not mean "draining requests." A single HTTP/2 connection might have 100 concurrent streams. The server needs to stop accepting new streams on existing connections, not just stop accepting new connections.

3. Sticky sessions break draining: If the LB uses cookie-based session stickiness, it might keep routing requests to a draining server because the cookie says so. The LB must override stickiness for draining targets, which ALB does natively but Nginx requires explicit configuration.

4. Database connection pools in the app: When your app shuts down, it also needs to drain its connection pool to the database. Closing a database connection while a transaction is in progress causes that transaction to roll back. The app must wait for all active transactions to commit or roll back before closing the pool.

5. Cascading drain during scale-down: If you are scaling from 10 servers to 5, and you start draining all 5 at once, the remaining 5 servers suddenly get double the load. Always drain one at a time, or limit parallel drain to a fraction of the fleet (Kubernetes maxUnavailable controls this).

What Most People Get Wrong

How I Would Communicate This in an Interview

Here is how I would actually say this:

"The 502 errors happen because of a timing gap between the server shutting down and the load balancer finding out about it. Health checks are periodic, so there is always a window where the LB routes traffic to a dead server.

Connection draining fixes this by inverting the sequence. Instead of kill-then-deregister, you deregister-then-drain-then-kill. The server tells the LB 'stop sending me new requests,' finishes the requests it already has in flight, and only then shuts down.

In practice, this means three things. First, on the LB side, you set a deregistration delay. For ALB, that is the deregistration delay timeout, typically 30 seconds. Second, in Kubernetes, you add a preStop hook with a 5-second sleep to handle the race condition between SIGTERM and endpoint removal. Third, your application needs a SIGTERM handler that stops accepting new work and waits for existing work to finish.

The tricky part is long-lived connections like WebSockets. HTTP request draining happens naturally because requests complete in milliseconds. But a WebSocket connection might stay open for hours. For those, you need application-level drain signaling: the server sends a 'please reconnect' message, and the client reconnects with random backoff to avoid thundering herd."

Interview Cheat Sheet

• When asked about 502 errors during deployment: "That is a connection draining problem. The LB is routing to a dead server because health checks have not caught up."
• On the drain lifecycle: "Deregister from LB, finish in-flight requests, timeout, force close. In that order."
• On Kubernetes specifics: "preStop hook sleeps 5 seconds for iptables propagation, then SIGTERM triggers the app's graceful shutdown handler."
• On drain timeout selection: "P99 request duration plus 10 seconds. For most web services, 30-45 seconds."
• On long-lived connections: "WebSockets need application-level drain signaling. Send a reconnect message with random backoff."
• On rolling update safety: "Drain one server at a time. maxUnavailable=1 in Kubernetes."
• On sticky sessions: "Stickiness must be overridden for draining targets. ALB does this natively."
• On the ALB default: "Default deregistration delay is 300 seconds. Almost always too long. Lower it."
• On HTTP/2: "Draining HTTP/2 means stopping new streams on existing connections, not just stopping new connections."
• On database connection pools: "The app must drain its DB connection pool too. Wait for active transactions before closing."

Test Your Understanding

Quick Recap

1. 502 errors during deployment happen because the load balancer routes requests to servers that have already shut down, before health checks detect the failure.
2. Connection draining inverts the shutdown sequence: deregister from LB, finish in-flight requests, enforce timeout, then terminate.
3. AWS ALB uses "deregistration delay" (default 300 seconds, lower it to 30-60 for typical services).
4. Kubernetes requires a preStop hook (sleep 5 seconds) to handle the race condition between SIGTERM and endpoint removal via kube-proxy.
5. Long-lived connections (WebSockets, gRPC streams) need application-level drain signaling because HTTP-level draining does not help.
6. Drain timeout should be P99 request duration plus a safety margin, not an arbitrary large number.
7. Rolling updates should drain one server at a time (maxUnavailable=1) to avoid overloading remaining servers.
8. Connection draining is proactive (planned shutdown), while circuit breaking is reactive (server misbehaving).

Related Concepts

• Rolling updates and blue-green deployments use connection draining as the mechanism that prevents errors during the transition between old and new versions.
• Circuit breakers complement draining by handling the case where a server becomes unhealthy unexpectedly, rather than being intentionally removed.
• Health checks and readiness probes are the signaling mechanism that load balancers use to detect draining servers.
• Kubernetes pod lifecycle (preStop hooks, SIGTERM, terminationGracePeriodSeconds) is the orchestration layer that coordinates draining across containers, sidecars, and the infrastructure.
• Service mesh drain semantics in Envoy/Istio use the same principles but implement them at the sidecar proxy level, draining the Envoy connection pool separately from the application.