Failover-aware model fallback

TL;DR

• Fallback chains define an ordered list of models (e.g., Claude Opus -> GPT-4o -> Claude Sonnet -> cached default). When the primary fails, requests automatically route to the next model in the chain.
• Semantic error classification distinguishes transient failures (timeouts, 429 rate limits) that benefit from fallback from permanent failures (auth errors, billing issues) where fallback wastes money.
• Quality-aware fallback adjusts prompt strategy per model. A prompt optimized for Opus may underperform on Sonnet without adaptation. Maintain per-model prompt variants or simplification rules.
• Production systems using multi-provider fallback report 99.9%+ effective availability even when individual providers experience 1-2 hour outages.
• Proactive health checking with synthetic canary requests detects provider degradation before user traffic is affected, enabling preemptive rerouting.
• Limitation: prompt portability is hard. Prompts optimized for one model produce inconsistent results on another. Fallback reduces downtime but may change output quality.

The Problem It Solves

It's 2 AM. Your production agent processes customer support tickets using Claude Opus as its primary model. Anthropic's API returns 503 errors for 47 minutes during an infrastructure incident. Every customer ticket in that window gets a hard failure. Your SLA promises 99.9% availability (8.7 hours downtime per year). You just burned 47 minutes of your annual budget in a single incident.

This isn't hypothetical. Every major LLM provider experiences multi-hour outages several times per year. OpenAI had 15 notable incidents in 2024. Anthropic, Google, and smaller providers have similar track records. If your agent depends on a single provider, your uptime ceiling is that provider's uptime floor.

I've been on-call during two separate provider outages where the fix was "wait for them to fix it." Forty-five minutes of staring at error dashboards while tickets pile up is enough to convince anyone that single-provider dependency is an unacceptable architecture choice.

What Is It?

Failover-aware model fallback defines an ordered chain of LLM providers and models. When the primary model fails, the system automatically routes the request to the next model in the chain. Each failure is classified by type (transient vs. permanent), and fallback behavior is tailored accordingly: transient errors trigger fallback to the next model, while permanent errors (bad credentials, billing failures) fail immediately.

Think of it as a hospital's backup power system. The primary grid powers everything. If it fails, the backup generator kicks in within seconds, keeping critical systems running. If the generator also fails, battery backup covers the most essential equipment. Each tier provides less capacity but keeps the core function alive. You don't run the backup generator at full capacity, and you don't expect the fallback model to match your primary's quality. But the patient (user request) doesn't die.

How It Works

Semantic error classification: retry vs. fallback vs. fail

The most important decision in a fallback system is knowing which errors deserve fallback and which don't. Blindly retrying every error wastes time and money. The system classifies each failure into one of three buckets:

Transient failures (retry, then fallback): Rate limits (HTTP 429), server errors (503), timeouts, and network errors. These are temporary. The provider will likely recover. Try the same provider once with backoff. If it fails again, fall back to the next model.

Permanent failures (fail immediately): Authentication errors (401, 403), billing errors (402), and invalid request errors (400). No amount of retrying or fallback will fix bad credentials or an unpaid bill. Fail fast and alert the operator.

Semantic failures (fallback with adaptation): Context window overflow, unsupported features, and format errors. These mean the request doesn't fit the current model's capabilities. Fallback to a model with a larger context window, or truncate the request and retry.

def classify_error(error) -> str:
    status = get_status_code(error)
    if status in (401, 403):
        return "auth"          # Permanent: fail immediately
    if status == 402:
        return "billing"       # Permanent: fail immediately
    if status == 429:
        return "rate_limit"    # Transient: retry then fallback
    if status in (500, 502, 503):
        return "server_error"  # Transient: fallback
    if status == 408:
        return "timeout"       # Transient: fallback

    message = str(error).lower()
    if "context" in message and ("window" in message or "length" in message):
        return "context_overflow"  # Semantic: fallback with adaptation
    if "rate limit" in message:
        return "rate_limit"
    return "unknown"  # Default to transient behavior

I've seen production systems that treated 402 (billing) as a transient error and kept retrying across all fallback models, burning through rate limits on three different providers before someone noticed the payment had failed. Classifying errors correctly saves both money and debugging time.

The fallback chain: quality-ordered model routing

A fallback chain is an ordered list of (provider, model) pairs. The system tries each one in order until a request succeeds or the chain is exhausted. The chain is typically ordered by quality (best first), but cost and latency can also influence ordering.