How graceful degradation keeps your system working when dependencies fail

The Problem Statement

Interviewer: "Your recommendations service goes down. Walk me through what happens to the rest of your e-commerce site. Does checkout break? Does the homepage break? How did you design the system so that one dependency failing does not take down the whole experience?"

This question tests three things at once. First, do you understand that not all features are equally critical? Second, can you explain circuit breakers, fallback hierarchies, and bulkheading without turning it into a Wikipedia recitation? Third, do you have the instinct to distinguish what must work from what is nice to have.

A weak answer says "we would just retry the call" or "we would return an error." A strong answer describes a tiered resilience model where the system keeps serving the most important flows, degrades gracefully on secondary features, and recovers automatically when the dependency comes back. I have seen candidates nail the happy-path architecture and completely fumble this question because they never thought about what happens when things go wrong.

Clarifying the Scenario

You: "Before I start, a few clarifying questions to make sure I scope this right."

You: "When you say the recommendations service is down, do you mean it is returning errors, or is it just slow? Those two failure modes require different responses."

Interviewer: "Assume it is returning 500 errors intermittently, about 60% of requests failing."

You: "Got it. And is the current implementation synchronous? Meaning the product page blocks on the recommendations call before rendering?"

Interviewer: "Yes. The page waits for recommendations before returning HTML."

You: "Should I focus just on recommendations, or design a general degradation strategy that works across multiple services?"

Interviewer: "Start with recommendations, then generalize."

You: "One more thing: should I also cover what happens when the recommendations service is slow but not down? That is actually the harder failure mode to handle."

Interviewer: "Yes, cover that too."

You: "OK. I will structure my answer in four parts: feature criticality tiers, the circuit breaker and fallback hierarchy that handles failures automatically, the stale-data strategy for secondary features, and load shedding for when the whole system is under pressure. Then I will briefly cover testing since untested degradation paths fail exactly when you need them most."

The clarifying question about latency versus total outage is the one most candidates skip. Raising it unprompted signals that you understand the difference between fail-fast and timeout-starvation failure modes.

My Approach

I break this into four areas:

1. Feature criticality tiers: Classifying which features must work versus which can degrade versus which can be dropped silently
2. Circuit breakers and feature flags: The runtime mechanisms that detect failures and serve alternatives automatically or on-demand
3. Fallback hierarchy and stale data: Serving cached or default content rather than an error page when live data is unavailable
4. Load shedding under systemic stress: Dropping low-priority requests actively when the system is overloaded

The core mental model is a traffic light. Green means everything is working. Yellow means a dependency is unhealthy and the system has fallen back to plan B. Red means the dependency is completely gone and the system is running on plan C, which still serves the most critical user journeys. What you want to avoid is the dependency failure cascading into a global outage.

Before any circuit breaker or fallback logic can work, you have to classify your features. I use three tiers. Tier 1 means must-work: checkout, auth, product details. If any of these fail, the business loses money directly. Tier 2 means should-work-can-degrade: recommendations, reviews, live inventory counts. The page still works without them. Tier 3 means nice-to-have: personalisation badges, view-count animations, recently-viewed widgets. These get dropped silently the moment the system is under pressure.

For your interview: state your tier classification out loud before describing any technology. Saying "first I classify features by criticality: must-work, can-degrade, drop-silently" signals architectural maturity in your first sentence. Most candidates jump straight to circuit breakers and lose the thread.

The Architecture

Here is the full graceful degradation architecture for an e-commerce platform with five services.

The architecture has three layers of protection. The circuit breaker at the gateway level prevents calls to unhealthy services from going out at all. The fallback tier serves cached or default content when a live call cannot happen. The feature flag store lets you disable entire features at runtime without a deploy.

Walking through the request path for a product page load during a recommendations outage:

1. The browser sends GET /product/123 to the API gateway.
2. The gateway checks the circuit breaker state for the recommendations service. If the error rate in the last 60 seconds exceeds 50%, the breaker is open.
3. With the breaker open, the gateway skips the recommendations call entirely. No timeout, no blocked thread. The failure returns in microseconds.
4. The gateway calls Redis for the cached recommendations for this user. If a cached result exists within its TTL (10 minutes), it returns that.
5. If the cache has nothing for this user, it falls through to the static top-sellers list for this product's category.
6. The catalog and checkout calls proceed normally (Tier 1, never degraded). The page renders with accurate product details, working checkout, and slightly stale or generic recommendations.
7. The user gets a functional page in under 200ms. They cannot tell anything is wrong.

For your interview: the key point is that the user never sees an error page. The page loads, the core experience works, and only the non-critical features are degraded. This is the difference between graceful degradation (shed features) and total failure (shed users).

Deep Dive 1: Feature Flags and Circuit Breakers for Degradation

The two main runtime mechanisms for graceful degradation are circuit breakers and feature flags. They solve different problems. Circuit breakers react to observed failures automatically. Feature flags let operations engineers intervene manually. You need both.

Circuit breaker state machine

A circuit breaker sits between a caller and a dependency. It has three states, and understanding the transitions is what separates a shallow answer from a deep one.

In the Closed state, all calls go through and the breaker counts failures. When the error rate crosses the threshold, the breaker opens. In the Open state, calls fail immediately without touching the dependency. This is the critical part: it prevents a slow or crashing service from consuming your threads, connection pool, or timeout budget. After a configured wait window, the breaker moves to Half-Open and lets a single probe request through. Probe succeeds, breaker closes. Probe fails, wait window resets.

The math matters here. Without a circuit breaker, if your recommendations service has a 5-second timeout and is fully down, every product page request blocks for 5 full seconds before timing out. At 1,000 requests per second with 5-second timeouts, you accumulate 5,000 blocked threads simultaneously. That is how a slow downstream service takes down an otherwise healthy upstream through thread exhaustion. The circuit breaker cuts that to zero: once open, calls return in microseconds.

Feature flags for runtime degradation

Feature flags solve a problem that circuit breakers cannot: the case where the dependency is technically healthy but you want to disable the feature anyway. A recommendations model that just deployed bad results returns 200 OK, so the circuit breaker stays closed. But you want to fall back to top-sellers immediately.

A feature flag store (LaunchDarkly, Unleash, or a simple Redis key) lets an on-call engineer flip a switch and disable the recommendations call entirely at runtime, without a deploy. The gateway checks the flag before making the outbound call.

For your interview: say you combine circuit breakers (automatic failure detection) with feature flags (manual operator control). They are complementary, not alternatives. Circuit breakers handle the "service is broken" case. Feature flags handle the "service is up but producing wrong results" case.

Deep Dive 3: Shedding Load Under Stress

Graceful degradation handles single-dependency failures. Load shedding handles the more dangerous scenario: the entire system is overloaded and you need to decide which requests to drop to keep the most critical flows working.

Picture a flash sale or a viral product launch. Requests spike to 10x normal volume. Your servers are at capacity. You have two options: let everything run at 20% quality, or give 100% capacity to checkout and drop lower-priority traffic entirely. The right answer is almost always the second one. A checkout flow that works for 70% of users is better than a checkout flow that technically works for 100% of users at 8-second response times.

When the system is under extreme load, you need to decide what to shed. I categorize every feature into one of four tiers:

During a partial outage, the system sheds features in reverse tier order: Tier 3 goes first, then Tier 2, then Tier 1. Tier 0 is protected at all costs, even if it means redirecting all available capacity to checkout and authentication.

The load shedding thresholds are based on system metrics: CPU utilization, memory pressure, P99 latency, and error rate. When CPU crosses 70%, the system starts shedding Tier 3 features. When it crosses 85%, Tier 2 goes. When it crosses 95%, only checkout and authentication remain.

The implementation works through a central "degradation controller" that publishes the current degradation level. Each service checks this level before executing features:

// Pseudocode for degradation-aware feature execution
degradation_level = degradation_controller.get_current_level()  // 0, 1, 2, or 3

if feature.tier > (3 - degradation_level):
    return feature.fallback_response()
else:
    return feature.execute_normally()

Bulkhead pattern: isolate failure domains

The bulkhead pattern prevents one service's resource consumption from starving others. The name comes from ship design: separate watertight compartments prevent a hull breach in one section from sinking the whole vessel.

Without bulkheads, a single slow service starves all other services. With bulkheads, the recommendations service can consume 100% of its allocated thread pool while checkout's thread pool remains completely untouched. A bulkhead is the physical enforcement of your tier classification.

The simplest implementation is a separate ExecutorService per downstream dependency in Java, or a separate connection pool per downstream service in Python. Each pool has a hard cap. Overflow gets rejected immediately with a fallback result, not queued indefinitely. I always implement bulkheads alongside circuit breakers: circuit breakers prevent calls to failing services, bulkheads prevent a slow-but-not-failed service from crowding out everything else.

Deep Dive 2: Stale Data vs No Data

When a dependency fails, you have two choices: serve no data (empty section, spinner, error toast) or serve stale data (the last successful result, cached from minutes or hours ago). The instinct is to prefer fresh data or nothing. That instinct is wrong for most user-facing features.

Amazon's product page is the canonical example. If the recommendations service is down, the page does not show an empty white box or a spinner. It shows top-sellers in that category, which might be hours out of date. The user does not care. They wanted to see product suggestions and they got product suggestions.

Netflix has the same model for genre rows. If the personalisation service is unavailable, the app shows popular titles in each genre. The row is full, the experience is intact, the user never sees a loading spinner.

The four-level fallback hierarchy

Level 1 is the live service call. Level 2 is a warm Redis cache holding the last successful response keyed by user ID, with a 10-minute TTL. Level 3 is a cold cache in blob storage populated every hour by a batch job, keyed by product category. Level 4 is a static list compiled into the application binary that requires no network call whatsoever.

The stale-while-revalidate pattern connects Levels 1 and 2. When a live call succeeds, the response is written to cache. When the next request hits within the TTL, you serve the cached result immediately and trigger an async background refresh. The user never waits. The data stays reasonably fresh.

Not all data is safe to serve stale. This is the most important nuance in this whole topic. Recommendations staleness of 10-60 minutes is invisible. Inventory count staleness of 30 seconds can result in overselling. Checkout price staleness of 1 second is a billing error. The rule: data that affects money or safety cannot use cached fallbacks. Data that affects convenience can.

The Tricky Parts

1. Cascading circuit breakers. Service A depends on Service B, which depends on Service C. If C fails, B's circuit breaker trips, which causes A's circuit breaker to trip. Now three services appear "down" when only one is actually broken. The fix: correlate circuit breaker events across the dependency graph. If B's breaker tripped because of C (not because B itself is unhealthy), A should only shed the features that require C, not everything from B.

2. Fallback data consistency. When the recommendation service recovers and the circuit breaker closes, the page suddenly switches from cached recommendations to live ones. If the user reloads the page during the transition, the product listing changes completely. This is disorienting. Use a fade-in strategy: when the breaker closes, blend cached and live results for a brief period before fully switching to live data.

3. Testing degradation paths. You cannot wait for production outages to test your fallback logic. Netflix's Chaos Monkey deliberately kills services in production to exercise degradation paths. Without regular "failure injection" testing, your fallbacks rot: cached data formats change, default response generators bit-rot, and the degradation controller's thresholds drift from the actual capacity limits.

4. The "too-eager breaker" problem. If your circuit breaker threshold is too sensitive, the breaker trips during normal variance (a brief network hiccup causes 3 failures in a row). Once the breaker is open, the fallback serves stale data even though the downstream service recovered in 500ms. The minimum request count threshold (at least 20 requests before evaluating the failure rate) prevents this, but tuning it requires real traffic data.

5. Communicating degradation to users. Do you tell the user that recommendations are degraded? If you show a banner ("Recommendations may not be personalized"), you make users anxious about a problem they would not have noticed. If you say nothing, users might report "the recommendations seem off." The Netflix approach: say nothing for Tier 2/3 features. Only communicate actively for Tier 1 degradation (e.g., "Search results may be limited" during a search outage).

What Most People Get Wrong

How I Would Communicate This in an Interview

Here is how I would actually say this:

"Graceful degradation means shedding features, not users. When a dependency fails, the system continues serving a functional, reduced experience instead of returning error pages.

I would implement this with three components. First, circuit breakers on every dependency call. When the failure rate exceeds 50% over a 10-second window, the breaker trips open and stops making calls to the failing service. This prevents cascade failures and frees up threads immediately.

Second, every dependency gets a pre-planned fallback chain. For recommendations, the fallback is cached personalized data, then top sellers by category. For reviews, it is cached reviews, then aggregate ratings. For pricing, there is no cache fallback because stale prices are dangerous, so I would hide the price and show a 'check for current price' link.

Third, I tier features by business criticality. Checkout, authentication, and payment are Tier 0 and never degrade. Recommendations and reviews are Tier 2 and get shed early. The system automatically sheds tiers based on real-time CPU, latency, and error rate thresholds.

The key design principle: every fallback decision is made at development time, not during the outage. The system knows exactly what to do when any dependency fails, and it does it in milliseconds without human intervention."

Interview Cheat Sheet

• Trigger: "How do you handle partial failures?" → "I use circuit breakers with pre-planned fallback responses. The system sheds features by criticality tier, not randomly."
• Trigger: "What is a circuit breaker?" → "Three states: closed (normal), open (fail fast, return fallback), half-open (probe to test recovery). Trips when failure rate exceeds a threshold over a sliding window."
• Trigger: "How do you decide what to degrade?" → "Four tiers: Tier 0 (checkout, auth) never degrades. Tier 3 (A/B tests, social proof) sheds first. Automatic triggers based on CPU and latency thresholds."
• Trigger: "What about retries?" → "Retries amplify load on a failing service. Circuit breakers are better because they stop sending traffic entirely and serve fallbacks instead."
• Trigger: "Can you cache everything?" → "Only data that is safe to serve stale. Recommendations, yes. Prices, no (revenue risk). Inventory, no (overselling risk)."
• Trigger: "What about timeouts?" → "Timeouts are necessary but not sufficient. A 5-second timeout still holds a thread for 5 seconds. Circuit breakers fail in microseconds because they never make the call."
• Trigger: "How does Netflix do it?" → "Every service call has a fallback. Hystrix (now Resilience4j) wraps every dependency. Chaos Monkey tests degradation paths by killing services in production."
• Trigger: "How do you test graceful degradation?" → "Chaos engineering. Inject failures in staging and production (Chaos Monkey, Gremlin). If you only test the happy path, your fallbacks rot."
• Trigger: "What about communicating failures to users?" → "Silent for Tier 2/3 features (users do not notice). Active communication only for Tier 1 degradation (search limited, delayed updates)."
• Trigger: "How do you recover from degradation?" → "Gradual ramp-up. The circuit breaker goes half-open, allows 10% of traffic, then 25%, then 50%, then closes. Prevents thundering herd on recovery."

Test Your Understanding

Quick Recap

1. Graceful degradation means shedding features, not users, so the system stays usable during partial outages.
2. Circuit breakers detect failing dependencies and stop sending traffic to them, preventing cascade failures and freeing resources.
3. The circuit breaker state machine has three states: closed (normal), open (fail fast with fallback), and half-open (probe for recovery).
4. Every dependency needs a pre-planned fallback chain: cached data, then default responses, then hide the feature, then error message.
5. Not all data can be served stale. Recommendations are safe. Prices and inventory are not.
6. Feature criticality tiers (0-3) define what to shed first and what to protect at all costs.
7. Automatic shedding based on real-time health metrics reacts in milliseconds, while a human reacts in minutes.
8. Recovery should be gradual (ramp traffic from 10% to 100%) to prevent thundering herd on a service that just recovered.

Related Concepts

• Circuit breaker pattern covers the detailed implementation of the three-state (closed, open, half-open) mechanism including failure detection algorithms, timeout strategies, and integration with service mesh proxies like Envoy.
• Bulkhead isolation explains how to partition system resources (thread pools, connection pools) so that a failing dependency can only consume its allocated share, preventing one bad service from starving all others.
• Chaos engineering covers the practice of deliberately injecting failures into production systems to test degradation paths, validate fallback logic, and build confidence in resilience mechanisms.
• Health checks and readiness probes explains how load balancers and orchestrators (Kubernetes) determine whether a service instance is healthy enough to receive traffic, which is the foundation for automatic traffic shifting during degradation.
• Retry storms and exponential backoff covers why naive retry strategies amplify failures during outages and how backoff with jitter prevents synchronized retry waves across distributed clients.