Debugging a production incident

TL;DR

• Every production incident starts with three questions: What broke? When did it start? What changed?
• Triage severity first (P0-P3), because severity dictates who joins, how fast you respond, and whether you wake people up.
• Isolate the blast radius before you debug. Know which users, regions, and services are affected.
• Mitigate first, root-cause later. Rolling back a deploy takes 2 minutes. Debugging it takes 30.
• Communicate early and often. Silence during an incident is more damaging than saying "we are investigating."

---

The Three-Question Framework

When a page fires at 2 AM, most engineers immediately start reading logs. That is the wrong first move. Before you touch a dashboard, answer three questions:

1. What broke? Not the technical cause, the user-visible symptom. "Checkout is returning 500s" is better than "the database is slow." You need the user-facing description because that determines severity.
2. When did it start? Get an exact timestamp from your alerting system. "Around midnight" is not good enough. You need "2024-11-15T00:12:34Z" because you will correlate that timestamp against deploy logs, config changes, and cron schedules.
3. What changed? Check the deployment log, the feature flag dashboard, and the infrastructure change log. In my experience, 70-80% of production incidents correlate with a recent change. If a deploy went out 8 minutes before the alert, you have your leading hypothesis.

These three answers contain the incident. A deploy 10 minutes before the alert usually explains what happened. A dependency status page showing degradation explains it without any log reading at all. I have seen incident durations cut from hours to minutes simply because the on-call engineer checked the deploy log before opening Kibana.

Here is how the three questions drive your investigation:

The framework is not optional. I use it every single time, even when I think I already know what happened. Skipping it leads to tunnel vision, where you chase one theory for 45 minutes when the answer was in the deploy log the whole time.

---

Triage Priority Levels

Not every incident deserves the same response. Waking up the VP of engineering for a broken admin tooltip is wrong. But so is treating a payment outage like a P2. Severity determines escalation, communication cadence, and time-to-respond expectations.

My rule of thumb: if you are unsure whether something is P0 or P1, treat it as P0. You can always downgrade. You cannot un-lose the 15 minutes you spent debating severity while users churned.

---

Step 1: Isolate the Blast Radius

Before you start debugging, understand the scope of the damage. The blast radius determines whether you keep debugging alone or escalate to a war room.

Three dimensions to check:

• Which services? Is it one microservice or a cascading failure across the dependency chain?
• Which regions? Is it us-east-1 only, or global? A single-region failure often points to infrastructure (a bad host, an AZ issue). Global failure points to code or config.
• Which users? Is it affecting all users, or a segment? If only mobile users are affected, the problem is probably in the mobile API gateway or a recent mobile deploy. If only users in one country, check CDN or DNS.

The mental model I use: think of blast radius like triage in an emergency room. Before you start treating anyone, you need to know how many patients there are and how critical they are. An incident affecting 100% of checkout traffic gets a very different response than one affecting 2% of search queries.

For your interview: when describing incident response, always mention blast radius isolation. It instantly signals that you think about impact before jumping to debugging. That is the mark of someone who has actually been on call.

---

Step 2: Check Recent Changes

Once you know the blast radius, correlate the incident start time with the change log. I check these in order:

1. Code deploys: Check the CI/CD dashboard. What deployed in the last 1 hour? Who deployed it? What PRs were included?
2. Config changes: Feature flag flips, environment variable updates, infrastructure-as-code applies. These are often invisible to standard deploy tracking.
3. Database migrations: Schema changes, index additions, bulk data operations. These can cause lock contention that appears as latency spikes minutes later.
4. Cron jobs and batch processes: A nightly ETL job that started at midnight might be saturating the database right when your user traffic is highest.
5. External dependencies: Check the status pages for your cloud provider, CDN, DNS provider, payment processor, and any third-party API you depend on.

If you find a correlating change, you have two options:

• High confidence: The change clearly explains the symptom (e.g., a deploy touched the checkout flow and checkout is broken). Roll back immediately.
• Low confidence: The timing correlates but the change seems unrelated. Keep it as your primary hypothesis but continue investigating.

The most dangerous changes are the ones that look harmless. A "logging-only" change that accidentally imported a heavy library. A feature flag that was supposed to target 1% but targeted 100%. A database index migration that locked a table for 3 minutes during peak traffic.

---

Step 3: Follow the Metrics

If the change log did not reveal the cause, switch to metrics. I use the RED method as my framework:

• Rate: How many requests per second is each service handling? Has throughput dropped (service is rejecting requests) or spiked (traffic surge)?
• Errors: What is the error rate? Which specific error codes? Are they 5xx (server-side) or 4xx (client-side)? Which endpoints?
• Duration: What are the latency percentiles? P50 tells you the typical experience. P99 tells you the worst experience. If P50 is fine but P99 is terrible, you have a tail latency problem (likely one slow downstream dependency).

The hierarchy matters. Error rate first (is something broken?), then latency (is something slow?), then throughput (are we under unusual load?). Most engineers look at CPU dashboards first, which is backward. CPU is a trailing indicator. Error rate is a leading indicator.

My recommendation: create a single "incident dashboard" in your observability tool that shows all three RED metrics for your top 5 services on one screen. During an incident, opening 6 different dashboards wastes critical minutes.

---

Step 4: Read the Logs Strategically

Logs are the last resort, not the first. Most engineers default to scrolling through logs, which is like reading a novel to find one sentence. Structured queries beat scrolling every time.

Effective log queries: