Writing a blameless post-mortem

TL;DR

• A blameless post-mortem asks "why did the system allow this?" not "who did this?"
• Follow a consistent template: summary, timeline, root cause, contributing factors, what went well, what went wrong, action items.
• Use the 5 Whys technique to dig past symptoms to systemic causes.
• Every action item must be SMART (Specific, Measurable, Assigned, Realistic, Time-bound) and tracked in your ticketing system.
• The post-mortem meeting is for alignment and discussion, not for writing the document. The doc should be written beforehand.

---

What Makes a Post-Mortem Blameless

A blameless post-mortem proceeds from one assumption: nobody came to work intending to cause an outage. When someone made a decision that contributed to the incident, they made the best decision available to them given the information and tools they had at that moment.

This is not about being soft. It is about being effective.

Blame-focused retrospectives produce three predictable outcomes: engineers start hiding mistakes (so you lose visibility), engineers become risk-averse (so your velocity drops), and the underlying systemic problems persist (so the incident happens again, caused by a different person making the same reasonable decision with the same bad tools).

The goal of a blameless post-mortem is to change the system so that the next engineer in the same situation, with the same information, cannot accidentally cause the same failure. That means fixing the process, the tooling, the monitoring, or the safeguards. Not asking a human to "be more careful."

Here is the difference in practice:

The pattern: every blame-focused finding can be rewritten as a systemic gap. That gap is what the action items should address.

---

The Post-Mortem Template

I am going to walk through each section of a complete post-mortem template. I will explain what belongs in each section, common mistakes, and then show a real example woven through all the sections.

The example incident: a checkout outage caused by an HTTP client library upgrade that changed default timeout behavior.

---

Section 1: Incident Summary

The summary is one paragraph. It answers: what happened, when, how long, and what was the user impact. Write it so someone who skips the rest of the document still understands the incident.

What belongs here:

• What broke (user-visible symptom, not technical cause)
• When it started and when it was resolved
• Duration of user impact
• Severity level assigned
• Quantified impact: number of affected users, failed transactions, or revenue impact if known

Common mistakes:

• Writing a technical root cause in the summary. Save that for Section 3.
• Vague impact: "some users were affected." How many? For how long?
• Missing the resolution method. Did you rollback, hotfix, or wait it out?

Example:

On November 15, 2024, from 13:55 to 14:09 UTC (14 minutes), users attempting to complete checkout received 500 errors. Approximately 2,300 checkout attempts failed during this window, affecting an estimated $47,000 in potential revenue. The incident was classified as P0. The issue was mitigated by rolling back a deploy that had gone out at 13:52 UTC. Root cause: an HTTP client library upgrade changed default timeout behavior, causing payment API calls to time out for high-risk transactions.

That is a good summary. In four sentences, you know exactly what happened, the impact, how it was fixed, and the root cause. A VP, a support agent, and a fellow engineer can all read that and understand the incident.

---

Section 2: Timeline

The timeline is a minute-by-minute reconstruction of events. It starts before the incident (what triggered it) and ends after resolution (verification that the fix worked).

What belongs here:

• Timestamps in UTC (always UTC, never local time zones)
• Actions taken and by whom (role, not necessarily name)
• What was discovered at each step
• Communication milestones (when stakeholders were notified, when the status page was updated)

Common mistakes:

• Starting the timeline at the alert. Start at the triggering event (the deploy, the config change, the traffic spike).
• Missing the gaps. If nobody did anything for 10 minutes, that is important information. Include it.
• Omitting failed attempts. "We tried restarting the service and it did not help" is valuable because it tells future engineers what not to try.

Example timeline:

Notice the 8-minute gap between the first error (13:55) and the alert (14:03). That gap is itself a finding for the post-mortem: can we detect faster?

---

Section 3: Root Cause Analysis

This is where most post-mortems go wrong. They stop at the first "why" instead of digging deeper. The 5 Whys technique forces you to keep asking until you reach a systemic cause.

The 5 Whys technique:

Start with the symptom and keep asking "why?" until you reach something you can fix systemically (not a human error, but a process or tooling gap).

Example:

1. Why did checkout return 500 errors? Because calls to the payment API timed out after 30 seconds.
2. Why did payment API calls time out at 30 seconds? Because the HTTP client library upgrade changed the default timeout from "no timeout" to 30 seconds.
3. Why was the default timeout change not caught before production? Because our integration tests mock the payment API and do not test with realistic latency.
4. Why do integration tests mock the payment API? Because we have no staging environment connected to a payment sandbox with realistic behavior.
5. Why do we not have a staging payment sandbox? Because setting one up was never prioritized, and the mocking approach had not caused a failure before.

The root cause is not "the engineer upgraded the library." The root cause is: our test pipeline cannot catch timeout-related regressions because we mock external dependencies instead of testing against realistic sandboxes.