← Browse skills

Knowledge

Write the incident review

Assemble a timeline from the systems that recorded it, and a review that names causes rather than people, within a day of the all-clear.

By Toolspoke

Skill procedure

The review is worth writing while the detail is still recoverable and worth nothing if it becomes a blame document. Everything below is assembled from records, not from memory, and the sections are in the order a reader needs them rather than the order things happened.

Gather first, write second

  • PagerDuty: when the alert fired, when it was acknowledged, when it resolved.
  • The incident channel in Slack: the first human message, and every message where somebody states a new fact. These are the timeline's real entries.
  • GitHub: the commits merged in the six hours before the alert, the revert or fix PR, and when it merged.
  • The deploy platform: when each of those reached production, which is never the same time as the merge.
  • Sentry or the metrics backend: when errors started, which is usually earlier than the alert, and when they stopped.

The review itself

Write it in this order, in plain sentences:

  1. What people experienced, in the words a customer would use, and for how long. Not "elevated 5xx on the API" — "for 38 minutes, signing in failed for about a third of attempts".
  2. The timeline, one line per entry, timestamps in UTC: first error, alert, acknowledgement, each finding, the fix, recovery confirmed.
  3. The cause, in as many sentences as it takes, ending at a change somebody made or a condition that was always going to happen eventually. Stop at the mechanism, not at the person.
  4. Why it took as long as it did. Detection time and repair time have separate causes, and the slow one is usually detection.
  5. What happens now: a short list of changes, each one owned and each one a ticket. A review with actions that are not tickets has no actions.

Post the review where the team reads, link it from the incident channel, and file each action in Linear or Jira with the incident linked.

Stop and ask a human

  • Before naming an individual anywhere in the document.
  • Before publishing anything that quotes a customer, includes a credential, or reproduces a stack trace containing personal data.
  • When the cause is genuinely not known. Publish the timeline and say the cause is open. An invented cause is worse than an open question.