Skip to main content

The Incident-Response Cycle

In one line: Contain harm through the approved recovery path, then investigate and regression-test the cause; a revert is not safe merely because an older image exists.

Do this: Prioritize safe mitigation while gathering the facts needed to select it. Check current deployed state, schema/data compatibility, in-flight work and irreversible effects before rollback. The authorized response may be isolation, pausing dispatch, traffic diversion, roll-forward or a tested restore—not necessarily flag-off or redeployment of an earlier SHA.

The cycle has six steps, in order:

  1. Detect. Capture the relevant authorized, sanitized observations before they rotate out; do not collect private conversations or request bodies by default.
  2. Mitigate (the approved recovery path). The accountable incident role selects the least harmful effective action under the runbook and available evidence. A flag-off may stop new work without undoing completed effects; an old image may be incompatible with current data or jobs. Respect fencing/draining and external-effect reconciliation. Verify the mitigation in the actual environment and escalate if it fails. Mitigation buys time; it is not the final fix.
  3. Root-cause. Run /systematic-debugging against the now-stable system: reproduce, find the actual cause, distinguish verified from hypothesized (the investigative discipline of §2.6). Do not skip to a fix on a guessed cause.
  4. Write the failing test (regression pin). Reproduce the relevant failure safely before the permanent fix. Record the covered incident behavior and remaining paths; one regression case does not prove the entire class can never recur.
  5. Postmortem. Record the incident in a short ADR or memory entry that names the incident class — what failed, why mitigation worked, what the regression pin now guards. A fault that produced no error where one was due is a silent-failure incident: the postmortem must call that out, so the next occurrence is loud, not invisible.
  6. Update authorized handover sources. Record the incident lesson and evidence within the correct audience. Refresh an adopted STATE.md snapshot if useful, but verify recovery against the actual environment; Git activity does not establish recovery or resolution.

Why: Separate urgent containment from permanent repair without making blind reversal a prerequisite. A tested recovery contract, clear incident authority and regression evidence reduce repeat risk; they do not guarantee future absence of the incident class.