Turn incident findings into release checks

Learning objective

Investigate incidents from logs and traces and feed findings back into evaluation sets and release gates.

Core idea

An incident investigation should produce two different results: restored service and a demonstrated explanation of the failure. A rollback can restore service before the explanation is complete. A plausible explanation does not establish that users have recovered.

Google SRE recommends explicit incident roles and a live incident state document. Use those artifacts to coordinate mitigation and preserve the investigation. Anthropic’s evaluation guidance supplies the other half: inspect transcripts, distinguish agent behavior from grader defects, and turn real failures into test cases. See 4.2.3–4.2.4 for evaluation layers and dataset maintenance; this lesson connects a production incident to those processes.

How it works

Preserve a bounded incident record

Start with the alert’s affected interval, segments, and service target. Record the last known healthy interval and recent prompt, model, application, tool, and index revisions. Assign an incident commander, an operations (Ops) lead, and the communication and planning roles; a small team can combine roles explicitly. Maintain a timestamped live incident state document containing observations, hypotheses, decisions, and recovery checks. The planning role tracks follow-up work and handoffs while Ops coordinates system changes.

Preserve the records needed to investigate within the access and retention rules from 3.4.5. Capture identifiers and approved diagnostic content before ordinary expiry removes them. Retaining an entire production conversation indefinitely is not a prerequisite for diagnosis; choose a bounded incident package and record what could not be retained.

Start from the affected population, then choose failed and successful comparison traces within it. Randomly opening transcripts from the entire service can miss the affected cohort. Comparing only failures can also hide the feature that distinguishes them from healthy requests.

Bound the investigation to the incident

Use the failed and healthy traces from the affected population to compare the same task class, interval, and configuration. See 4.4.5 for localizing the broken transition and 4.4.3 for stale retrieval. Record which hypothesis the comparison supports, which observation would contradict it, and what remains unknown in the live incident state document.

If provider support is needed, include the affected API request-id with sanitized incident details. Keep that escalation separate from the decision to mitigate: support investigation can continue while the operations team restores service.

Mitigate, then verify the same user outcome

Use the rollback path 4.3.4 defines, including its treatment of external effects; record the selected artifact revision, decision owner, and change time in the live incident state document.

Verify recovery using the original affected population and service target, including quality. A return of low error rates is insufficient if the incident involved incorrect answers. Keep unrelated deployments out of the recovery window where possible so the result remains interpretable. The incident commander records when mitigation is verified and which root-cause questions remain open.

Make the postmortem produce owned follow-up actions

Produce a blameless postmortem: a written record of the incident, its impact, mitigation or resolution, contributing causes, and follow-up actions to prevent recurrence. Google SRE requires review of the document and its action plan. Use the live incident state document as input, then distinguish verified conclusions from hypotheses still under investigation. Explain which system conditions made the failure possible without assigning blame to an individual.

Make the regression case and the release requirement two owned actions in that postmortem. For each, record a responsible owner, completion criterion, priority, and target date. These action fields are the team’s implementation of the reviewed action plan. The evaluation owner accepts the case when it reproduces the failure and checks the intended outcome; the release owner accepts the deployment requirement when a failing candidate is actually prevented from progressing. Linking two tickets without checking their completion does not close the prevention work.

The postmortem produces two owned actions, and each owner accepts its action only when its own check holds.

Create a sanitized case with the essential user input, relevant source or tool fixtures, configuration provenance, expected outcome, and explicit failure condition. Preserve the relationship that caused the failure; removing every source date from a stale-policy incident would destroy the test’s purpose.

Add the narrow test where the defect belongs and an end-to-end case that reproduces user harm. An ingestion revision bug needs an ingestion check as well as answer evaluation. A model grader that rejected an acceptable answer needs grader adjudication, not a prompt patch that teaches the model to satisfy a defective rubric. Anthropic recommends reading transcripts and grades together to verify what the evaluation actually measures.

Define a release requirement for the repaired candidate and its unaffected neighbors. Name the evaluation dataset version, grader version, model and prompt configuration, required checks, and owner. Use several trials where model variation matters, and isolate external writes with fixtures or a test environment. Replaying the live transaction is not a safe regression harness.

The immediate incident case becomes regression coverage. Keep separate held-out cases and broader suites, as 4.2.4 describes, so repeatedly tuning on the incident does not become the only proof of readiness. Require both the targeted repair and acceptable quality, latency, and cost on the broader workload. After deployment, compare the affected production segment with its recovery criteria; offline success does not replace that check.

Key distinctions and trade-offs

Mitigation reduces impact; root-cause repair prevents recurrence. Keeping those decisions separate lets the team restore service without pretending that a rollback explains every failure.

Logs and traces establish observed execution, while a controlled reproduction tests a hypothesis. A regression suite establishes that specified behaviors remain acceptable, within its population and trial coverage. None alone guarantees that the incident cannot recur in another form.

More incident content can improve diagnosis but increases exposure and review cost. Preserve the minimum package that can test the hypothesis, with named access and an owner for deletion or continued retention.

Architecture / decision scenario

A government benefits assistant must answer eligibility questions using the policy effective on the application date. At 09 the team publishes a prompt revision and an index revision. By 09, sampled correctness falls from 98% to 84%, while provider errors remain below 1%. The tempting response is to rewrite the prompt because it changed at the same time.

The incident commander compares affected and healthy traces. Both prompts receive the same superseded policy passage; the serving index pointer did not move to the published revision. The retrieval owner changes that pointer to the verified index, and reviewers confirm the affected date-sensitive cohort returns above its 97% acceptance floor. The prompt owner leaves prompt repair pending because the observed failure occurred before generation.

The postmortem records the missed serving-index update and the absent release check as contributing causes. Its reviewed action plan assigns the regression fixture to the evaluation owner and the deployment requirement to the release owner, each due before the next index release. The fixture contains the policy transition, application date, and two document revisions. The retrieval check requires the effective revision; the end-to-end evaluation requires the corresponding eligibility answer. The release owner blocks deployment if either fails, runs the broader suite, and verifies the cohort after release. The post-incident work therefore repairs both the serving defect and the missing release check rather than merely adding the disputed question to a transcript collection.

Common exam traps

• Treating a nearby release marker as proof of the cause.
• Changing the model before tracing source and tool outcomes.
• Replaying an uncertain business write to reproduce an incident.
• Declaring recovery from HTTP status when quality caused the incident.
• Adding a test case without an expected outcome or release requirement.
• Closing a postmortem after writing it without reviewing and completing its owned actions.
• Repairing the prompt when the transcript exposes a source or grader defect.

What to remember

Scope the incident, reconstruct the first consequential divergence, mitigate, and verify the same user outcome. Record the regression case and release requirement as reviewed, owned follow-up actions in a blameless postmortem. A useful incident review changes what the system must prove before its next deployment.

Knowledge check

References

• Google, Managing Incidents
• Anthropic, Claude API errors
• Anthropic, Demystifying evals for AI agents
• Google, Postmortem Culture: Learning from Failure