Rank risks by harm and recoverability

Learning objective

Prioritize LLM-system risks by likelihood, impact, detectability, and reversibility.

Core idea

For an architect allocating a limited risk-reduction budget, frequency alone is a poor ordering rule. A frequent drafting defect that is caught before publication can deserve less urgency than an uncommon disclosure that nobody detects until the information has left the organization.

NIST AI RMF defines negative risk through the likelihood and magnitude of harm. MAP 5.1 calls for documenting those judgments, and MANAGE 1.2 prioritizes treatment using impact, likelihood, and available resources or methods. Detectability and reversibility add operational questions about whether your system can catch a failure and recover before harm spreads. They do not replace the underlying impact assessment.

How it works

Define a comparable risk entry

Begin with a cause, event, and consequence in a named operating context. “Hallucination” is a limitation category. “The drafting agent invents an eligibility exception and a caseworker publishes an incorrect denial” is a risk you can assess. Record the affected population, relevant workflow volume, existing controls, and time horizon.

Separate the risk before a proposed control from the residual risk after controls that actually operate. If a reviewer checks every output today, estimate the present exposure with that review included; do not silently assume review also exists in a proposed autonomous rollout. Keep both operating modes visible when comparing them.

The artifact is a risk-register entry supporting NIST’s MAP and MANAGE activities. The fields and scales in this lesson are an example organizational method. Neither NIST nor the Claude API supplies a universal four-factor score or a pass threshold for every LLM system.

Estimate likelihood with a stated denominator

Likelihood concerns the harmful event in the relevant deployment, not just a model’s raw error rate. State whether an estimate describes one case, one agent session, one day, or the rollout period. A one-percent unsupported-draft rate and a one-percent rate of published harmful decisions describe different risks.

Use representative evaluations, incidents, comparable deployments, and feedback, as MAP 5.1 describes. Distinguish measured observations from expert estimates. A rare case absent from a small pilot is not demonstrated impossible. Where exposure is adversarial, also identify whether an attacker can repeatedly trigger the entry point; ordinary traffic prevalence may understate that opportunity.

Your register should carry the uncertainty that could change the decision. “No observed incident in 300 reviewed cases” is useful information. “Zero likelihood” is a different claim requiring much more than that observation. See 4.3.3 for interpreting sample size and uncertainty.

Describe impact beyond average cost

Impact is the magnitude of the consequence for the people and systems affected. Include the worst credible scope, not only mean recovery time. Distinguish one incorrect draft from a faulty rule propagated across thousands of cases; distinguish a delayed response from a disclosure to an unauthorized recipient.

NIST’s prioritization guidance says unacceptable negative risks can require development and deployment to cease safely until sufficiently managed. Its examples include imminent significant negative impacts, severe harms occurring, and catastrophic risks. A favorable average score is not a reason to continue an unacceptable use.

Define the organization’s severity categories and tolerance with the appropriate business and risk authorities. AI RMF does not prescribe that tolerance. An architect can assess the failure path and propose controls; the organization decides which consequences it is authorized and prepared to bear. Regulatory determinations require the compliance and legal owners, rather than being inferred from this ranking exercise.

Assess detection before the consequence

Detectability asks whether the deployed checks expose the failure in time to act. Record the detection method, its coverage, and the delay relative to the harmful action. An alert after an email is delivered gives incident visibility but does not prevent the recipient seeing its contents.

For a factual claim, Anthropic recommends quotes and claim verification, but warns that hallucination-reduction techniques do not eliminate the problem. A checker that confirms a quote appears in a document does not establish that it supports the conclusion. Score the detection you demonstrated, not the detection implied by the label “validated.”

NIST’s discussion of validity and reliability calls for ongoing testing or monitoring and recognizes human intervention where a system cannot detect or correct errors. Link your rating to the evaluation that exercises that limitation. A planned detector without measured coverage remains a treatment proposal, not a reason to lower the present rating.

Assess recovery after the event

Reversibility asks what can actually be restored, by whom, and in what time. A draft can be discarded before use. A database value may be restored while decisions based on it still require separate correction. A disclosed confidential record cannot be made unseen by restoring a backup.

NIST MANAGE 4.3 addresses documented processes for tracking, responding to, and recovering from incidents and errors; MANAGE 4.1 addresses post-deployment plans. Use those artifacts to justify your recoverability judgment: a rehearsed restoration path is stronger support than an assumed ability to undo. Do not call an apology, compensation, or a replacement answer a full reversal of every consequence.

Record detection and reversal separately. A failure may be obvious but irreversible, or difficult to notice yet reversible after discovery. Treating both dimensions as one “controllability” rating hides the intervention the team actually needs.

Key distinctions and trade-offs

A qualitative comparison can preserve more information than multiplying ordinal scores:

These are illustrative assessments, not standard NIST ratings. You may use a score to organize discussion, but retain the underlying dimensions and any severity override. Equal arithmetic products can conceal very different release decisions.

Architecture / decision scenario

A government service pilots a case-summary assistant under a two-week remediation budget. In 1,000 reviewed drafts, staff find 80 formatting defects and two unsupported denial recommendations. A separate security test demonstrates a route to sending a confidential attachment outside the approved audience. The release manager proposes fixing formatting first because it accounts for most observed defects.

The architect uses all four dimensions. Formatting is frequent, immediately detected, limited in impact, and reversible before use. Denial errors are less frequent but consequential, with uncertain detection if manual review is removed. The disclosure has no demonstrated ordinary-traffic frequency, but the tested attack path, delayed detection, serious impact, and poor reversibility prevent treating it as negligible.

The privacy owner requires the disclosure path contained before release. The service owner retains review for denial-related outputs while the team evaluates detection. Formatting receives a scheduled improvement rather than the whole remediation budget. The decision records the estimates, operating mode, and uncertainty; it does not invent comparable probabilities to make a neat numerical ranking.

Common exam traps

• Ranking by the number of observed defects without considering consequence.
• Treating a small incident-free pilot as zero likelihood.
• Using a planned control to lower the present risk rating.
• Equating an after-the-fact alert with detection before harm.
• Calling a restored database a reversal of every downstream action.
• Presenting a four-factor formula as a NIST or Anthropic requirement.

What to remember

State the harmful event, denominator, consequence, detection window, and recovery limits. Preserve uncertainty and the four dimensions when ranking. Prioritization allocates scarce resources within an organization’s tolerance; it does not turn unacceptable or irreversible harm into an acceptable average.

Knowledge check

References

• NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0)
• Anthropic, Reduce hallucinations