Communicate incident impact, limits, and verified results

Learning objective

Communicate outages, limitations, and evidence-backed outcomes to users.

Core idea

Users need to know whether they can rely on an agent’s result and what to do next. An outage message explains the affected capability and available alternative. A limitation disclosure explains what the system cannot establish even when it is operating. An outcome statement explains what this particular run completed and what supports that claim.

These are separate communication obligations. A status page reporting healthy infrastructure cannot certify an individual answer, and a citation beside an answer cannot establish that a requested external action succeeded. NIST AI RMF 1.0 separates transparency from accuracy and calls for information tailored to the people interacting with the system. Your goal is informed use, rather than reassurance unsupported by the task’s result.

How it works

Give incident communication an owner

Google’s incident-management model separates incident command, operational work, communication, and planning. The communication role issues periodic updates to responders and stakeholders; the incident commander maintains a living incident state document. Apply that division so engineers restoring service do not also have to improvise every customer-facing update.

Use the incident state document as the shared record of observed impact, mitigation, unresolved questions, and ownership. Publish user-facing updates through a channel that remains available when the agent service fails. Google specifically warns against depending on the system under repair for incident management. A message generated only by the unavailable model has the same dependency problem.

For each update, state the observation time, affected task or customer population, observed behavior, available workaround, and next update time. Google’s Example Incident State Document supplies a starting artifact with a summary, status, timestamped detailed status, exit criteria, and incident timeline. Adapt that record into the service’s user-facing communication template so successive updates answer the same operational questions. Label a suspected cause as a hypothesis until operations verifies it. Give a next communication time even when you cannot give a trustworthy restoration estimate.

NIST’s MANAGE 4.3 calls for communicating incidents and errors to relevant AI actors, including affected communities, and following documented tracking, response, and recovery processes. Communication is therefore part of recovery ownership. A private incident channel alone cannot inform customers whose work remains blocked.

Disclose operating limits where they matter

NIST’s MAP 2.2 calls for documentation of knowledge limits and how humans can use and oversee system outputs. Convert that into a capability statement specific to the user’s decision: supported sources and tasks, source freshness, required verification, and the path for reporting an incorrect result. Place a consequential temporary limit beside the affected task, rather than expecting a user to find it in a general policy page.

For example, if a retrieval dependency is unavailable, tell users that document-grounded answers are unavailable before offering a general drafting mode. If a fallback supports text but not image interpretation, disclose that distinction when an image arrives. See 8.5.1 for the recovery path; this lesson adds the explanation needed to use that path responsibly.

NIST’s MANAGE 4.1 includes mechanisms for user input, appeal and override, incident response, and recovery in post-deployment monitoring plans. Give the user a named support or review route linked to the affected interaction’s reference. The product owner maintains that route; the operations owner verifies the service-state claims it carries. For high-consequence decisions, the responsible domain reviewer owns whether the available information supports proceeding.

Attach support to the actual outcome

See 1.1.2 for source displays, relevance-score limits, and confirmed action status, and 6.5.2 for retrieval attribution. In a live incident, use those interaction records to identify which outputs require correction after service recovers.

Keep an affected-output register alongside the incident state document: interaction reference, production time, affected capability, limitation shown to the user, review owner, and resolution. These are service-designed fields that implement the tracking and recovery practices in MANAGE 4.3. Scope the register by the degraded capability and observed incident window rather than flagging every answer indiscriminately. If the window’s start is uncertain, record that uncertainty and widen the review population until operations can bound the impact.

For a retrieval outage, inspect the affected outputs’ supporting passages. The RAG Server schema’s ChainResponse.citations and its results entries supply document identity and content for that review. Mark a summary awaiting source review separately from a reviewed correction. The domain reviewer decides whether the supporting material establishes the consequential claims; the communication owner tells affected users which earlier result is superseded and where to obtain the reviewed replacement.

Restoration and correction need distinct exit criteria in the incident record. Successful source queries can establish that retrieval is available again, while the affected-output register still contains unresolved summaries. Report the restored capability and remaining correction work separately. A blanket “all results verified” message is premature until the review owner has resolved the population to which that claim applies.

NIST’s MEASURE 2.9 calls for explained, validated, documented models and interpretation of outputs within context. At release level, communicate evaluation population and known exclusions alongside measured results from 8.3.1. At interaction level, communicate the sources and checks that apply to this result. An aggregate pass rate cannot certify an individual response.

Key distinctions and trade-offs

More detail is not automatically more useful. Give users enough information to choose their next action, while keeping raw prompts, internal traces, and sensitive source material within the appropriate access boundary. Tailoring transparency to the audience is different from publishing every diagnostic artifact.

Architecture / decision scenario

A commercial-property research assistant produces lease summaries for an acquisitions team. At 14 UTC, access to the authoritative lease repository fails. A separate drafting mode still responds. Of 200 requests during the incident, 140 request source-grounded lease analysis and 60 request general drafting. These are scenario observations.

The product manager proposes a green status because all 200 requests receive text. The incident commander rejects that interpretation: 140 requested capabilities remain unavailable. The communications owner publishes: “Lease-source analysis is unavailable as of 14 UTC. General drafting remains available without lease verification. Next update: 14 UTC.” The interface labels drafting results accordingly instead of presenting them as grounded lease advice.

After access returns, operations confirms 20 consecutive successful source queries, but three earlier summaries still lack supporting passages. The owner reports service restoration separately from those unresolved results, identifies the affected interactions, and offers reviewed regeneration. The affected-output register keeps the three unresolved summaries open until their owners review the returned citations.results passages and publish corrections. The domain reviewer decides whether each summary supports the acquisition decision. Restoring the dependency repairs availability; it does not retroactively validate the incident’s outputs.

Common exam traps

• Declaring full recovery because the model produces text again.
• Waiting for a confirmed root cause before reporting verified user impact.
• Publishing incident updates only through the agent’s own interface.
• Using aggregate evaluation results to guarantee a particular answer.
• Closing the affected-output review when the dependency recovers.

What to remember

Assign a communication owner and publish scoped impact with a next update time. Explain limitations at the task where they affect a decision. Track and review outputs from the degraded window; distinguish restored service from completed correction work and state which claims each check supports.

Knowledge check

References

• Google, Site Reliability Engineering: Managing Incidents
• Google, Site Reliability Engineering: Example Incident State Document
• NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0)
• NVIDIA, API - RAG Server Schema