Prove incident causes before applying repairs

Learning objective

Diagnose incidents using trace reconstruction, safe replay, and evidence-backed remediation.

Core idea

For an architect responding to an agent incident, a plausible story is not enough. Reconstruct what the system actually executed, identify the earliest observable divergence from its contract, and test a repair against preserved evidence. A final wrong answer can originate in model behavior, a missing tool result, a routing defect, or a failed external action.

Separate observation, hypothesis, and experiment. Trace reconstruction establishes recorded execution. Safe replay tests a hypothesis under controlled dependencies. Remediation changes the responsible mechanism and needs evidence that the change fixes the incident without introducing a regression.

See 1.2.7 for minimized replay evidence, 3.3.2 for decision correlation, and 3.3.8 for trace setup. Here the task is to use those records during an incident, not to collect private model reasoning or assume that repeating a production request is harmless.

How it works

Locate the affected execution

Start with the reported business request and its outcome. Preserve the response ID, conversation ID, trace ID, time window, agent version, model deployment, and relevant application configuration when available. Keep the production evidence read-only. A matching user message alone can select the wrong repeated attempt.

Foundry’s Traces view supports trace inspection, and its conversation results can be located by Response ID or a mapped Trace ID. Selecting a Conversation ID exposes dialogue history, response information and tokens, ordered actions, and tool calls. Use conversation history to establish what the service retained, and execution spans to establish what operations ran.

Foundry tracing is generally available for prompt and hosted agents; workflow and external tracing is preview. Its documented trace view covers ingested traces from the last 90 days. If the incident predates retained evidence or some components were not instrumented, state the gap explicitly. An empty view is not proof that no execution occurred.

Reconstruct causality rather than sorting timestamps

Agent Framework records invoke_agent <agent_name>, chat <model_name>, and execute_tool <function_name> spans. Align model calls, tool execution, and terminal outcomes, then identify the operation where the actual result diverged from the expected contract. Duration and error evidence help distinguish a failed call from a successful call whose result was unsuitable.

Workflow evidence needs more than parent-child nesting. Agent Framework’s workflow observability documentation states that message.send is a child of the sending executor.process, but the receiving executor’s executor.process is linked to that send span rather than nested under it. edge_group.process also uses links for causality, including fan-in. Follow span links when reconstructing handoffs; a tree-only reading can miss the responsible producer.

Inspect edge_group.delivery_status before blaming the model for missing information. Documented values include delivered, dropped type mismatch, dropped target mismatch, dropped condition false, exception, and buffered. A buffered fan-in message is waiting for more input, while dropped type mismatch means the target cannot handle that message type. They imply different fixes.

Compare the causal sequence with the workflow contract. If the source sent a valid result but routing dropped it, reformulating the receiving agent’s prompt does not address the observed defect. If routing delivered it and the next model call omitted a required fact, investigate context selection or compaction; see 3.2.3–3.2.4.

Choose a replay mode that answers the hypothesis

See 1.2.7 for versioned replay packages, fixture-based evidence replay, and live reruns. During an incident, choose the mode that isolates the suspected failure without repeating production writes. Keep the replay credential’s permissions no broader than the original execution’s; broader permissions can make a broken path appear healthy while bypassing its cause.

Checkpoint resumption is a third operation. Agent Framework Python resumes a workflow through workflow.run(checkpoint_id=..., checkpoint_storage=..., stream=True). .NET uses InProcessExecution.ResumeStreamingAsync(...). A checkpoint restores saved workflow execution state; it does not by itself establish that an external business effect should run again.

The .NET checkpoint documentation requires the same workflow structure and executor identities. Reconstructed local ChatClientAgent instances need stable ChatClientAgentOptions.Id, and an assigned Name must also remain stable. Do not interpret an incompatible rebuilt workflow’s resume failure as evidence that the original production incident persists.

Treat checkpoint storage as trusted, private infrastructure and never load untrusted or potentially tampered checkpoints. Use only a verified checkpoint lineage. Compare the saved execution point with downstream action records before resuming. Idempotency and action reconciliation belong to your business integration; a restored checkpoint is not proof of exactly-once execution.

Turn the incident into a controlled regression case

Select the incident’s exact traces with Add to dataset in Foundry or a nonempty trace_ids list on the SDK trace source; see 1.2.7 for the generation call, sampling, and redaction controls. Representative sampling can miss a rare failure. Verify that the resulting query and response retain the evidence needed to reproduce the incident under the permitted content policy.

Trace-based dataset generation requires the Foundry User role or higher on the project. The project’s managed identity needs the Reader role on connected Application Insights; protected trace tables also require Privileged Monitoring Data Reader. Run generation in a supported data-generation region, as listed on the referenced dataset-generation page. Being able to open a trace does not establish these service permissions or region support.

A dataset row supports evaluation; it is not a full workflow checkpoint or a license to execute captured tool arguments. Add application fixtures for the routing, tool, or side-effect contract that the incident depends on. See 3.3.3 for comparative quality evaluation and 3.3.4 for bounded operational remediation.

Key distinctions and trade-offs

Reconstruction versus explanation: Reconstruction lists evidenced operations and causal relationships. An explanation names a hypothesis about why the divergence happened. Preserve uncertainty when telemetry cannot distinguish causes.

Replay versus resume: Replay tests selected evidence under controlled dependencies. Resume continues saved workflow state. Neither guarantees an identical model answer or safe repetition of external writes.

Containment versus correction: Containment limits impact, such as pausing a write-capable path. Correction repairs the evidenced defect. Keep containment and permanent repair separately reviewable.

Fixtures improve repeatability but cannot reveal every live-service interaction. Sandbox re-execution exercises more integration behavior but introduces variable state and cost. Choose the narrowest experiment that can distinguish the competing causes, then widen when evidence requires it.

Architecture / decision scenario

A claims workflow sometimes stalls after two evidence agents complete, and an operator suggests replaying the whole request in production. One tool can issue a payment, so the team first preserves the trace and downstream action record and blocks production writes in the diagnostic harness.

Reconstruction follows both agents’ message.send span links into the fan-in edge group. One path reports buffered; the other reports dropped type mismatch. The expected typed result never reached the receiver, explaining the stalled join without attributing it to model reasoning. The team captures both observations and the failing message contract.

A fixture-based replay reproduces the drop with payment replaced by a recorded stub. The team corrects the sender/receiver message contract and verifies both branches reach delivered and the join completes. It selects the incident’s exact trace_ids for a redacted evaluation dataset and adds a regression case for the routing defect.

Before any operational resume, the operator checks the trusted checkpoint, compatible executor identities, and whether payment already occurred. The repair evidence includes the failing and repaired causal paths, the isolated test result, and the downstream action reconciliation. If those records cannot establish the payment outcome, the workflow remains paused for that action’s owner rather than repeating it speculatively.

Common exam traps

• Reading a timestamp-sorted tree without following workflow span links.
• Treating a missing trace as proof that a call did not run.
• Rewriting prompts when evidence shows a typed routing failure.
• Replaying production tool calls that can repeat business effects.
• Treating checkpoint resumption as exactly-once action execution.
• Assuming representative sampling must include the incident trace.

What to remember

Reconstruct the causal path, distinguish missing evidence from a measured failure, and test one supported cause at a time. Replay with isolated dependencies and verified versions; resume only from trusted, compatible state after reconciling business effects. A remediation is defensible when the evidence shows both the original failure and the repaired contract.

Knowledge check

References

• Microsoft, Set up tracing for AI agents in Microsoft Foundry
• Microsoft, Observability
• Microsoft, Microsoft Agent Framework workflows: Observability
• Microsoft, Workflow checkpoints
• Microsoft, Convert agent traces into evaluation datasets