Locate failures along the execution path

Learning objective

Attribute failures to individual agents, tools, handoffs, or end-to-end workflows.

Core idea

The agent that speaks last is not necessarily the component that failed first. A final answer can be wrong because a specialist misinterpreted a request, a tool failed, a message never reached its destination, or the workflow assembled valid local outputs incorrectly. Attribute failure by finding the first violated contract along the execution path, then distinguish that cause from its downstream effects.

For architects diagnosing a multi-agent solution, combine behavioral evaluation with execution evidence. A trace explains what ran and how operations connect. An evaluator judges whether the observed behavior met its task. Neither a completed span nor a failed final score alone identifies the responsible component.

See 3.1.4 for detailed tool-trajectory evaluation. Here the task is to join those local checks with agent, routing, and workflow evidence so that the repair reaches the component that owns the defect.

How it works

Correlate operations before assigning responsibility

Microsoft Agent Framework emits agent, model, and tool spans. In its Python observability documentation, invoke_agent <agent_name> represents an agent invocation, chat <model_name> represents a model call, and execute_tool <function_name> represents function execution. Preserve trace_id, span_id, and parent_id to associate nested operations with the invocation that caused them.

Use stable agent identifiers rather than display names alone. The documented attributes include gen_ai.agent.id, gen_ai.agent.name, gen_ai.response.id, and gen_ai.operation.name. Two invocations of the same named agent are different operations; a retry or parallel branch must not collapse into one diagnostic row.

Pin the instrumentation and semantic-convention configuration with the run. When OTEL_SEMCONV_STABILITY_OPT_IN is unset, Python uses the experimental GenAI conventions. A set value containing the case-sensitive token gen_ai_latest_experimental also selects them; a set value without that token, including an empty value, selects v1.36. Experimental-mode span attributes emit gen_ai.provider.name; v1.36 emits gen_ai.system. Missing fields are not proof that the agent skipped an operation when your parser expects a different convention.

Agent Framework’s documented Python spans include prompt, response, tool arguments, and results only when sensitive-data capture is enabled. Decide which evaluation fixtures can safely include this content. Metadata can locate a tool invocation, but it cannot establish whether its arguments were semantically correct. If content is unavailable, record that limitation rather than inventing a local diagnosis.

Inspect the handoff as its own operation

Workflow observability identifies the executor and routing stages. In Python, workflow.run represents a workflow execution and executor.process {executor_id} represents an executor processing a message. executor.id and executor.type identify that processing component. C# and Go use workflow.session and workflow_invoke for different workflow lifetimes; do not assume the Python span names apply to every SDK language.

A handoff involves a send, routing decision, and receiving execution. message.send is a child of the sending executor’s processing span. The receiving executor’s processing span is linked to that send span rather than nested under it. Follow span links as well as parent-child relationships; a tree-only view can hide the causal path to a receiver.

Inspect edge_group.process and its edge_group.delivered and edge_group.delivery_status attributes. The documented status values distinguish delivered, dropped type mismatch, dropped target mismatch, dropped condition false, exception, and buffered. Those values describe different routing outcomes, not a single generic handoff error.

A false routing condition might be correct for that case. A buffered fan-in message might be waiting for another branch rather than lost. Compare the observed outcome with the expected workflow contract before assigning failure. In Python, message.destination_executor_id on message.send identifies the intended destination; message.source_id and message.target_id on edge processing identify the routed relationship.

Judge each component against its assigned contract

Microsoft scopes system evaluators to single agents and, in multi-agent systems, the main orchestrator or final completion agent. Task Adherence (preview) judges compliance with instructions; Intent Resolution (preview) judges whether the agent identifies and addresses the user’s intent. To test a specialist, evaluate it separately as a single agent on its recorded input and instructions. This is a diagnostic replay, not a documented in-workflow specialist evaluation. A specialist can correctly handle malformed input while its sender fails the handoff contract.

For tools, separate selection and arguments from execution and result use. Microsoft’s process evaluators include Tool Selection, Tool Input Accuracy, Tool Call Success, and Tool Output Utilization. See 3.1.4 for their input mappings and interpretation. Here, join their evidence to the corresponding execute_tool span and owning invocation rather than aggregating all tools under the final agent.

A successful transport operation can return business data that does not satisfy the task. Conversely, a tool exception can be handled correctly by the agent. Keep technical execution outcome, behavioral judgment, and workflow outcome in separate fields in your application-defined diagnostic record.

For end-to-end evaluation, Microsoft lists Task Completion as preview. It assesses whether the task has a usable deliverable meeting all user requirements. Set the run’s evaluation_level to turn (default) or conversation; conversation level requires messages data. Check supported_evaluation_levels in the evaluator catalog: a conversation-level run skips turn-only evaluators. Map the orchestrator or final completion agent’s evidence at that scope. A passing local score does not establish the system-level result.

Test the proposed cause with a controlled replay

Locate the earliest divergence from expected evidence, then replay the same case with one suspected input corrected or one tool result replaced by a verified fixture. This is an application-level diagnostic experiment. The trace and evaluator scores supply its evidence; they do not automatically prove causation.

Hold agent configuration and surrounding behavior fixed. If correcting the handoff repairs the downstream result, investigate the sender or router. If the receiver still fails with correct input, inspect its local contract.

Record both the primary defect and consequential failures. For example, a tool timeout can cause a missing specialist result and a final incomplete answer. Counting those as three independent root causes misdirects remediation. Also permit multiple causes: a broken handoff and an independent generation error can coexist in one run.

Key distinctions and trade-offs

Execution success versus task success: A span can complete without satisfying a business requirement. Evaluate returned content and final deliverables alongside execution status.

Sender versus receiver: The receiving agent owns its interpretation of delivered input. The sender and router own what is sent and whether it reaches the correct destination. Trace links preserve that distinction.

Local versus end-to-end: Local checks explain component behavior; final checks explain user outcomes. Keep both because recovery can mask a local failure and composition can break otherwise valid local results.

Detailed content improves diagnosis but increases exposure of prompts and tool data. Restrict content capture to an appropriate evaluation environment and report uncertainty when the available trace lacks the evidence needed to judge meaning.

Architecture / decision scenario

A claims workflow routes intake to a policy specialist, invokes a policy lookup tool, and sends the specialist’s result to a final decision agent. The final answer omits a required exclusion. The team records the failed end-to-end case but does not immediately change the final prompt.

In the first run, the specialist never executes. Its expected handoff has edge_group.delivery_status="dropped type mismatch". The team follows the send span and receiver link expectations, then corrects the message contract. The failure belongs to handoff delivery, with final incompleteness as its consequence.

In a second run, delivery succeeds and the specialist executes, but the lookup tool fails. Tool Call Success evidence and the tool span locate the technical defect. The specialist’s fallback can still pass its local handling contract while the overall task remains incomplete. The team records both outcomes.

In a third run, delivery and lookup succeed, and the specialist returns the exclusion. The final agent ignores it. With the same correct specialist output replayed, the final answer still omits the requirement. This supports a final-agent consumption defect rather than a retrieval or routing repair. If every local output satisfies its contract but the workflow merges the wrong branch, the composition logic owns the failure instead.

Common exam traps

• Blaming the final speaker for every unsuccessful task.
• Treating an error-free span as proof of correct behavior.
• Following parent spans while ignoring links across handoffs.
• Treating buffered or a deliberate false condition as automatic message loss.
• Combining retries and parallel invocations under an agent display name.
• Inferring semantic correctness from metadata when content capture is absent.

What to remember

Follow trace identifiers, parent relationships, and handoff links. Compare each component’s actual input and output with its contract, then check the final deliverable independently. Attribute the first demonstrated defect, preserve downstream consequences, and use controlled replays to test the repair hypothesis.

Knowledge check

References

• Microsoft, Agent observability
• Microsoft, Workflow observability
• Microsoft, Agent evaluators
• Microsoft, Built-in evaluators