MicrosoftMicrosoft Certified: Multi-Agent AI Solutions Expert2026-103.3 Design and implement observability and monitoring for multi-agent solutions
Measure agent health against workflow outcomes
Completed by the knowledge check
Learning objective
Monitor per-agent health, workflow failure rates, and cross-agent coordination outcomes
Core idea
A healthy agent invocation does not prove that a multi-agent workflow completed. Each specialist can return without an exception while the coordinator drops a message, waits indefinitely for another branch, or accepts an unusable result. Monitor three separate questions: did each agent execute, did the workflow reach its terminal outcome, and did the required exchanges between agents succeed?
For architects and senior engineers, the useful design is a layered health model with explicit counting rules. Microsoft Agent Framework provides agent and workflow spans; Microsoft Foundry connects agent monitoring to Application Insights. Your application supplies the business completion contract. See 0.15 for telemetry fundamentals and 3.1.7 for request-level failure attribution. Here the focus is the aggregate evidence that reveals repeated operational problems.
How it works
Establish the collection path
Connect an Application Insights resource to the Foundry project before expecting the Agent Monitoring Dashboard to show agent telemetry. Its Monitor tab includes token usage, latency, run success rate, and evaluation metrics. The documentation marks the agent metrics view as preview. These charts provide operational evidence; they do not define your workflow’s business success criteria.
Foundry tracing has a separate support scope: tracing is generally available for prompt and hosted agents, while workflow and external-agent tracing in Foundry is preview. Agent Framework’s workflow instrumentation is a distinct SDK mechanism. Do not infer a portal feature’s support or status from the presence of an SDK span.
For Python Agent Framework workflows, enable_instrumentation() activates workflow telemetry, and configure_otel_providers() configures exporters. In .NET, enable workflow telemetry through WithOpenTelemetry and configure WorkflowTelemetryOptions. Verify the exported data using a known successful run and a deliberately failed run before interpreting a quiet dashboard as good health.
Measure each agent separately
In Python, Agent Framework documents invoke_agent <agent_name>, chat <model_name>, and execute_tool <function_name> spans. These separate the agent invocation from its model and tool work. Its documented histograms include gen_ai.client.operation.duration, gen_ai.client.token.usage, and agent_framework.function.invocation.duration; the duration histograms use seconds. Use these names when selecting measurements rather than inventing a generic agent-performance metric.
Group observations by stable agent role and deployed version using your telemetry schema. A role that uses several model calls can have higher invocation latency even when individual model durations remain stable. A tool-duration increase points toward a different investigation from increased model-call count. Record the version identifiers needed to connect a changed signal to a deployment; version labels you add are application metadata, not automatic Foundry fields.
Maintain both attempt and terminal counts. A specialist that succeeds on its third attempt has one successful logical invocation and two failed attempts. Combining those counts hides retry amplification or exaggerates workflow failures. Preserve the distinction in application counters or structured records and state it beside the chart.
Define the workflow failure rate
Python workflows expose workflow.run; .NET exposes workflow.session for the session lifetime and workflow_invoke for an input-to-halt cycle. Choose the operation that corresponds to your business unit of work. Dividing failed sessions by input cycles mixes different populations and gives a misleading rate.
Define application terminal outcomes such as completed, failed, cancelled, and awaiting human input. These are proposed business labels, not built-in Agent Framework status values. Decide whether cancellation belongs in the failure denominator and whether a paused interaction remains active. Emit one terminal record per logical workflow request; keep retry attempts subordinate to that record.
For example, an application can define its failure rate as failed terminal requests divided by all terminal requests eligible for the service objective. It should also display active-request age and waiting counts. A workflow that never terminates can disappear from a terminal-only denominator, making the failure rate look healthy while callers remain blocked.
Interpret coordination evidence
Workflow spans include executor.process, message.send, and edge_group.process. The routing attributes edge_group.delivered and edge_group.delivery_status expose delivery outcomes. The documented status values are delivered, dropped type mismatch, dropped target mismatch, dropped condition false, exception, and buffered.
See 1.2.8 for routing-status semantics; here, aggregate each outcome against the intended graph so expected buffering and condition-based rejection do not inflate incident counts.
Track required handoff completion separately from mere send attempts. In a fan-in, three sends can still yield no usable combined output. Record the expected branch set and terminal coordination result in an application span or record. In Python, use get_meter().create_counter() and counter.add() for application-defined counters, while retaining workflow spans for diagnosis. Trace IDs belong in diagnostic records rather than counter dimensions.
Key distinctions and trade-offs
Invocation success versus task success: A returned agent result proves execution reached a return point. A business validator decides whether that result satisfies the task. Quality evaluation belongs in 3.3.3.
Delivery versus coordination: edge_group.delivered=true describes routing. It does not establish that every required specialist contributed or that the coordinator accepted their evidence.
Failure rate versus backlog: Terminal failures describe finished work. Active age and waiting populations reveal work that has not finished. A design needs both to detect stalled workflows.
Detailed spans improve diagnosis but increase telemetry volume. Stable role-level aggregation makes trends readable; per-request evidence explains an individual defect. Keep both views connected without using unique request identifiers as metric labels.
Architecture / decision scenario
An incident-summary service delegates to a retrieval agent and a classification agent, then combines their results. The retrieval agent’s invocation durations rise, but classification remains stable. The coordinator’s failure rate is low because blocked requests have not produced terminal records.
The team compares per-agent spans, logical request outcomes, and active-request age. Python workflow.run identifies the workflow execution, while executor.process isolates the slow specialist. Routing spans show buffered at the fan-in, with the retrieval contribution missing. The team treats buffering as expected until the application’s waiting deadline expires; that deadline is a service policy, not a documented framework timeout.
A second group of requests shows dropped type mismatch after a deployment. Those requests need a contract repair rather than more time at the fan-in. The team records distinct coordination outcomes, attributes them to the deployed versions, and keeps retry attempts out of the logical-request denominator. Foundry’s preview Monitor view supplies a convenient per-agent summary, while Application Insights receives the detailed evidence used to explain the workflow result.
Common exam traps
- Treating an agent’s successful return as end-to-end workflow success.
- Counting retry attempts as separate completed business requests.
- Calling every
bufferedordropped condition falseoutcome an error. - Mixing Python
workflow.runwith .NET session and invocation semantics. - Using terminal failure rates alone to detect stuck requests.
- Assuming a preview Foundry monitoring view has the same status as prompt-agent tracing.
What to remember
Monitor agent execution, logical workflow outcomes, and coordination separately. Use the documented agent histograms and workflow routing attributes, then add explicit business outcome records. A healthy chart needs a defined population, a stable grouping, and evidence for unfinished work.
Knowledge check
A workflow’s terminal failure rate stays low, yet callers report requests that never return. Which additional measurement detects this condition?
A specialist fails twice and succeeds on its third attempt. The workflow then completes. How should the application count this for logical workflow reliability and retry diagnosis?
A fan-in requires two specialists. Its routing telemetry shows one contribution as buffered, and the application waiting deadline has not expired. What should the monitor conclude?
A .NET workflow session can process several customer inputs before it stops, and each input is one business request. The team wants a failure rate per business request. Which span should define the denominator?
workflow.sessionworkflow_invokeworkflow.runexecutor.processworkflow_invoke covers one input-to-halt cycle, which matches one business request here. workflow.session spans the whole session and would mix populations, workflow.run is the Python workflow span, and executor.process measures a single executor step.