Skip to lesson

MicrosoftMicrosoft Certified: Multi-Agent AI Solutions Expert2026-103.3 Design and implement observability and monitoring for multi-agent solutions

Measure reliability against explicit service objectives

Completed by the knowledge check

Learning objective

Monitor availability, latency, throughput, reliability, and SLA/SLO adherence

Core idea

A service objective is meaningful only when its measurement matches the promise. An endpoint can respond while a multi-agent task fails, and a high request rate can reflect retries rather than useful throughput. Monitor availability, latency, throughput, and reliability as separate dimensions, then relate each to a defined population and time window.

For architects and senior engineers, a service-level indicator (SLI) is the measured behavior, a service-level objective (SLO) is the team’s target, and a service-level agreement (SLA) is a contractual commitment with its own terms. Azure Monitor and Application Insights provide measurements; they do not turn an application’s chosen target into a Microsoft contractual guarantee. See 3.3.1 for workflow outcome counting and 3.3.4 for alert dispatch.

How it works

Measure externally observable availability

Application Insights Standard availability tests send a single request to an endpoint and validate the response. They support HTTP verbs, custom headers, custom data, and TLS/SSL certificate checks. Use them to observe whether the tested endpoint responds from the configured locations under the selected validation criteria.

Standard tests are the documented choice for this design. URL ping tests are deprecated, with retirement extended to September 30, 2028. The availability page describes custom tests using TrackAvailability() as archived Classic API guidance. Do not choose an archived implementation merely because its name appears familiar in an exam scenario.

The metric availabilityResults/availabilityPercentage reports the percentage of test runs that pass validation; availabilityResults/duration measures test duration; availabilityResults/count counts test runs. These describe the synthetic-test population. A successful one-request endpoint test does not prove every specialist completed a business task.

The default Alert on X out of Y locations reporting failures rule supports location-based detection. Microsoft recommends at least five test locations and an alert location threshold of locations minus two; with Enable retries for availability test failures, each location reports failure only after three successive attempts fail.

Use a scoped health endpoint for infrastructure reachability, and separately record real workflow completion or run controlled business-path probes through your own test harness. A Standard test does not orchestrate an arbitrary multistep agent conversation. If the endpoint accepts work asynchronously, an acceptance response needs a separate observation of terminal completion.

Measure the latency the caller experiences

Application Insights exposes requests/duration for server response time and requests/count for tracked server requests. These measurements depend on where the request starts and ends. An asynchronous intake API can respond quickly while the workflow continues for minutes; its response duration is not the business task’s completion latency.

Record logical request start and terminal timestamps in your application telemetry to calculate the end-to-end indicator. Define whether the interval includes queue waiting, human approval, retries, and result validation. The measurement cannot be compared to an objective until that definition is stable.

Use Python Agent Framework’s gen_ai.client.operation.duration and agent_framework.function.invocation.duration to investigate model and tool contribution. These histograms use seconds and describe component operations, not the complete user journey. Component duration helps explain a tail-latency increase but does not replace an end-to-end measurement.

Choose percentiles that reflect your commitment rather than reporting only an average. If most requests finish quickly but a small cohort repeatedly waits for a missing fan-in contribution, the mean can conceal the caller-visible defect. Keep the eligible cohort and observation window beside each percentile.

Distinguish traffic from useful throughput

Application Insights requests/rate measures server request rate, while requests/count counts requests. In a multi-agent solution, one business request can produce several model calls, tool calls, and retries. Those counts answer a capacity question rather than proving how many tasks the application successfully delivered.

Count successfully completed logical tasks per unit of time as an application-defined useful-throughput indicator. Preserve attempt counts for diagnosing amplification, and track active work separately. If intake exceeds completion for a sustained interval, backlog can grow even when HTTP requests remain successful.

Choose stable grouping dimensions such as task class, agent role, and deployment cohort. Do not create a metric series for every request ID. Keep detailed traces for diagnosis and aggregate records for reporting. The log-based customMetrics records retain all dimensions, while the metrics-store version omits dimensions by default. To retain them in the metrics store, select With dimensions under Send custom metrics to Azure Metric Store on Usage and estimated cost. This feature is preview, and storing dimensions incurs separate billing.

Define reliability and the error budget

Application Insights requests/failed counts tracked server requests marked failed, and dependencies/failed counts failed dependency calls. Neither automatically identifies an unusable agent answer, a missing required contribution, or a business action that was never completed. Join transport-level evidence with the workflow terminal outcomes defined in 3.3.1.

Define a good request as one that meets the service’s completion contract within its applicable deadline. Then calculate reliability from good eligible requests divided by all eligible requests. Specify exclusions, cancellations, and requests awaiting human input before producing the report. Changing exclusions during an incident changes the promise rather than fixing performance.

Suppose your team chooses a request-based SLO of 99.9% good outcomes over a defined reporting period. The corresponding allowed bad-outcome fraction is 0.1% of eligible requests. For one million eligible requests, that is 1,000 bad outcomes. These are illustrative application targets and arithmetic, not Microsoft service guarantees. A time-based availability objective uses a different denominator and must not borrow this request budget unchanged.

Track how quickly the application consumes the budget as well as whether the period’s total has been exhausted. Configure Azure Monitor alert conditions for the measured indicator using 3.3.4’s mechanisms. The business policy determines the burn thresholds and recovery response; a low current error rate does not erase earlier budget consumption.

Keep the contract and feature scope explicit

Report the indicator’s source, calculation, population, window, and target together. Use Application Insights for measurements and correlated evidence, and compare them against the actual SLA terms when a contractual report is required. A dependency’s SLA does not itself establish the end-to-end agent application’s SLA.

Foundry’s agent metrics view and recurring evaluations are preview and have no service-level agreement. Preserve the tracing support distinctions from 3.3.1; neither a dashboard nor an evaluation score supplies the service commitment.

Key distinctions and trade-offs

Synthetic availability versus business reliability: Availability tests verify their configured endpoint criteria. Business reliability measures eligible tasks against a completion contract.

Request duration versus completion latency: The server response interval can end before asynchronous work finishes. Measure the interval promised to the caller.

Traffic versus throughput: Request rate measures incoming or tracked work. Useful throughput measures completed logical outcomes. Retries can increase traffic while reducing useful capacity.

Stricter objectives protect callers but require capacity and operational investment. Narrow probes are cheap and easy to interpret, while business-path checks expose more dependencies. Use both according to the promise each measurement can substantiate.

Architecture / decision scenario

The incident-summary service accepts asynchronous jobs. Standard tests show the intake endpoint is available, and requests/duration remains low. At the same time, retries increase requests/rate, a retrieval specialist slows down, and the number of delivered summaries per minute falls.

The team keeps the synthetic availability result, intake latency, completion latency, and useful throughput as separate indicators. It uses agent operation and tool histograms to investigate the bottleneck, and logical terminal records to calculate the fraction of eligible jobs delivered before the agreed deadline. Active-job age exposes callers still waiting.

For the illustrative 99.9% request-based SLO, the team accounts for bad outcomes against the reporting-period budget and triggers the configured Azure Monitor recovery and escalation process. It does not claim SLA adherence from a passing endpoint probe or borrow a provider’s guarantee. The final report states which measurements support each application objective and which contractual terms require a separate comparison.

Common exam traps

  • Reporting a synthetic test percentage as successful business-task completion.
  • Measuring asynchronous completion with intake API response duration.
  • Treating retry-inflated request rate as useful throughput.
  • Using dependency failures alone as the application’s reliability denominator.
  • Applying a request-based error budget to a time-based objective unchanged.
  • Inferring an SLA from a preview dashboard or a team-selected SLO.

What to remember

Define the promise before choosing the metric. Use Standard tests for scoped availability, server metrics for HTTP behavior, component histograms for diagnosis, and logical outcomes for end-to-end reliability. Keep SLIs, SLOs, and contractual SLAs explicit, with compatible populations and windows.

Knowledge check

References