Detect quality changes in production traffic

Learning objective

Detect behavioral drift and quality regression through continuous evaluation

Core idea

An agent can remain available and fast while its answers or tool choices deteriorate. Operational monitoring detects execution problems; continuous evaluation detects changes in the behavior of completed interactions. Behavioral drift means the observed behavior changes. Quality regression means the behavior performs worse against an explicit quality criterion. The first does not automatically prove the second.

For architects and senior engineers, the central design combines live-traffic evidence with a stable benchmark. Live traffic reveals what the deployed agent encounters; controlled cases reveal whether the same tasks became harder for the agent after a change. See 3.1.2 for evaluation configuration and 3.1.3–3.1.4 for response and trajectory criteria. This lesson connects those evaluation contracts to ongoing detection.

How it works

Choose the evaluation trigger

Microsoft Foundry distinguishes scheduled evaluation, which runs on a fixed schedule, from continuous evaluation, which samples live traffic as it occurs. Its monitoring documentation marks recurring evaluations as preview. Treat their results as quality evidence with the feature’s documented status; do not describe the monitoring feature as a production service-level guarantee.

The Python path creates an evaluation definition with openai_client.evals.create(), including data_source_config and testing_criteria. For live responses, the documented data source configuration uses type="azure_ai_source" and scenario="responses". Each testing criterion names the evaluator to run. The definition identifies the quality contract; a rule identifies when and for which agent to apply it.

Create the continuous rule with project_client.evaluation_rules.create_or_update() and an EvaluationRule. The documented combination is ContinuousEvaluationRuleAction(eval_id=..., max_hourly_runs=...), EvaluationRuleEventType.RESPONSE_COMPLETED, EvaluationRuleFilter(agent_name=...), and enabled=True. These identifiers matter: completing a response triggers evaluation of selected traffic, not execution of every possible task variant. See 1.2.8 for why this trigger needs separate monitoring of work that never completes.

The documentation’s example sets max_hourly_runs=100. That is an example configuration, not a universal service quota or proof that every response is evaluated. Choose the budget for your workload and track the evaluated population. A low-traffic or filtered cohort can yield too little evidence even when the rule is enabled.

Give the evaluator access

The monitoring page requires assigning the project managed identity the Foundry User role to enable continuous evaluation rules. Connecting Application Insights does not by itself grant that identity the evaluation permissions it needs. Separate telemetry access for investigators from the identity executing the evaluation workflow.

Verify that the rule actually produces results for the intended agent after a test response completes. An enabled configuration with no results is not evidence of perfect quality. Check the filter, triggering traffic, identity, and data collection path before interpreting a flat dashboard. Keep a separate indicator for evaluation coverage or result age so missing evidence does not become an apparent pass.

Select a criterion that detects the failure

Use agent evaluators whose input and output match the behavior under investigation. Foundry’s Task Adherence evaluator, identified as builtin.task_adherence, checks actions against assigned instructions and constraints; its page marks it preview. Tool Call Accuracy evaluates whether the agent makes the right tool calls with correct parameters, while Tool Call Success measures technical execution errors. These are different questions, even when both concern tools.

A successfully executed tool call can use the wrong argument or be irrelevant to the task. Conversely, a correct tool choice can fail because its dependency is unavailable. Keep the evaluator’s verdict separate from transport error counts. Feed the evaluator the interaction evidence it requires, and avoid attributing every low score to a prompt defect.

Retain the evaluation definition, evaluator configuration, agent version, and relevant deployment changes with the result cohort. These associations are your comparison design, not a claim that every identifier is automatically a dashboard dimension. A changed judge or threshold can change the pass rate even when agent behavior is unchanged.

Compare live traffic and fixed cases

Use continuous results to observe production behavior by stable task cohort, then compare like populations. If the agent begins receiving more complex requests, its overall score can fall without a regression on previously supported tasks. Segment the evidence using application task labels and inspect representative interactions before concluding that a deployment caused the change.

The monitoring page’s scheduled example evaluates production traces through the preview azure_ai_trace_data_source_preview data source. Its trace_source uses type="agent_filter", agent_name, start_time, end_time, and max_traces to select a time-windowed agent sample. Python Schedule, RecurrenceTrigger, DailyRecurrenceSchedule, and EvaluationScheduleTask, submitted through project_client.beta.schedules.create_or_update(), provide the fixed recurrence. The distinction from RESPONSE_COMPLETED is the trigger, not an exclusively benchmark population.

For a controlled comparison, use the dataset runs and mappings in 3.1.2 to evaluate a maintained benchmark repeatedly. That is this architecture’s design choice: hold cases constant to separate behavior changes from changes in incoming traffic.

For SDK scope, the page documents Python and .NET rules and schedules; the .NET path uses RecurrenceTrigger and EvaluationScheduleTask. JavaScript/TypeScript supports continuous evaluation rules, but the page directs fixed-recurrence scheduled evaluation to the Foundry portal.

Turn trends into a regression decision

Define a baseline window, minimum evidence requirement, allowed degradation, and review owner as application policy. Microsoft provides the evaluation and scheduling mechanisms; it does not determine your business acceptance threshold. State the criterion and denominator beside the trend so a score change has an interpretable meaning.

Compare both score distributions and pass/fail outcomes where the evaluator supplies them. A stable average can hide a worsening minority of tasks. Keep failure examples and trace associations for investigation, and check whether a model, prompt, tool contract, retrieval source, or evaluator configuration changed. Continuous evaluation detects an observable pattern; attribution still requires evidence.

Key distinctions and trade-offs

Drift versus regression: Drift is a change in observed behavior or traffic. Regression is worse performance against the agreed criterion. Changed traffic can produce either apparent or real degradation.

Continuous versus scheduled: Continuous evaluation responds to completion events; scheduled evaluation runs at a fixed recurrence and can also evaluate production traces. A controlled benchmark is a separately chosen comparison population.

Rule enabled versus coverage established: enabled=True permits the rule to operate. Its filter, hourly budget, and observed result production determine what evidence you actually receive.

More live evaluation improves visibility but adds evaluation work and judge cost. A fixed benchmark improves comparability but can miss production tasks it never includes. Keep both useful by recording their scope and maintaining representative cases.

Architecture / decision scenario

The incident-summary service deploys a changed classification prompt. Availability and latency remain stable, but a continuous evaluation rule for that agent reports weaker instruction adherence. The rule uses RESPONSE_COMPLETED, filters by agent name, and applies the team’s chosen hourly budget through max_hourly_runs.

The team verifies result coverage and the Foundry User assignment rather than interpreting missing evaluations as passes. It compares task cohorts and discovers that production now includes a larger share of multilingual incidents. The team also configures scheduled production-trace evaluation, then repeats the dataset runs from 3.1.2 on established English cases and a maintained multilingual benchmark cohort, with the evaluator configuration held constant.

English cases remain stable, while the multilingual cohort degrades after the prompt change. That supports a scoped quality regression, not a claim that the entire service is unavailable. The team links failed evaluations to traces and version metadata, tests a correction against the benchmark, and uses 3.3.4’s alert and escalation path for repeated policy-significant failures. Task Adherence and recurring monitoring remain explicitly identified as preview features.

Common exam traps

• Treating HTTP success as evidence of answer or tool-choice quality.
• Calling every production score change a deployment regression.
• Reading max_hourly_runs=100 as a universal limit or full-traffic coverage.
• Assuming response-completion evaluation tests unfinished or abandoned requests.
• Comparing results after changing the evaluator or threshold without accounting for it.
• Assuming Python scheduling support implies the same JavaScript/TypeScript API.

What to remember

Continuous evaluation samples completed production responses; scheduled evaluation can sample production traces at a fixed recurrence. Fixed benchmark runs provide controlled comparisons. Name the event, filter, evaluator, identity role, and budget. Detect changes first, then establish coverage and comparable populations before declaring a quality regression.

Knowledge check

References

• Microsoft, Monitor agents with the Agent Monitoring Dashboard
• Microsoft, Agent evaluators