NVIDIANVIDIA-Certified Professional: Agentic AI2026-108.3 Continuously benchmark deployed agents against prior versions.
Separate release effects from production drift
Completed by the knowledge check
Learning objective
Compare versions online with A/B tests and detect drift.
Core idea
An online version comparison asks whether a candidate changes outcomes under production conditions. Drift monitoring asks whether the conditions or behavior of a deployed system change over time. A candidate can outperform a baseline while both become unacceptable because the workload or a shared dependency changes.
Use concurrent A/B cohorts to estimate the release effect, then compare each cohort with its own history and absolute service requirements. Google SRE’s canarying guidance explains why before/after comparisons make attribution difficult and why shared dependencies can degrade both groups together. NIST AI RMF MEASURE 2.4 makes monitoring production behavior a continuing responsibility.
See 8.3.1 for the maintained offline benchmark. Passing that benchmark earns consideration for live exposure; online results answer questions about real traffic, contention, and user outcomes that a fixed dataset may miss.
How it works
Define the experiment before allocating traffic
Write an experiment record containing the baseline and candidate artifact identifiers, eligible population, assignment unit, allocation, primary outcome, safety limits, planned duration, and decision owner. Specify the minimum useful improvement and analysis plan before observing results; otherwise repeated inspection encourages promotion on a temporary favorable fluctuation.
Use the session-stable cohort routing from 4.2.3 to keep each conversation in one arm. For experiment analysis, cross-arm conversation state creates contamination: a response in B can change a later request served by A, as Google SRE describes.
Check allocation and task mix before interpreting the outcome difference. If eligibility or routing sends difficult requests disproportionately to B, the versions are not receiving the comparison population you intended. A weighted routing percentage describes exposure; it does not by itself establish that the groups are comparable.
Use NeMo Agent Toolkit’s general.telemetry.tracing exporters to collect per-request execution traces, then join their trace IDs to the experiment record. Keep assignment, served artifact, task outcome, and observation time in that record, using fields such as experiment_id, assigned_arm, served_revision, and outcome_status. Record fallbacks under the assigned cohort as well as the version that actually answered. Otherwise failures that escape to the baseline can disappear from the candidate’s reported results.
Measure outcomes and constrain exposure
Choose an outcome close to the business task, such as a validated completed requisition rather than a nonempty response. Define the denominator to include eligible failed requests. Track task success alongside end-to-end latency, errors, and resource or monetary cost. For delayed user reactions, POST /feedback accepts observability_trace_id and reaction_type when using the Weave FastAPI plugin worker; see 3.3.1 for feedback capture. Store the trace ID with assigned_arm and served_revision so that a late reaction joins the original assignment. For reactions and other delayed task outcomes, wait for the specified observation window rather than comparing incomplete outcomes with mature ones.
A fallback and a late reaction both join the experiment record under the original assignment.
Use the staged canary and operational stop conditions from 4.2.3 to bound exposure while the experiment estimates outcome differences.
Evaluate separate cohort metrics over intervals aligned with the experiment. An hourly aggregate mostly containing pre-experiment traffic can obscure a ten-minute canary. Enforce absolute service-level objectives (SLOs) as well as relative differences. If both cohorts violate the latency SLO because they share an overloaded tool service, a small A/B difference is no assurance of service health.
Reserve promotion for the planned outcome analysis with sufficient completed observations. The evaluation owner assesses uncertainty; the service owner accepts the quality and cost trade-off. Neither a traffic percentage nor an arbitrary elapsed hour guarantees an adequately powered comparison.
Distinguish workload drift from outcome degradation
Maintain a production-monitoring record with reference windows, task and cohort breakdowns, data completeness checks, quality measures, response owners, and escalation conditions. NIST’s Generative AI Profile describes post-deployment monitoring plans under MANAGE 4.1 and tracking dataset modifications for provenance under MG-4.1-006. Apply that structure to the deployed agent and its knowledge sources.
Monitor input characteristics such as task proportions, language, document type, and tool selection, using approved aggregate data. A changed input distribution signals workload drift; it does not establish declining quality. Compare reviewed task outcomes within those slices. Input characteristics can stay stable while expected answers change after a procurement policy revision, so outcome monitoring remains necessary.
Keep historical and concurrent views. If B declines relative to A under matched conditions, investigate the candidate. If both decline while the difficult-task share rises, inspect workload coverage and the golden dataset. If both decline within unchanged task slices, investigate shared tools, retrieval content, policy changes, and scoring changes. Check missing telemetry before assigning any of these causes: absent outcomes can resemble improved success if failures disappear from the denominator.
Use the remote nat eval --endpoint mechanism from 8.3.1 to replay the maintained benchmark against identified deployed versions. NeMo Agent Toolkit’s workflow_output.json and configured evaluator outputs support the per-case comparison. This controlled replay complements the live experiment: stable-case deterioration suggests a system or dependency change, whereas stable replay alongside worsening production outcomes points toward a coverage gap to investigate. Neither result alone proves the cause.
Key distinctions and trade-offs
Choose the comparison according to its question:
| Method | Deciding condition | Mechanism | Cost |
|---|---|---|---|
| Concurrent A/B test | Does the candidate improve live outcomes? | Stable cohort assignment and per-arm outcome analysis | Exposes users and splits traffic |
| Drift monitoring | Has the operating environment or behavior changed? | Reference windows, slice distributions, and reviewed outcomes | Requires maintained references and labels |
| Fixed-case replay | Does an identified deployment still pass known tasks? | nat eval --endpoint on a versioned dataset | Cannot cover every production change |
Broader exposure produces observations faster but increases potential harm and consumes shared capacity. A preserved control helps attribution during a bounded experiment, but cannot protect against common dependencies. Continue absolute monitoring after the experiment ends.
Architecture / decision scenario
A manufacturing procurement assistant has a six-second p95 latency SLO. Its service owner approves a 10% candidate cohort with stable session assignment. The tempting alternative is to compare this week’s candidate with last week’s baseline, avoiding the cost of running two versions. A supplier migration changes the workload during the week, making that comparison unreliable.
After the planned outcome window, A completes 920 of 1,000 reviewed tasks and B completes 950 of 1,000. These illustrative observations suggest a benefit, subject to the team’s planned uncertainty analysis. However, both cohorts now have eight-second p95 latency. The relative quality improvement does not authorize expansion beyond the absolute six-second limit.
Monitoring also shows that multilingual requisitions rose from 5% to 25% of traffic. The evaluation owner separates that workload shift from the release effect by comparing success within language slices, as taught in 3.2.2. The platform owner finds shared supplier-tool saturation and increases capacity for that service while promotion stays paused. The procurement owner reviews multilingual failures for the next benchmark revision. Promotion waits for acceptable service health and the completed quality analysis; drift triggers investigation rather than an automatic model retraining decision.
Common exam traps
- Treating a before/after comparison as equivalent to concurrent cohorts.
- Assuming weighted routing alone creates comparable populations.
- Reassigning turns inside a stateful conversation.
- Counting fallback successes while dropping the candidate failures that caused them.
- Calling a changed input distribution proof of worse answers.
- Promoting the better cohort when both violate an absolute SLO.
What to remember
Compare versions concurrently with stable assignment and explicit outcome definitions. Bound live exposure with operational limits, and analyze drift through reference windows and task slices. A/B differences, absolute service health, input shifts, and fixed-case replay answer different questions; use them together before deciding whether to promote, investigate, or refresh the benchmark.
Knowledge check
A travel-booking agent handles multi-turn booking conversations and has a release candidate. To save serving cost, the team proposes replacing the baseline for one week and comparing the candidate’s task success with the previous week’s. A fare-rule change affecting all bookings takes effect mid-week. Which design lets the team attribute an outcome difference to the release?
A support agent has a five-second p95 latency SLO. After the planned observation window, the candidate cohort completes more reviewed tasks than the control, but both cohorts run at a seven-second p95, and both call the same inventory tool service. The control also had seven-second p95 latency before candidate exposure. The service owner asks whether to expand the candidate to 50% of traffic. What should the team do?
When the candidate’s tool call fails, a gateway retries the request on the baseline, which usually succeeds. The experiment dashboard groups outcomes by the version that answered, and the candidate cohort shows a higher success rate than the control. How should the experiment record count these requests?
assigned_arm ), with the version that actually answered recorded beside it ( served_revision ); otherwise candidate failures that escape to the baseline vanish from the candidate’s results. Excluding fallbacks drops eligible failures from the denominator, and a separate group takes them out of the candidate’s results just the same.No release is in progress for a deployed agent. Monitoring shows Spanish requests rising from 5% to 25% of traffic while overall task success falls four points. Within each language slice, reviewed success is unchanged, and telemetry completeness checks pass. How should the team treat this?
The team replays benchmark revision 9 against a deployed agent with nat eval --endpoint and gets the same per-case results as at approval; the replay calls the same live tools and retrieval content as production traffic. Over the same weeks, reviewed production outcomes worsen, and telemetry completeness checks pass. Which hypothesis should the team investigate first?
References
- Google, Site Reliability Workbook: Canarying Releases
- NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0)
- NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
- NVIDIA, Agent Evaluation in NVIDIA NeMo Agent Toolkit
- NVIDIA, NVIDIA NeMo Agent Toolkit API Server Endpoints
- NVIDIA, Observe Workflows