Keep release benchmarks representative and comparable

Learning objective

Maintain production-representative golden datasets and run regression evaluations on every release.

Core idea

A golden dataset is a reviewed reference for deciding whether a release remains acceptable. Its value comes from correct expected outcomes and coverage of deployment conditions, rather than from its size alone. A frozen collection can preserve comparability while becoming irrelevant as users, policies, and tools change. A continually overwritten collection can reflect production while making release comparisons meaningless.

Maintain both an immutable benchmark revision and a deliberate refresh process. Compare the candidate and the deployed baseline on the same revision. When production requires a revised benchmark, evaluate both versions again before interpreting the difference. NIST AI RMF’s MEASURE 2.1 calls for documented test sets and metrics; MEASURE 2.3 ties evaluation to deployment-like conditions. Those principles make benchmark maintenance an operational responsibility.

See 3.1.1 for constructing task cases and 3.1.4 for the evaluation harness. This lesson adds the lifecycle that keeps those cases useful across production releases.

How it works

Maintain two complementary collections

Keep a stable regression core containing consequential historical failures, policy requirements, and critical tool outcomes. Alongside it, maintain a representative collection refreshed from an explicitly defined production window. Document that window, sampling rules, exclusions, task proportions, and labeling ownership in a revision record your team keeps beside the dataset.

Preserve rare severe failures in the regression core even when they disappear from the production sample. Conversely, do not describe a deliberately oversampled hard-case collection as an estimate of ordinary traffic quality. Report its results separately. See 3.3.2 for privacy-aware feedback curation; the maintenance question here is which populations a release benchmark must still represent.

Refresh when the monitored workload changes, not only when someone edits the model. A return-policy revision can invalidate expected answers even if the agent binary stays identical. Record a correction as another dataset revision with its reason and reviewer. Keep the previous revision available for historical interpretation instead of silently rewriting its answers.

For a task with time-dependent data, bind the expected result to a dated policy or a controlled tool response. Otherwise a correct answer under one inventory snapshot can be scored against another snapshot. The benchmark record should distinguish a real agent regression from an obsolete expectation. The domain owner approves answer changes; the evaluation owner implements and checks them.

Run the deployed baseline and candidate

Use the matched-comparison controls from 3.2.1, and replay the deployed baseline through nat eval --endpoint on every release using the same dataset revision as the candidate. See 4.2.2 for the release bundle that identifies the remote deployment independently of its endpoint URL.

Publish each benchmark revision under a distinct object path in shared storage. NeMo Agent Toolkit’s eval.general.dataset.remote_file_path selects that object; the s3 block supplies endpoint_url, bucket, access_key, and secret_key for access. The dataset downloads to the local file_path before the run. Preserve the published object instead of replacing it with different contents under the same path.

The --dataset=BENCHMARK_REVISION_PATH argument selects a specific local dataset file without editing the evaluation configuration; replace BENCHMARK_REVISION_PATH with the revision’s local file path. config_metadata.json records this argument. Retain a content digest beside the revision record so that a named path can be checked against the actual bytes both runs used.

Run release tests against isolated data and tool permissions appropriate to the benchmark. An agent that issues a real refund during evaluation changes the world for subsequent cases. A controlled refund fixture makes the expected result repeatable; testing the real payment integration is a separate release responsibility with its own authorization and cleanup.

Make regression results a release decision

Keep the separate run artifacts described in 3.1.4, attaching the same dataset revision and digest to the baseline and candidate results. For a critical slice, use eval.general.dataset.filter with allowlist or denylist as taught in 3.2.2, and report that result separately from the representative sample’s aggregate.

Define acceptance criteria before running the candidate: critical-case requirements, tolerated quality changes, latency and cost limits, and treatment of incomplete runs. These thresholds belong to the service’s release policy. A completed command is not equivalent to passing that policy. Compare per-case failures as well as aggregates so that a gain on common tasks cannot erase a severe regression.

Execute this comparison for every release candidate that changes observable behavior, including a retrieval or tool deployment without a model change. If an external dependency changes independently, schedule another baseline run and attach the dependency revision. Continuous benchmarking combines release-triggered runs with periodic checks of the deployed system; it does not end when the candidate is promoted.

Key distinctions and trade-offs

The collections answer different release questions:

Preserving every historical case forever increases execution cost and can overweight obsolete tasks. Retire a case through a documented revision when its requirement disappears, while retaining the historical dataset. Keeping the requirement but deleting its difficult case would weaken the benchmark precisely where it found a defect.

Architecture / decision scenario

A retailer’s returns assistant releases weekly. Its benchmark contains a 600-case regression core and a 400-case production sample. These are the team’s chosen sizes. After a cross-border returns expansion, those requests rise from 2% to 18% of traffic, but remain 2% of the sample. The tempting choice is to reuse the benchmark unchanged because it produces comparable weekly scores.

The evaluation owner publishes a refreshed 400-case sample containing 72 cross-border cases, matching the 18% traffic share, and reruns both versions on it. Separately, a reviewed critical-case suite deliberately oversamples 200 cross-border cases. On that same suite, the baseline passes 188 and the candidate passes 176, despite the candidate improving overall quality on the representative sample. The release policy permits at most a two-percentage-point decline in that critical slice; the observed decline is six points. The release owner withholds promotion and assigns the failures to the policy-tool team.

The returns-policy owner reviews the expected outcomes, confirming that the changed policy is reflected correctly. After repair, the candidate must pass the stable core, the refreshed sample, and the separately scored critical-case suite. Replacing the benchmark and reporting only the candidate’s aggregate would hide whether the release improved anything relative to the deployed version.

Common exam traps

• Treating an unchanged golden dataset as automatically production-representative.
• Editing expected answers in place and comparing scores across different revisions.
• Removing rare severe cases because they are uncommon in production.
• Assuming a remote endpoint’s workflow configuration is saved by local evaluation.
• Promoting because an aggregate improves while a critical slice fails.
• Running release evaluations only when model weights change.

What to remember

Preserve comparability within a benchmark revision and refresh representativeness through reviewed revisions. Run both the deployed baseline and candidate on identical cases, retain configuration and per-case outputs, and apply explicit release criteria. A changing production workload requires benchmark maintenance even when the agent itself has not changed.

Knowledge check

References

• NVIDIA, Agent Evaluation in NVIDIA NeMo Agent Toolkit
• NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0)