Keep evaluation sets useful across releases

Learning objective

Version and maintain evaluation sets, guarding against contamination and adding production failures as new cases

Core idea

A score is reproducible only when you know which tasks, labels, graders, environment, and candidate produced it. If those inputs change silently, an improving chart can reflect an easier dataset or a weaker rubric instead of a better agent.

Anthropic treats evaluation suites as living artifacts with clear ownership, recommends converting production failures into tasks, and warns that shared trial state can inflate scores. Turn those principles into a versioned evaluation lifecycle. Protect comparison sets while allowing deliberate, reviewed changes that keep the suite relevant.

How it works

Version the experiment, not only the prompt

Keep an immutable dataset release identifying its case membership, fixtures, labels, and split assignments. Pair it with a run record identifying the candidate prompt, model configuration, tool implementation, grader configuration, and environment snapshot. Anthropic’s evaluation harness concept includes task execution, recording, grading, and aggregation; versioning those inputs makes its result interpretable.

Keep case identity stable when correcting a label, but publish another dataset revision and record why the expected answer changed. If the underlying task changes materially, introduce a distinct case with provenance to its predecessor. This avoids representing two different tasks as the same historical measurement.

When comparing candidates, run both against the same dataset and graders. When comparing dataset revisions, retain a common subset or rerun a baseline across both. Otherwise, a rising score cannot distinguish agent improvement from a changed population. Preserve both original and corrected results when a grading defect invalidates an earlier conclusion.

Protect holdouts from development feedback

Anthropic’s evaluation documentation uses held-out test sets, and its tool-engineering article reports held-out performance separately from optimization. Implement that separation through dataset roles: development cases guide iteration, regression cases protect supported behavior, and a held-out sample supports release comparison without becoming a prompt-editing target.

Restrict access to held-out inputs, labels, and detailed failure reports for people or agents tuning the candidate. Keep the expected answer and grader-only material outside the agent’s task environment. Do not insert held-out cases into few-shot examples or retrieval material that the candidate can access during evaluation.

Group related examples before splitting. Two paraphrases of the same incident, or synthetic variants sharing the same source document, can expose the solution across splits despite having different case IDs. Assign related variants to one case family and keep that family in one split. Validate split membership as part of publishing the dataset release.

Holdout secrecy has a lifecycle. If developers need detailed failures to diagnose a problem, record that exposure and move the exposed family into development or regression. Replace the held-out coverage with independent tasks. Even aggregate scores can guide repeated tuning; limit release-set consultation and refresh the sample under an explicit owner rather than treating one unchanged set as permanently independent.

Isolate trial state as well as dataset roles

Contamination also occurs inside the evaluation environment. Anthropic reports agents gaining an unfair advantage by inspecting git history from previous trials. Leftover files and cached task artifacts can similarly make later attempts easier than the intended task.

Initialize every trial from its specified fixture and isolate writable state. Ensure that restoration removes prior outputs and solutions, not just the final answer file. Inspect the environment when a surprising score increase coincides with repeated execution.

These controls reduce observable evaluation leakage. They do not prove that a model’s training data never included a public benchmark. Do not describe a private holdout procedure as a guarantee about a vendor’s training corpus. For a consequential release, favor independently constructed business tasks whose origin and exposure you can document.

Promote incidents through a review queue

A production failure is a candidate case, not automatically a valid reference label. Capture the authorized input, relevant source state, observed behavior, and expected outcome. Reproduce the failure in an isolated fixture; remove unnecessary sensitive information under the organization’s data-handling rules while retaining the mechanism that caused the defect.

Have the domain owner verify the expected outcome and the evaluation owner confirm that the grader rejects the reproduced failure and accepts a known valid solution. Then admit the case to the next dataset revision, tag its failure family, and include it in the appropriate regression subset.

A case changes role as it is exposed or reviewed: an exposed held-out family moves into development or regression, and a reviewed incident is admitted to a regression subset.

If the incident reflects ambiguous policy, resolve that policy before labeling. If it reflects a broken test dependency, repair the harness and retain an environment check rather than mislabeling the event as a model failure. Deduplicate incidents into families so a frequently reported defect does not overwhelm the traffic estimate.

Read failed transcripts to determine whether the agent made a mistake or the evaluation rejected a valid, potentially better solution. Anthropic recommends revising the evaluation when grading is unfair, tasks are ambiguous, valid solutions are penalized, or harness constraints distort performance; version that repair before treating the score drop as an agent regression.

Review coverage as the product changes. A saturated capability set can become regression coverage while more demanding tasks measure further progress. Anthropic recommends dedicated evaluation infrastructure ownership with domain experts and product teams contributing tasks. Assign those roles explicitly rather than leaving case maintenance to whoever notices a failing dashboard.

Key distinctions and trade-offs

Stable benchmark versus relevant coverage: An unchanged set supports longitudinal comparison but can miss changed traffic. Version changes and rerun a common baseline to preserve comparability while extending coverage.

Incident archive versus regression suite: An archive records events. A regression case needs reproducible inputs, accepted expectations, a trustworthy grader, and an owner. Adding raw logs alone does not create a test.

Exposure versus invalidity: A disclosed holdout case can remain useful as a regression test. It loses its role as independent evidence for the candidate tuned against it; preserve the case and change its role.

Architecture / decision scenario

An education provider evaluates an assistant that explains enrollment requirements. On the development set it improves from 88% to 98%, while sampled production answers remain at 89%. The release owner has two specialist days to investigate before enrollment opens.

The tempting proposal is to add special instructions for every failed release case and reuse the same release score. The architect instead audits dataset provenance and finds that held-out answers were copied into prompt examples, while repeated trials could read a previous solution file. Neither score supports the claimed generalization.

The evaluation owner records the exposure, moves affected families into regression, removes prior-trial artifacts, and creates an independently labeled replacement holdout. The baseline and candidate run under the same corrected harness and dataset revision. The report preserves the invalidated run rather than overwriting its history.

A production incident involving conflicting admission dates becomes a reviewed case: the policy owner settles the authoritative date and the expected clarification behavior. Its fixture, label revision, failure family, and grader enter the next dataset release. The release owner accepts the staffing cost because it restores a meaningful comparison and prevents the same incident from disappearing after its prompt fix.

Common exam traps

• Versioning the prompt while silently editing labels or grader thresholds.
• Treating different case IDs as proof that related examples cannot leak across splits.
• Calling a repeatedly inspected release set an untouched holdout.
• Resetting conversation history while leaving prior-trial solution files accessible.
• Adding raw production logs without reproducing and labeling the failure.
• Deleting saturated tasks instead of preserving their regression value.

What to remember

Version tasks, labels, graders, environment, and candidate together. Separate development feedback from held-out comparison and isolate each trial. Convert verified incidents into reviewed regression cases, retain historical provenance, and give dataset maintenance explicit infrastructure and domain owners.

Knowledge check

References

• Anthropic, Define success criteria and build evaluations
• Anthropic, Demystifying evals for AI agents
• Anthropic, Writing effective tools for agents — with agents