Build evaluation cases that mirror production

Learning objective

Build representative evaluation datasets from real task distributions, expert cases, edge cases, adversarial inputs, and validated synthetic data

Core idea

An evaluation score describes a population of tasks under specified conditions. A convenient collection of clean examples cannot establish readiness for traffic containing ambiguous requests, unusual documents, or attempts to redirect the agent. As the architect, you choose which population a score represents and which risks require separate tests.

Anthropic recommends task-specific evaluations that mirror real task distributions and include edge cases. Its agent-evaluation guidance also recommends balanced cases where a behavior should and should not occur. Build both a representative sample for expected performance and targeted challenge cases for consequential failures; preserve the distinction when reporting results.

How it works

Define a case before collecting examples

Write a task specification that includes the input, relevant environment, permitted actions, and observable success criteria. For an agent that changes business records, include the expected final state. A fluent completion message is insufficient when the task requires an actual change.

Each evaluation case records its source, stratum, input fixture, expected outcome, and label status so the release owner can inspect provenance and coverage.

A reference solution establishes that the task can be solved and the graders accept a valid result. Anthropic recommends tasks on which two domain experts independently reach the same pass/fail verdict. If experts disagree, clarify the task or its success criteria before counting the case. Sometimes the correct outcome is to request missing information; do not manufacture an answer to make every example answerable. Anthropic also advises checking the task specification and graders when a frontier model scores 0% pass@100, because this most often signals a broken task rather than model incapability; see 4.3.3 for interpreting repeated-trial metrics.

Give each source a distinct purpose

Use the bug tracker and support queue to find user-reported failures that can become realistic tasks, as Anthropic recommends. Build the population from several sources, each addressing a different blind spot:

For adversarial cases, include ordinary documents that resemble an attack without containing one. Otherwise, a system can improve its challenge score by refusing every unusual document. For missing-data cases, include neighboring examples with sufficient information, so always asking a question cannot become the winning strategy.

Separate frequency from challenge coverage

Stratify on properties that plausibly change behavior: task type, source quality, language, number of required steps, or authority to write. Record the prevalence of each stratum in the sampled traffic. Oversample rare but consequential tasks when ordinary sampling leaves too few examples to inspect them.

Oversampling changes the denominator. Report per-stratum counts and scores. If you publish an expected-traffic score, weight the representative strata using measured traffic shares. Keep constructed adversarial cases in a separate challenge result unless you have a defensible prevalence estimate. An unweighted score over a deliberately difficult suite answers a different question.

Anthropic describes 20–50 simple tasks drawn from real failures as a useful starting point for early development, when changes have large effects. That is not a production certification threshold. Smaller differences and narrower slices require more observations; avoid claiming that a handful of successes establishes a rare-error rate.

Validate synthetic expansion

Anthropic’s evaluation documentation suggests generating additional cases from a baseline set. Treat generation as candidate production, followed by admission review. Ask a domain expert to check whether the input is realistic, the reference answer follows from the supplied material, and the case adds a distinct failure opportunity.

For numeric tasks, verify the reference calculation independently. For source-grounded tasks, verify that the source actually supports the label. Reject impossible combinations and duplicates that only change names. Keep synthetic provenance visible so a release owner can see whether an apparent improvement exists only on generated examples. Dataset splitting and maintenance belong to 4.2.4.

Key distinctions and trade-offs

Representative versus comprehensive: A representative sample estimates performance on a defined traffic mix. A challenge collection exposes specific weaknesses. Neither replaces the other, and combining their scores without an explicit weighting rule obscures both.

Expert cases versus expert labels: Experts can propose a difficult problem or judge an existing one. You need both useful task selection and trustworthy expected outcomes; expert authorship alone does not settle an ambiguous rubric.

Synthetic volume versus independent coverage: More cases reduce manual collection effort, but repeated templates can amplify one generator’s assumptions. Spend review effort on distinct task families before expanding surface variations.

Architecture / decision scenario

A retail architect must approve a returns assistant before a seasonal launch. Observed requests are 80% ordinary returns, 15% damaged goods, and 5% disputed purchases. The team can label 200 cases this week. Product proposes 200 generated ordinary returns because they are quick to score.

The architect instead reserves 120 real cases in those traffic proportions: 96 ordinary, 18 damaged, and 6 disputed. Specialists add 20 expert policy conflicts, 20 legitimate edge cases, 20 adversarial messages, and 20 synthetic cases for poorly represented language and document combinations. Every synthetic case receives independent label review.

Suppose ordinary returns pass at 98%, damaged goods at 89%, and disputed purchases at 60%. The expected-traffic score is 94.75%, using the stated traffic shares, but the disputed-purchase weakness remains visible. Challenge cases receive their own results. A high combined score cannot authorize disputed-purchase automation.

The returns-policy owner resolves disputed labels and defines which cases should request human review. The architect owns coverage and the release recommendation. This costs more expert time than generating 200 easy cases, but tests the business decision the launch actually depends on.

Common exam traps

• Treating convenient clean examples as representative traffic.
• Counting generated answers as verified reference labels.
• Mixing oversampled challenges into a headline traffic score without weighting.
• Testing only when an action should occur, so unconditional action passes.
• Labeling missing evidence as a model failure when clarification is correct.
• Reading 20–50 starting tasks as a sufficient production sample for every decision.

What to remember

Define the population and success criteria first. Combine real samples with independently validated expert, edge, adversarial, and synthetic cases. Report counts and results by slice, preserve provenance, and distinguish expected-traffic performance from challenge coverage.

Knowledge check

References

• Anthropic, Define success criteria and build evaluations
• Anthropic, Demystifying evals for AI agents