AnthropicClaude Certified Architect – Professional2026-104.2 Design evaluation datasets and test frameworks using mixed methodologies
Locate failures across the evaluation layers
Completed by the knowledge check
Learning objective
Layer tests: unit checks, component evaluations, end-to-end and agent trajectory evaluations, and regression suites
Core idea
Behavioral tests and output validators establish local correctness. A production evaluation architecture adds controlled component experiments, complete-workflow outcomes, trajectory inspection, and regression coverage across changes. Each layer answers a different question; passing one cannot establish that the assembled agent works.
Anthropic distinguishes the outcome, the final state after a trial, from the transcript, the record of what happened during that trial. Its evaluation harness runs tasks, records steps, grades results, and aggregates scores. Design that harness to expose both integration defects and unacceptable behavior hidden behind a successful final response.
How it works
Separate checks by the system they exercise
Use the following test layers to isolate different sources of failure:
| Layer | Controlled input | What it establishes | What it leaves open |
|---|---|---|---|
| Unit checks | Explicit values passed to deterministic logic | Normalization, calculations, and policy predicates behave as specified | Whether Claude supplies correct inputs |
| Component evaluations | Fixed inputs to retrieval, extraction, planning, or grading | That component’s behavior under its own success criteria | Whether upstream outputs satisfy its assumptions |
| End-to-end evaluations | Complete task and initialized environment | The assembled system produces the required outcome | Whether it used an unacceptable intermediate action |
| Trajectory evaluations | Recorded calls, results, and actions from a trial | Required behavioral constraints hold throughout execution | Whether a polished final message matches real state |
| Regression suites | Selected tasks rerun against a candidate change | Previously established behavior remains reliable | Whether untested capabilities have improved |
Regression is a purpose that can contain every other layer. It is not simply a fifth scope of execution. A deterministic unit check and a complete agent task can both protect a fixed defect. Binary fail-to-pass tests check that a targeted failure becomes a success; pass-to-pass tests check that previously passing behavior stays correct.
Make component boundaries observable
Evaluate an extractor on fixed source material, a retriever on fixed queries and a known corpus, and a grader on fixed candidate outputs with adjudicated labels. That lets you change one component without the others changing the test input.
Retain the integration path as well. A retriever can pass its component cases while the assembled application sends it the wrong query. An extractor evaluated on clean text can fail when document processing drops a page. These are reasons to add an end-to-end case, rather than to assume all passing component scores imply system success.
Keep inputs faithful to the deployed interface. Anthropic recommends an evaluation environment in which the agent functions roughly as it does in production. A component test using a simplified substitute should state that limitation. It can diagnose local behavior without certifying integration with the actual dependency.
Verify outcome and trajectory independently
For a task that changes state, inspect the resulting state in the test environment. Do not award success because the final response says the operation completed. Anthropic uses the example of a booking that must exist in the database, rather than only in the agent’s claim.
For a Messages API agent loop, the evaluation transcript includes the messages record of calls and returned responses. Capture tool calls and results as well as the final answer. Tool calls verification checks the tools used and their parameters; transcript analysis checks turns taken and token usage. Anthropic also recommends collecting runtime for individual tool calls and complete tasks, total tool-call count, token consumption, and tool errors. Use those observations to check meaningful constraints, such as whether the required approval preceded a write or whether any operation targeted an unauthorized record.
The harness grades the outcome from the resulting state in the test environment and the trajectory from the recorded tool calls and results, not from the final response.
Optionally specify which tools a task should use to check whether the agent understands their purpose. Avoid asserting one exact tool sequence unless order is itself a requirement. Anthropic warns that agents can find valid paths designers did not anticipate. A task might allow two search strategies while still requiring authorization before mutation. Grade that prerequisite and the final state, rather than rejecting every alternative search path.
A trajectory grader measures behavior; it does not enforce it during a run. Keep authorization in the trusted execution controls described in 3.2.2 and test the access trails from 3.2.5 against the observed outcome. Do not make the test judge the only protection against an unsafe write.
Control noise before interpreting regressions
Start every trial from a clean environment. Anthropic describes leftover files, caches, resource exhaustion, and access to previous trials’ git history as sources of misleading results. Reset the fixtures and isolate writable state so one attempt cannot teach the next how to pass.
Run multiple trials for model-driven tasks when assessing stability; see 4.3.3 for interpreting repeated-trial pass rates. A changed pass rate can reflect output variation, a harness defect, or a real system regression. Retain trial-level outcomes and inspect failing transcripts before attributing the change. A missing tool service is an environment failure that still makes the run incomplete; it is not proof that a prompt regressed.
Run automated evaluations in continuous integration and delivery (CI/CD) on agent changes and model upgrades, as Anthropic recommends. Use a fast subset during routine changes and a broader run before release when cost requires staged coverage. Name which layers and case families each subset includes. A skipped or failed-to-start suite is missing evidence, not a passing score. Dataset revision and incident-case maintenance belong to 4.2.4.
Key distinctions and trade-offs
Component isolation versus integration realism: Fixed inputs help locate a defect and reduce noise. Complete-workflow tests expose interactions that controlled components omit. Both are needed for useful diagnosis.
Outcome success versus acceptable execution: Correct final state establishes task completion. Trajectory assertions establish required behavior during execution. A successful outcome does not excuse an unauthorized intermediate action.
Capability versus regression: Anthropic describes capability evaluations as challenging tasks with room for improvement and regression evaluations as protection for previously working behavior. A difficult capability case should not automatically become a mandatory release blocker before the team has committed to supporting it.
Architecture / decision scenario
A technology company runs an incident assistant that investigates alerts and drafts a remediation change. Its service owner requires human approval before applying that change. The team has a 15-minute pull-request evaluation budget and a longer scheduled pre-release window.
An engineer proposes checking only the assistant’s final answer because component tests already pass. The architect rejects that shortcut after one trial reports a repaired service while the test environment still contains the failing configuration. Another trial reaches the correct configuration but applies it before approval.
The suite now separates deterministic approval predicates, retrieval on fixed incident records, planning on fixed tool results, and complete incident tasks. End-to-end graders inspect configuration state. Trajectory graders inspect whether approval preceded mutation, while allowing different valid investigation paths.
The pull-request subset includes both reproduced defects, representative component cases, and unit checks. The broader release run repeats full tasks in isolated environments and retains transcripts for adjudication. The service owner blocks a candidate whose final-state score improves while approval-order checks fail. Paying for both observations makes the production constraint testable; final-answer grading alone misses both kinds of failure.
Common exam traps
- Treating passing components as proof that their integration is correct.
- Checking an agent’s success claim instead of the resulting business state.
- Confusing trajectory evaluation with runtime permission enforcement.
- Requiring an arbitrary tool-call sequence that rejects valid solutions.
- Sharing state across trials and interpreting the resulting score as independent performance.
- Calling a skipped broader suite a successful release evaluation.
What to remember
Use deterministic checks for local rules, controlled component cases for diagnosis, and end-to-end trials for assembled outcomes. Inspect trajectories for meaningful behavioral constraints. Regression suites combine these scopes and require isolated, observable runs before their scores can support a release.
Knowledge check
A support agent processes refunds in a test environment, and completion means the refund exists in the payments ledger. In one trial the agent’s final message says “Refund issued”, and its transcript shows a call to the refund tool. How should the end-to-end grader decide success?
An incident assistant may investigate an alert with either log search or metrics queries, but it must obtain human approval before applying a configuration change. The current trajectory grader requires one exact sequence of tool calls and rejects several runs that reached the right configuration with approval first. Which change preserves the required checks with the least maintenance as new valid investigation paths appear?
Because a trajectory grader reliably flags any configuration write made before human approval, an engineer proposes removing the approval check from the production tool executor to simplify it. What should the architect decide?
A policy assistant’s retriever passes its component evaluation on fixed queries and a known corpus, yet the assembled system’s answers often cite the wrong policy. Transcripts show the application sending the retriever the user’s greeting instead of the question. What should the suite add?
Before a model upgrade, the release run combines the fast pull-request subset with a broader suite of full agent tasks. The pull-request subset passes, but the broader suite fails to start because a tool service in the test environment is down. What should the release report show for the broader suite?
References
- Anthropic, Define success criteria and build evaluations
- Anthropic, Demystifying evals for AI agents
- Anthropic, Writing effective tools for agents — with agents