AnthropicClaude Certified Architect – Professional2026-104.1 Define evaluation metrics (accuracy, latency, cost, safety, security)
Building Evals
Completed by opening the official lab
Why it matters
A metric is only as useful as the grader that produces it. Writing the cases is a one-time cost, but grading is paid on every run, so the grading method decides what you can afford to measure.
What to practise
Build each eval from an input, the model’s output, a golden answer and a score. Grade a leg-counting task by exact match, then grade open-ended answers first by hand and then with a grader prompt that has Claude reason against a rubric before it outputs correct or incorrect. Watch the case whose golden answer requires Claude to decline to send an email: a correct answer can be an abstention.
How it fits
The rubric that accepts a justified refusal is the abstention measure of 4.1.2, and the grading method decides which of that scorecard’s measures you can afford to rerun on every change. Compare the three graders with the split between deterministic checks, model judgments and expert adjudication in 4.2.2.