Require verification to earn its delay

Learning objective

Quantify trade-offs on a shared evaluation set — including whether extra verification calls earn their latency — and justify the chosen configuration

Core idea

For an architect defending a Claude deployment, an extra check is an investment with a measurable return. It can catch an expensive error, repeat the original mistake, reject a correct answer, or add delay without changing the outcome. Compare the complete workflow with and without that call on identical cases, then judge the gain against the task’s fixed requirements.

Anthropic’s evaluation guidance treats success as multidimensional: task fidelity, latency, price, consistency, and other relevant dimensions. A configuration is defensible when its measured result meets the required dimensions and its remaining weaknesses are explicit. “More verification is safer” is not a substitute for that comparison.

How it works

Fix the cases and the acceptance rule

Build a task-specific evaluation set with expected outcomes and deliberately included edge cases. Separate development cases used to tune prompts from held-out cases used to assess the final candidates. Have the domain owner define the costly errors and the acceptance rubric before comparing configurations.

Choose the grader from the answer type. Anthropic documents exact-match grading for clear categorical answers, code-based grading for computable criteria, and LLM-based grading for judgments that need a rubric. A verification model is part of the candidate workflow; it should not also be the sole independent judge of that workflow’s success. Use known expected answers or expert-reviewed labels where the business decision permits them.

Version the candidate configuration: model, supported output_config.effort, prompt content, retrieval settings, tool sequence, and verification placement. The same question against a changed index is not an identical evidence condition. Keep the document snapshot and grading rule fixed for a controlled comparison, then test changing-data behavior separately.

Record the answer actually accepted by the workflow. If a check forces a correction, include that correction’s result and time. If the task times out or the check rejects the answer without a usable recovery, record a failed or incomplete task according to the predeclared rule. Dropping these cases rewards configurations that avoid difficult work.

Measure latency at the same boundary

Start timing at the user action and stop at the accepted outcome. Measure first visible text separately where streaming is allowed; see 3.3.2. A configuration that displays a draft before verification might improve that first-text measurement while leaving accepted completion slower.

Report p50 and p95 completion, not only mean model time. Keep workload, concurrency, cache conditions, and output requirements comparable. Use usage.cache_read_input_tokens to identify warm reads instead of assuming every run benefits from the configured cache. Record cold runs too if production includes them.

Repeated runs help reveal output and latency variability. Retain per-case records so the team can inspect disagreements and compare the same case across candidates. A small gain over a small sample should lead to more assessment, not a claim of established superiority. Public model benchmarks cannot settle an application-specific verification decision.

Isolate the marginal verification effect

Run a paired ablation: keep the base configuration fixed, and change only whether the proposed verification call executes. Print the verifier’s actual scope and inputs. A tool named verify_fact that receives only the answer and no authoritative record cannot establish whether that record supports the answer.

For an application-defined verifier on the Messages API, the proposed call appears as a tool_use block. The dispatcher returns the outcome as tool_result correlated with tool_use_id. If the verifier is another model request, account for that request and the subsequent continuation or correction. A feature name does not tell you how many round trips the implementation adds. When the verifier is another model request, the path with the proposed verification call on identical cases adds that request and the subsequent continuation or correction before the accepted outcome.

Count at least three kinds of change: incorrect answers corrected, correct answers rejected or spoiled, and errors that both paths miss. Report the net task-level improvement as well as the verifier’s own detection rate. A check with excellent detection but frequent unnecessary rejections can lower usable completion.

Measure the added time for the whole accepted result. If verification is required before a consequential action, moving it after text display does not remove it from the action’s critical path. If it is only a later audit, it cannot be credited as preventing the action’s errors. The experiment must evaluate the decision the product actually makes.

Filter constraints before ranking value

Remove candidates that fail a nonnegotiable accuracy or completion requirement. Among the survivors, compare the remaining dimensions and document the reason for the choice. A configuration dominated by another candidate with equal or better accuracy and lower latency needs a separate reason to survive, such as an independently stated operating constraint.

If no candidate meets the constraints, report that result. Consider a changed workflow, a narrower scope, or an explicitly renegotiated target. Do not silently average safety-critical failures into a general quality score or claim that a pleasant streamed experience compensates for missed accepted completion.

Inspect segments as well as the aggregate. Anthropic’s evaluation guidance explicitly includes edge-case handling. A verifier that helps ordinary cases but misses the expensive exception is a poor investment for that exception. A segment-specific verification route requires its own measured routing and completion behavior; see 2.1.2 for tier-routing design.

Key distinctions and trade-offs

Verification accuracy and workflow accuracy are different quantities. The check can identify a defect without producing a correct replacement within the deadline. Conversely, a narrowly scoped check against a definitive value can improve the workflow more than a broad second opinion.

More information, more effort, and more checking can each raise quality on some cases while raising latency on all cases. Prefer the intervention whose measured gain addresses the business error. A held-out result supports the choice under its measured conditions, not a guarantee that it holds after model, prompt, index, or traffic changes.

Architecture / decision scenario

A financial-services fee-explanation assistant requires at least 97% correct accepted outcomes and p95 completion within six seconds. The fee-policy owner labels 1,000 held-out cases, including exception schedules. The following results are illustrative rather than Claude performance claims:

The narrow verifier corrects 32 incorrect cases but rejects or spoils seven previously correct ones, yielding 25 additional correct accepted outcomes. Its 1.6-second increase in measured p95 earns a 2.5-percentage-point gain. The broad review adds only three accepted successes beyond the narrow verifier while exceeding the deadline.

The tempting proposal is to ship the broad review because it has the highest accuracy. The architect chooses the narrow verifier because it is the only measured candidate that meets both constraints. The fee-policy owner reviews the remaining exception failures; the platform owner records cold-cache and peak-load results before release.

The decision record includes the cases and rubric version, exact model and effort, verifier inputs, measured options, the rejected alternative, residual failures, and reopening triggers. If exception-case accuracy falls below its agreed floor or p95 crosses six seconds under the production workload, the owners reopen the configuration decision. The additional review has a defensible place only if a changed workflow or requirement makes its delay acceptable.

Common exam traps

• Comparing configurations on different cases, source snapshots, or cache conditions.
• Using the verifier’s approval rate as the application’s accuracy score.
• Counting corrected errors while ignoring correct answers rejected by the check.
• Timing a streamed draft instead of the accepted result that requires verification.
• Choosing the most accurate candidate after it fails the mandatory deadline.
• Reporting only aggregate improvement when costly exception cases regress.

What to remember

Use a shared held-out set, a fixed rubric, and timing that ends at accepted completion. Isolate the verification call’s net accuracy gain and full added delay. Filter mandatory constraints first, then defend the survivor with measured options, residual risk, ownership, and reopening conditions.

Knowledge check

References

• Anthropic, Define success criteria and build evaluations
• Anthropic, Reducing latency
• Anthropic, Streaming Messages
• Anthropic, Prompt caching
• Anthropic, Parallel tool use