Verify findings before trusting confidence scores

Learning objective

Run finding-verification passes and calibrate reported confidence on labeled outcomes before routing human review by thresholds.

Core idea

A review finding is a candidate claim about code behavior. Before routing it, run a verification pass that attempts to establish or refute that behavior from the code, requirements, and relevant execution evidence. Ask the verifier to report both its evidence and confidence that the defect is real.

Confidence becomes useful for routing only after comparison with labeled outcomes. A model assigning a high score is not itself evidence that findings with that score are usually correct. You need measured outcomes from representative findings to decide which score bands justify automated reporting and which need a person.

Anthropic’s managed Code Review documents verification against actual behavior, followed by deduplication and severity ranking. The confidence calibration and threshold routing here describe a custom pipeline implementing task statement 4.6, rather than a documented configuration field of that managed service.

How it works

Ask verification to challenge the candidate

Give the verification pass the candidate’s location, claimed triggering condition, expected behavior, and alleged consequence. Supply the actual implementation and any caller, guard, or requirement needed to check those claims. The local and integration passes in 4.6.2 produce these candidates; verification determines which claims their evidence supports.

Anthropic’s Claude Code best practices describe a second-opinion verification approach in which a fresh model tries to refute the result. Apply that posture to review findings. “Explain why this bug is real” encourages confirmation. “Check whether the claimed input reaches this operation; look for guards or contracts that invalidate the claim” gives the verifier a concrete falsification task.

For example, a candidate alleges division by zero. A verifier should inspect whether an earlier branch returns when the count is zero, whether the caller guarantees a positive count, and whether that guarantee is enforced. If an executable reproduction is available, use it as evidence. A confidently repeated description of the division does not settle its reachability.

In managed Code Review, root-level REVIEW.md can set a verification bar such as requiring file:line evidence for behavior claims. That file reaches finding and verification agents. In your custom pipeline, explicitly provide the same rubric in the verification request; do not assume your own claude -p invocation automatically implements the managed service’s review pipeline.

Keep the result inspectable

For a custom structured finding contract, define application-owned fields such as finding_id, verification_status, evidence, and confidence. Here confidence means the verifier’s estimate that the claimed defect is real under the stated conditions, not the severity of its consequence. These are fields you choose, not standard Claude response fields.

Use --json-schema in Claude Code print mode to request validated structured output for that contract; see 3.6.2 for consuming the result. Schema validation ensures the output shape, not whether the cited code actually establishes a defect. The verifier can return a well-formed but mistaken claim, so outcome evaluation remains necessary.

Distinguish confirmed, refuted, and unresolved candidates in the contract. A supplied guard can refute the zero-count claim. Missing deployment configuration can leave a concurrency claim unresolved. Low confidence alone is neither proof of a false positive nor permission to claim verification succeeded.

Compare confidence with labeled outcomes

Build an evaluation set of past candidates whose defect status reviewers have established from code and behavior. Include true defects, false alarms, and difficult cases representative of the pipeline’s task distribution. Keep unresolved cases distinguishable rather than assigning them a convenient false label.

Anthropic’s evaluation guidance recommends task-specific tests, edge cases, measurable criteria, and held-out evaluation. It distinguishes code-based, human, and LLM-based grading, and says to test LLM grader reliability before scaling. Use executable checks where they settle a claim; use qualified human judgments where evaluating the claimed behavior requires contextual interpretation.

Group verifier outputs by reported confidence and compare each group’s scores with the observed fraction of real defects. If 100 labeled candidates receive scores around 0.90 but only 62 are real defects, the observed precision is 62%, not the roughly 90% the scores suggest. Those numbers are an illustrative calibration failure, not an Anthropic benchmark or recommended threshold.

Use labeled development outcomes to select or adjust routing bands, then assess them on held-out outcomes before relying on them. Inspect categories separately: boundary errors with direct reproductions may behave differently from concurrency claims with incomplete environmental evidence. A strong pooled result can hide an unreliable category. See 4.1.4 for repairing noisy categories.

Route by measured bands and consequence

The following table describes a possible application policy after calibration:

Set the thresholds from measured outcomes and acceptable review noise, not from a universal confidence cutoff. Keep severity separate: a low-confidence claim of cross-account access can deserve urgent human attention, while a high-confidence minor defect need not interrupt an on-call engineer. Routing decides who checks the finding and how it is presented.

See 4.1.2 for confidence filtering versus defect criteria: Claude Code’s local /code-review low reports the findings it is most confident in, while medium through max broaden coverage. That coverage control does not calibrate the confidence scores in your custom verification contract. Keep uncertain candidates available for human triage and outcome labeling; remove refuted claims on evidence rather than treating a low score as disproof.

Labeled outcomes set the routing thresholds, and human review keeps its outcomes for subsequent evaluation.

Key distinctions and trade-offs

Detection versus verification: Detection proposes a defect. Verification checks the trigger and consequence against implementation evidence, including evidence that contradicts the claim.

Confidence versus calibration: Confidence is a reported estimate. Calibration is its measured relationship to labeled correctness. A numeric schema field provides the first, not the second.

Confidence versus severity: Confidence estimates whether the defect exists. Severity describes its effect if it does. Route serious uncertainty to people rather than treating uncertainty as low impact.

Managed service versus custom routing: Managed Code Review ranks findings by severity. Your confidence bands and human queues are application choices; severity labels do not establish calibrated probabilities.

Verification and labeling add cost. Their value is reducing unsupported reports while preserving uncertain candidates for appropriate attention. Increasing the automatic-reporting band can reduce noise but increase human workload; evaluate both effects on held-out findings.

Architecture / decision scenario

A Claude Code CI pipeline produces three candidates: a day-30 refund rejection, division by zero, and a possible cross-account lookup. A separate verification pass reads each claim and its relevant code. It confirms the boundary mismatch against the ticket, refutes the division claim because an enforced zero-count guard returns first, and leaves the account claim unresolved because the authorization configuration is absent.

The verifier reports confidence and evidence through the pipeline’s structured contract. Before automatic reporting, the team evaluates those scores on labeled historical outcomes. It discovers that high-scored authorization claims are less reliable than high-scored boundary claims. It selects category-aware routing thresholds and checks their behavior on held-out cases.

The boundary finding enters the permitted automated-reporting route. The refuted division claim is retained with its disproof rather than posted. The unresolved account claim goes to a human with the missing authorization question clearly identified; its possible consequence warrants attention despite uncertainty. The team keeps outcomes from that review for subsequent evaluation instead of treating the model’s number as a final verdict.

Common exam traps

• Calling a second model’s agreement verification without checking behavior.
• Treating a confidence field as a calibrated probability merely because it is numeric.
• Choosing a universal threshold without labeled outcomes.
• Evaluating thresholds only on examples used to select them.
• Confusing certainty with severity or using severity as a probability.
• Deleting uncertain candidates instead of routing them to human attention.
• Assuming schema-valid output proves that a reported defect exists.

What to remember

Verify the behavior claim first, then measure how reported confidence relates to labeled correctness. Use held-out results to justify routing bands, preserve evidence and unresolved questions, and keep consequence separate from certainty. A human-review threshold is an evaluated application policy, not a substitute for verification.

Knowledge check

References

• Anthropic, Best practices for Claude Code
• Anthropic, Code Review
• Anthropic, CLI reference
• Anthropic, Define success criteria and build evaluations