Separate confidence filtering from defect detection

Learning objective

Explain why instructions such as “be conservative” or “high-confidence only” do not replace precise detection criteria.

Core idea

When you design a Claude review prompt, distinguish the definition of a finding from the threshold for publishing it. “High-confidence only” addresses certainty about a judgment. It does not say whether that judgment belongs in your review. Claude can be certain that a comment is terse while your policy excludes terse comments entirely.

A qualitative threshold can also suppress genuine bugs. Anthropic’s Claude Opus 4.8 prompting guide describes review harnesses in which the model investigates thoroughly but omits findings below the stated importance bar. The page says precision typically rises while measured recall can fall. Fewer reports therefore do not prove that the detection rule improved.

The correction is to specify the eligible defect categories and evidence requirements, as in 4.1.1. Then choose where confidence and severity influence publication. Keep those choices separate enough to measure their effects.

How it works

Identify which decision the instruction controls

A review contains at least two decisions: whether the code satisfies a defect criterion and whether the resulting candidate should reach a developer. A prompt that only says “be conservative” leaves both decisions to interpretation. One run may exclude minor style suggestions; another may exclude a real edge-case crash.

A precise criterion answers an observable question. For example, does a comment claim retry behavior that the execution path does not implement? Confidence answers how strongly the available evidence supports that conclusion. Severity answers how consequential the mismatch is. These are different judgments, even when a single response reports all three.

Consider a comment describing a single attempt as “fast.” Claude might be highly confident that the description is vague. That confidence does not make vagueness reportable under a contradiction-only rule. Conversely, an unfamiliar callback may contain a real authorization error whose exploit path requires further inspection. Filtering solely by confidence can retain the irrelevant observation and discard the consequential candidate.

Choose a documented reporting pattern

The Opus 4.8 guide offers two approaches for code-review harnesses. Treat its model-specific behavior as guidance for that model and evaluate other models separately; it is not a universal guarantee about every Claude configuration.

For a staged harness, make the discovery step favor coverage. The guide recommends asking for issues including uncertain or low-severity candidates, with confidence and estimated severity available for downstream filtering. If a separate verification, deduplication, or ranking step exists, tell the discovery step that those publication decisions belong there.

The stages still need an explicit defect definition. “Report every issue” should mean every candidate in the eligible categories, not every possible stylistic improvement. A downstream verifier checks the candidate against the inclusion and exclusion rules before the harness publishes it.

For a single-pass harness, the guide recommends a concrete reporting bar: bugs that could cause incorrect behavior, a test failure, or a misleading result, excluding pure style or naming preferences. Adapt that boundary to your task and state it in the prompt. Do not replace it with “important issues only.”

A decision clause for a staged review can read:

Discovery: identify candidates in the enabled defect categories.
Include uncertain candidates with their unresolved evidence questions.
Verification: check each candidate against the category's criteria.
Publish only verified qualifying findings; keep rejected candidates
available for evaluation rather than presenting them as defects.

Managed Code Review implements a staged pattern: agents analyze the diff, verification checks candidates against actual behavior, and results are deduplicated and ranked by severity. See 4.1.1 for its research-preview availability and REVIEW.md rules. The local /code-review low reports its most confident findings to reduce false positives; medium through max broaden coverage. That documented confidence control changes the reporting trade-off without defining which categories qualify.

Measure what the threshold removes

Precision is the share of reported findings that are real qualifying defects. Recall is the share of known qualifying defects that the review reports. A shorter list can improve precision while lowering recall, particularly if the prompt silently suppresses lower-severity bugs.

Suppose an illustrative test contains ten known defects. One prompt reports eight true findings and eight false findings: 50% precision and 80% recall. Adding a qualitative threshold yields four true and one false: 80% precision but 40% recall. The improvement in precision is real; the missing half of the previously reported defects is also real. Neither number alone describes whether the workflow satisfies the team’s needs.

Anthropic recommends iterating on review prompts against evaluations or test cases to validate recall or F1 gains. Compare candidate discovery and published results on the same labeled cases. If discovery contains the real bug but publication removes it, changing the code-reading instructions may not address the failure. Inspect the reporting criterion and filtering stage instead.

Key distinctions and trade-offs

Confidence versus eligibility: Confidence expresses certainty. Eligibility comes from the team’s reportable-defect definition. A confident observation can be ineligible.

Severity versus evidence: A potential effect can be serious while the evidence remains incomplete. Severity should not manufacture certainty, and uncertainty should identify what needs verification.

Coverage versus publication: Broad candidate discovery can preserve recall, but publishing all unverified candidates exposes developers to noise. A staged harness pays for another check in exchange for control over that boundary.

A single pass is simpler, but its reporting rule must be concrete. A staged harness makes rejected candidates observable, but still requires a verifier that applies the intended criteria. Neither architecture turns an ambiguous word such as “conservative” into a shared definition. A staged harness adds a verification check between discovery and publication and keeps rejected candidates observable.

Architecture / decision scenario

A Claude Code CI review reports harmless TODOs and misses a subtle refund calculation defect after the team adds “only report high-confidence, high-severity issues.” The architect first checks whether the refund candidate was discovered and later suppressed, or never identified.

The team defines behavioral defects using realistic inputs and wrong outcomes, and comments using the contradiction rule from 4.1.1. In a staged variant, discovery preserves the uncertain refund candidate with its evidence question; verification inspects the rounding path and either confirms or rejects it. The TODO fails eligibility regardless of confidence.

In a single-pass variant, the prompt explicitly admits wrong results, test failures, and misleading behavior while excluding style and naming preferences. The team compares precision and recall on the same known cases before selecting the variant. It does not accept fewer comments as sufficient evidence of a better reviewer.

Common exam traps

• Equating a confident statement with a qualifying defect.
• Claiming confidence filtering can never improve measured precision.
• Treating higher precision as proof that recall remained acceptable.
• Publishing every discovery candidate because a staged prompt favors coverage.
• Increasing severity requirements to repair an undefined category boundary.

Where the exam guide differs

What to remember

Define the defect before filtering the finding. Confidence, severity, and eligibility answer different questions. Concrete single-pass criteria or a separate verification stage make the reporting boundary inspectable; evaluate suppressed real bugs as well as removed noise.

Knowledge check

References

• Anthropic, Prompting Claude Opus 4.8
• Anthropic, Prompting best practices
• Anthropic, Claude Certified Architect – Foundations Exam Guide
• Anthropic, Code Review