Pause noisy categories and verify their repair

Learning objective

Measure false positives by issue category and temporarily disable noisy categories while their detection rules are improved.

Core idea

For an architect operating a Claude CI reviewer, aggregate quality can hide a category that wastes developer attention. A reviewer can identify useful security defects while flooding the same pull requests with invalid comment findings. The exam guide connects high false-positive categories with reduced trust in accurate categories. Its rationale is that developers judge the reviewer as a whole: an accurate security finding beside many invalid comment findings receives the same skim. Pausing the noisy category lets useful findings regain attention and supports restoring developer trust.

Measure each category separately. Temporarily remove an unreliable category from developer-facing review while you repair its inclusion and exclusion rules. Keep useful categories active, and evaluate the disabled category against known cases before restoring it.

In managed Code Review, pause a category through REVIEW.md Skip rules, then remove that rule to restore it. This is the research-preview Team/Enterprise service introduced in 4.1.1, which also names its Zero Data Retention and HIPAA exclusions. In a custom reviewer, change the enabled categories in the prompt and enforce that selection before publication. See 4.1.2 for the risk of substituting confidence filtering.

How it works

Label outcomes against the review contract

A true positive is a reported finding that satisfies the category’s defect criteria. A false positive is a reported finding that does not. A false negative is a qualifying defect present in the evaluated code but missing from the report. Use the same criteria for adjudication that you expect Claude to apply.

Developer actions are useful feedback, but “fixed” and “dismissed” are not equivalent to true and false. A valid defect can be deferred. A developer can accept an optional improvement that the review contract excludes. Confirm whether the finding actually meets the criterion before counting it as correct or incorrect.

Preserve the category and enough finding evidence to adjudicate it: location, claimed violation, triggering condition, and effect. If multiple reports describe one defect, reconcile them before computing defect-level counts. Otherwise, repeated reports of the same correct issue can inflate apparent quality without increasing useful coverage.

State the denominator

Use precise metric names, especially when the team casually calls every noise measure a “false-positive rate.” For each category, calculate:

precision = true positives / (true positives + false positives)
false-discovery proportion = false positives / all reported findings
recall = true positives / (true positives + false negatives)

The first two require adjudicated reports. Recall requires known defects, including those the review missed. The statistical false-positive rate instead requires a denominator that includes true negatives, so it cannot be derived solely from reported findings.

For open-ended code review, precision and the false-discovery proportion often provide the clearest noise view because the universe of nondefects is not naturally enumerated. Keep the counts with the percentages. A category with zero reports has no observed report precision; do not present its silence as 100% precision or evidence of adequate recall.

Choose the category-specific action

The following figures are invented for illustration. The team adjudicated each report against its category’s rules:

The combined precision is 38 out of 60, about 63%. That aggregate obscures the 20% comment precision and does not imply that security findings should be disabled too. The percentages are not Anthropic thresholds; your acceptance policy determines which category can remain developer-facing.

In managed Code Review, add “Do not report comment contradictions” under REVIEW.md’s skip instructions while you repair that category. Its Nit volume pattern also supports a cap such as at most five Nit comments per review; a volume cap limits noise but does not repair category criteria. In a custom claude -p or Agent SDK harness, remove comment contradictions from the prompt’s enabled list and reject disabled categories in the publisher before posting findings.

Repair offline and restore with evidence

Replace the noisy rule with a precise test, such as the comment contradiction criterion from 4.1.1. Run the revised category on past changes with adjudicated answers, including harmless comments, real contradictions, and cases where helper behavior resolves an apparent mismatch.

Anthropic’s evaluation guidance recommends task-specific cases, edge cases, and measurable success criteria. Apply that guidance to the repair: choose category precision and recall targets before comparing the prompt variants. These are team acceptance targets, not product limits. Score fixed labels with code where suitable. For complex code-dependent judgments, validate an LLM grader against a human-labeled sample before scaling it; Anthropic recommends avoiding slow, expensive human grading where possible.

Keep the category disabled for publication while its revised prompt runs in an offline or unpublished evaluation. Re-enable it only when it meets the stated criteria on cases beyond those used to tune the rule. Check that improved precision did not come from suppressing every real contradiction. Preserve the other categories’ evaluation results so the repair does not accidentally change their reporting boundary.

A noisy category leaves developer-facing review, runs only in evaluation while its rule is repaired, and returns only when it meets the stated criteria on cases beyond those used to tune the rule.

Key distinctions and trade-offs

Category pause versus reviewer shutdown: A pause removes one unreliable output stream. A shutdown also removes accurate categories that still provide value.

Noise versus coverage: Fewer false reports improve the experience, but zero reports can conceal missing real defects. Evaluate recall alongside precision before restoration.

Feedback versus adjudication: Developer behavior identifies candidates for analysis. The review contract determines whether a candidate belongs in the metric’s true-positive or false-positive count.

A pause deliberately sacrifices developer-facing coverage for that category while its rules are repaired. Keep the scope explicit: comments omitted during the pause were not verified as correct. The review can remain useful because its reliable categories continue to operate, but a partial review must not be interpreted as complete coverage.

Architecture / decision scenario

A Claude Code CI reviewer covers security, behavioral bugs, and comment contradictions. Developers begin ignoring its comments. Category-level adjudication produces the illustrative table: security is useful, behavioral bugs remain acceptable under team policy, and most comment findings concern harmless TODOs or terse wording.

The architect adds a category skip rule to REVIEW.md for managed Code Review, or disables the category in both the prompt and publisher for the custom CI reviewer. Security and behavioral bug findings continue to reach pull requests. The paused category runs only in evaluation, where its prompt now requires a behavioral claim and contradictory implementation evidence.

The team tests real mismatches alongside harmless comments and delegated retry implementations. It measures category precision and recall against known answers, rather than counting how many reports disappeared. Once the revised rule meets the predetermined acceptance targets on held-out cases, the architect restores the category and continues recording adjudicated outcomes by category.

This addresses the source of noise while preserving useful findings. A global “be conservative” instruction or a complete reviewer shutdown would not provide the same targeted repair.

Common exam traps

• Using aggregate precision to decide which category should be paused.
• Counting every dismissed finding as a false positive without adjudication.
• Rejecting an exam option about category-level “false positive rates” merely because it uses informal metric terminology.
• Disabling the entire reviewer when only one category is unreliable.
• Restoring a category because report volume fell without checking missed defects.

Where the exam guide differs

What to remember

Measure adjudicated findings by category and state each denominator. Pause a noisy category in the review instructions and publication control, repair its criteria, and test precision and recall before restoring it. Preserve accurate categories so the repair does not remove the review’s useful coverage.

Knowledge check

References

• Anthropic, Define success criteria and build evaluations
• Anthropic, Claude Certified Architect – Foundations Exam Guide
• Anthropic, Code Review