Contrast acceptable code with true defects

Learning objective

Contrast acceptable code patterns with true defects in examples to reduce false positives without masking novel errors.

Core idea

A reviewer that flags every suspicious-looking pattern creates false positives. A reviewer that exempts every familiar-looking pattern misses defects. Use contrasting examples to show the evidence that makes a pattern acceptable in one case and harmful in another.

Anthropic recommends examples that are relevant, diverse, and structured, with enough variation to avoid unintended patterns. Apply that guidance to code review by pairing similar-looking inputs with different justified verdicts. The contrast should reveal a behavioral distinction, such as whether a suppressed failure invalidates the result the caller relies on.

The examples guide Claude’s judgment; they do not prove the reviewed code correct. Anthropic’s hallucination guidance also recommends grounding claims in source evidence and admitting uncertainty. For a review, that means demonstrating the difference between a supported defect, an acceptable operation, and a case whose surrounding contract is unavailable.

How it works

Choose a pair whose difference changes behavior

Suppose a review prompt repeatedly flags error handling that deliberately tolerates an already-absent temporary artifact. Adding more examples of harmful exception handling may reinforce the association between a catch block and a defect. Supply the acceptable counterexample alongside the harmful one.

The following are invented application cases. Their contracts are part of the supplied review input, so the desired verdict does not depend on guessing a library’s behavior:

The third case prevents the first pair from teaching “scratch cleanup is the only acceptable exception.” It changes the domain while retaining the deciding distinction: an explicitly permitted outcome remains truthful about the operation’s state.

Avoid calling a broad catch acceptable merely because it contains a comment. A comment explaining intent is evidence of intent, not proof that all caught failures are harmless. The acceptable example must show the limited condition and the postcondition that make the outcome valid.

Write the boundary into the example’s explanation

State why the acceptable case passes and why the defect fails. “Ignore cleanup issues” is too broad: a permissions failure might leave an artifact present. “Do not report AlreadyAbsent when absence is the required postcondition” identifies the evidence that justifies silence.

Likewise, “payments are important” is weaker than “the catch path returns paid: true without evidence that charging succeeded.” The second explanation transfers to inventory reservations, credential checks, and conversion results without requiring examples from every domain.

Use a short justification tied to visible input. The desired explanation names the observable consequence and the contract it contradicts. See 4.2.1 for selecting a compact example set and separating it from the live case with <examples> and <example> tags.

Keep the output convention explicit. In this workflow, a known acceptable case produces an empty findings collection, while a supported defect produces a finding. That meaning belongs to the application, so state it in the instructions and demonstrate it.

Preserve a path for unfamiliar defects

A contrast pair is not an allowlist. Include an instruction to judge unseen inputs against the same contract-and-consequence criterion. For example, a cached conversion routine that returns zero after a failed rate lookup might contain no catch block at all. If zero is presented as a verified converted amount, the same hidden-failure criterion applies.

The review input must include enough context to justify that conclusion. If zero is instead a documented unavailable-value marker that every caller checks, the apparent defect may disappear. Demonstrate an unresolved case when the caller’s interpretation is missing: identify the missing contract instead of inventing a failure path.

Anthropic recommends explicitly permitting uncertainty and restricting claims to supplied information. Implement that guidance in the review instructions by asking for supporting code or contract evidence and a stated evidence gap when the verdict cannot be established. See 4.2.3 for turning a supported finding into an actionable report.

Key distinctions and trade-offs

Negative examples versus weaker review criteria: Showing that one case is acceptable clarifies the boundary. Broadly excluding all catch blocks or all cleanup files removes a class of inputs from review and can hide genuine errors.

Intent versus effect: A developer’s explanatory comment can help interpret the contract, but the review still needs evidence that the result matches the intended state. Contrast examples should show both, rather than treating a comment as an exemption.

False positives versus missed defects: More silent outputs alone do not establish improvement. Compare acceptable cases that stop being flagged with genuine and unfamiliar defects that remain detected. The purpose of the pair is better discrimination, not fewer comments at any cost.

Familiar syntax versus transferable behavior: Similar syntax makes a contrast easy to inspect. Different syntax in evaluation checks whether the example taught a consequence rather than a textual signature. Anthropic’s diversity guidance supports varying the incidental details while keeping the intended distinction clear.

Architecture / decision scenario

A Claude Code review workflow flags both tolerated scratch-artifact absence and a payment routine that reports success after failure. Developers dismiss many findings, and the team proposes excluding error handlers from review.

The architect instead adds the three cases in the table to the review instructions. Each has the supplied contract, a verdict, and a brief justification. The acceptable outputs demonstrate silence for a narrowly tolerated condition and for an explicitly labeled degraded preview. The payment output demonstrates the specific false-success consequence.

An unseen change then converts a lookup failure into a zero-valued currency amount. The evaluator supplies the caller, which displays that amount as verified. The expected finding follows the behavioral criterion even though the code uses a result object rather than exception handling. A separate scratch cleanup change catches every error, including failures that leave the artifact present; its superficial resemblance to the acceptable example does not exempt it.

Finally, the evaluator withholds the consumer of another fallback value. The expected response identifies the missing consumer contract rather than asserting that every fallback is a defect. Accept this prompt revision only if it reduces the harmless findings while retaining both supported defects and honest uncertainty.

Common exam traps

• Showing only true defects when the problem is false positives on acceptable patterns.
• Turning one acceptable example into an exemption for an entire file type or syntax pattern.
• Treating a comment about intent as proof of safe behavior.
• Evaluating only the examples already included in the prompt.
• Rewarding fewer findings while ignoring missed novel defects.

What to remember

Pair similar-looking inputs whose consequences justify different verdicts. State the contract and the evidence that separates acceptable handling from a hidden failure. Vary incidental details, preserve uncertainty when context is missing, and test the criterion on unfamiliar code rather than expanding a list of exemptions.

Knowledge check

References

• Anthropic, Prompting best practices
• Anthropic, Reduce hallucinations