Price classifier errors before choosing controls

Learning objective

Choose deterministic controls where violations are unacceptable and model-based classifiers where judgment is needed, and test both for bypasses, over-blocking, and false-positive cost

Core idea

A rule that must hold on every execution belongs in code at that execution boundary. A semantic question such as whether a passage threatens someone requires contextual judgment, so a classifier can supply a decision signal. The Professional decision is how to combine those mechanisms without hiding either missed harms or the cost of blocking legitimate work.

Anthropic’s moderation guidance recommends semantic classification and precision-and-recall tracking. Its jailbreak guidance recommends pre-screening and chained safeguards. Neither makes a model’s policy judgment infallible. Your release decision needs two populations: requests that must be stopped and legitimate requests that resemble them.

A control that blocks everything has excellent catch coverage and no usable service. Define success as preventing specified harms while preserving an agreed level of legitimate task completion.

How it works

Divide predicates from judgments

Start with the exact decision. An application can check whether a requested record belongs to the authenticated principal or whether an integer amount exceeds a configured ceiling. Those predicates use trusted state and have definite expected outcomes. A PreToolUse denial carries an execution decision for an Agent SDK tool; 5.1.3 covers that operating path.

A classifier addresses meaning that a predicate does not capture. Anthropic’s content-moderation guide contrasts contextual understanding with simple pattern matching and shows classification into risk levels. A threatening sentence, a quotation of a threat in an incident report, and a harmless idiom can share words while requiring different treatment.

Do not replace a checkable predicate with the classifier’s explanation. Conversely, do not claim that a keyword list completely implements a semantic policy. Retain a deterministic operation check even when a semantic screen has accepted the request.

Make the classifier’s interface explicit

For a Claude API classifier using Claude Haiku 5.5, use output_config.format with type: "json_schema" to specify its verdict shape. Define application labels such as pass, review, and block; these labels are your policy interface, not Anthropic stop reasons. Include a short explanation where it helps adjudication, rather than requesting private reasoning.

Keep the rubric outside the candidate text and describe ambiguous examples. The moderation guide’s risk_level illustrates graded responses rather than an undifferentiated yes/no decision. A middle category can feed review, but only if the review path has an owner and capacity. A queue that silently releases unresolved items is another fail-open path.

Treat transport and generation outcomes separately from verdicts. stop_reason: "refusal" can return HTTP 200 and take precedence over the structured schema. Anthropic’s harmlessness-screen guidance treats a screening refusal as harmful. stop_reason: "max_tokens" can leave an incomplete result; the documented remedy is a higher max_tokens budget. Hold the candidate while retrying within the application’s deadline, then withhold it if no accepted verdict arrives.

Structured outputs also document case variation in string enum values. Compare labels case-insensitively, avoid labels that differ only in case, and reject unknown labels. Parsing valid JSON is not the same as receiving a recognized safe verdict.

Refusal and max_tokens outcomes are treated separately from verdicts, and unknown labels are rejected.

Test two kinds of failure

Keep a must-block set and a must-pass set, with adjudicated expected outcomes. Anthropic recommends red-teaming jailbreak defenses and refining moderation from precision and recall. Build these sets from the application’s own policy and production cases; see 4.2.1 for dataset construction.

Evaluate deterministic controls through the actual execution route. Try a wrong principal, a threshold boundary, malformed input, and an alternate tool that reaches the same effect. Check the resulting side effects as well as the returned denial. A perfectly repeatable predicate still fails its purpose if one path bypasses it.

For classifiers, include role-play attacks, paraphrases, multi-turn attempts, quoted harmful text, legitimate discussion, and language variations the service supports. Rerun after changes to the classifier prompt, model, and surrounding workflow. An attack screen must also be challenged with text that tells the screen to label itself safe.

Use separate denominators:

An aggregate accuracy score hides the difference. Report each result by consequential category, and keep classifier mistakes separate from unavailable checks. See 4.1.4 for safety metric definitions across the wider system.

Key distinctions and trade-offs

A false positive has an operational cost: lost task completion, review work, and additional user attempts. Calculate that cost using expected traffic, rather than pricing only model calls. The moderation guide recommends analyzing flagged content and refining the rubric; review legitimate blocked cases to discover which distinctions it misses.

Set hard release criteria for deterministic prohibitions and explicit tolerances for semantic screening. Zero observed misses on a finite classifier suite supports a measured result, not a universal guarantee. For a checkable prohibition, any execution bypass is a defect in implementation or coverage.

A review category can reduce inappropriate automatic blocking while adding delay and staffing. Lowering a threshold may catch more harmful text and also increase false positives. Choose the operating point with the policy owner and the service owner, using both populations and the available review budget.

Architecture / decision scenario

An education platform moderates student discussion and lets instructors publish course notices. Its rule forbids publishing into another instructor’s course. Separately, it must distinguish bullying from quotations used in literature discussion. The academic safety owner approves the semantic rubric; the course service owns publication authorization.

On a held-out scenario dataset, classifier A catches 98 of 100 harmful comments and wrongly blocks 30 of 1,000 legitimate comments. Classifier B catches 96 harmful comments and blocks five legitimate comments. A keyword-only alternative misses contextual bullying and blocks quoted passages, so it cannot carry the semantic policy.

At 100,000 legitimate comments per month, the measured rates imply 3,000 versus 500 false blocks if the dataset represents that traffic. At four review minutes each, that is 200 versus about 33 staff hours. The team has 40 hours available. These are illustrative workload assumptions, not vendor performance figures.

The architect cannot select A solely for recall or B solely for cost. The safety owner requires at least 97% catch coverage on the agreed suite, so neither candidate meets both operating constraints. The team refines the rubric and tests a review category before release. Automated comment moderation remains unreleased while the existing instructor-review process continues; the publication ownership check alone cannot satisfy the semantic safety requirement. Publication ownership remains a deterministic service check in every candidate, including forced calls to an alternate publish route.

Common exam traps

• Using a model judgment for a checkable ownership rule.
• Treating keyword matching as complete semantic moderation.
• Optimizing attack catch rate without legitimate look-alikes.
• Counting a classifier outage as a policy-positive result.
• Treating structured JSON as proof that the classification is correct.
• Claiming universal safety from zero misses on a finite test set.

What to remember

Enforce checkable prohibitions in the execution path and use classifiers for contextual judgment. Test both bypasses and over-blocking. Price false positives at production volume, and release only when safety targets and operating capacity can both be met.

Knowledge check

References

• Anthropic, Content moderation
• Anthropic, Mitigate jailbreaks and prompt injections
• Anthropic, Structured outputs
• Anthropic, Intercept and control agent behavior with hooks