Building a moderation filter with Claude

Why it matters

A semantic classifier is the screening layer you place on input or output when no deterministic rule can describe the harm. Knowing how one is built shows you where it errs.

What to practise

Write the moderation rules as BLOCK and ALLOW category descriptions in the prompt, insert the user text, and have Claude return only the label under a small max_tokens budget. Keep a rollercoaster forum on topic by editing the descriptions alone. Then ask for reasoning inside <thinking> tags and the label inside <output> tags, which needs a larger max_tokens, and add a few labelled examples. Watch which borderline posts change category.

How it fits

This is the input and output screening of 5.1.1. Before deploying it, price its false blocks and missed harms as 5.1.2 asks, and pair it with a deterministic control for anything it must never let through. The cookbook returns free text, so compare it with the schema-constrained output_config.format verdict that 5.1.1 and 5.1.4 use.