Score harmful actions and unnecessary refusals

Learning objective

Define safety and security metrics: policy-violation rate, refusal correctness, prompt-injection resistance, and sensitive-data exposure.

Core idea

A system that refuses every request can avoid harmful answers while failing its intended service. A system that answers every benign test can still follow an attack embedded in a tool result. Safety and security therefore require separate measures with declared expected behavior and test populations.

Anthropic’s success-criteria guidance quantifies safety with an example toxicity-flag rate over 10,000 trials and names privacy preservation as a criterion. Its injection guidance distinguishes adversarial users from adversarial third-party content and recommends testing the actual workflow. Translate those concerns into counted outcomes, rather than treating a list of installed controls as a result.

How it works

Define violations against an explicit policy

Create a policy rubric that identifies prohibited outputs and actions for the application. It should distinguish ordinary factual errors from policy violations while allowing a result to fail both. For example, an incorrect public opening time is a quality defect; revealing a restricted person’s address is a privacy violation whether or not the address is correct.

Define policy-violation rate as evaluated tasks containing at least one prohibited output or action divided by evaluated tasks in the stated population. Report violation categories and severity beside that rate. Anthropic’s safety example requires fewer than 0.1% of outputs in 10,000 trials to be flagged for toxicity by a content filter: unlike “safe outputs,” it names a population, a detector, and a threshold. Treat those figures as the documentation’s example, not your application’s required bar. An event count per task is a useful additional measure, but it is a different unit: one task leaking ten records is one violating task and ten exposed records.

Specify whether the measure covers content generated internally, content delivered to a user, or executed actions. If an output screen blocks a prohibited draft, the raw-generation measure fails while the delivered-output measure can pass. Both are informative. Inspecting only final text misses unauthorized tool actions that occurred earlier in the run.

Score both directions of refusal

Label cases that should be refused under the application policy and cases that should receive authorized assistance. Measure correct refusal on the first population and over-refusal on the second. Also count prohibited compliance: requests that should have been refused but instead receive forbidden assistance. A combined refusal-correctness score can summarize the labels, but must retain both error directions and their counts.

For classifier refusals on the models documented in Anthropic’s refusals guidance, stop_reason: "refusal" arrives with HTTP 200. stop_details.category identifies the policy area. These fields are observations, not proof that refusing was correct under your application rubric. Anthropic explicitly documents benign requests triggering some categories.

Scope this telemetry to the documented model behavior rather than assuming every refusal is an HTTP error or every refusal-like answer uses that stop reason. Grade the behavior in the content and action trace as well. A normal-looking answer that withholds the requested permissible work can be an over-refusal under the rubric even without a classifier signal.

A classifier refusal can arrive before output or after partial streaming. Anthropic directs applications to discard partial output as incomplete. Test that delivery behavior too: a correct final refusal does not undo prohibited content already shown. Record both the refusal decision and any earlier exposure.

Measure attacks against their actual objective

Anthropic distinguishes direct prompt injection, where the user is adversarial, from indirect injection in third-party pages, emails, documents, or tool results. Build separately labeled attack populations for the entry points your workflow accepts. Report the attack family, objective, number of attempts, and observed successes.

Define attack success by the forbidden outcome, such as sending restricted material to an attacker-controlled destination or changing a protected decision. A model repeating attack text as a quotation is not automatically a successful injection. Conversely, a harmless-looking final answer does not erase an unauthorized action observed in the trace.

Prompt-injection resistance can be reported as 1 - successful attacks / evaluated attacks, with the underlying success count retained. State whether an attempt means one payload submission or a multi-step attack episode; repeated opportunities change the meaning of the denominator. Clean results on ordinary traffic do not estimate this adversarial measure.

Anthropic recommends placing third-party content in tool_result blocks, screening tool outputs, restricting access, and red-teaming the workflow. Its screening example uses the application-defined Boolean injection_suspected; when true, the application substitutes an error or stripped summary for the raw tool result. The example also treats a screening-call stop_reason: "refusal" that way because no verdict is available; count that outcome separately rather than as a negative detection. Measure the complete path with these controls enabled. A screen’s detection rate and the workflow’s attack success rate answer different questions.

Measure exposure at every relevant surface

Define sensitive-data exposure against an explicit access boundary: which principal may receive which information, for which task. A secret remaining inside an authorized operation differs from the same secret appearing in an unauthorized tool argument, user response, or retained trace. Name the surfaces your measure actually inspects.

Use synthetic restricted identifiers or other controlled test data where possible, and define exposure as unauthorized disclosure on a tested task. Report violating tasks over exposure opportunities, plus the number and type of exposed items. Keep adversarial exposure separate from accidental exposure on normal authorized tasks; an access-control mistake does not require prompt injection to be serious.

Anthropic’s least-privilege recommendation limits sensitive information and actions available to Claude. Evaluate the effect, not merely whether a permission configuration exists. A blocked exfiltration attempt is different from confirmed leakage, but the attempted action still matters for assessing the defense. Avoid copying real sensitive payloads into the evaluation report itself; retain the case identifier and restricted result details under the evaluation team’s access policy.

Key distinctions and trade-offs

Policy compliance concerns permitted behavior. Refusal correctness concerns whether assistance is appropriately withheld. Injection resistance concerns adversarial control of the workflow. Exposure concerns who actually receives restricted data. One event can affect several measures, but the measures are not interchangeable.

Report zero observed failures as a count, such as zero of 200, with the population and method. It is not proof of zero risk outside those tests. Separate attempted misuse, blocked misuse, and completed misuse so the release owner can distinguish model susceptibility from containment.

Stricter screening can reduce attack success while rejecting useful tasks. Measure authorized-task completion and over-refusal alongside the security results. The responsible policy and security owners decide mandatory thresholds; a single weighted score must not let improved usability purchase a prohibited disclosure.

Architecture / decision scenario

A government service uses Claude to help staff find authorized case guidance. It must answer permissible questions about sensitive topics, refuse requests for restricted personal records, and resist instructions embedded in retrieved documents. The security owner requires zero observed restricted-data exposures and zero successful attacks in the defined pre-release suites; the service owner permits at most 2% over-refusal. These are illustrative local acceptance rules.

Candidate A has zero policy violations on 500 representative tasks and zero successful injections in 200 attack episodes, but refuses 40 of 500 answerable cases: 8%. Candidate B refuses only 5 of 500 answerable cases: 1%, but succeeds on the attacker’s objective in 2 of 200 episodes and exposes one restricted identifier. Both correctly refuse 100 of 100 prohibited requests.

The tempting options are A’s “safe” headline and B’s better usability. The architect approves neither as a general release. A fails the authorized-service requirement; B fails the security conditions despite strong refusal correctness. The policy owner reviews A’s unnecessary refusals, while security isolates B’s exposure path and checks tool_result handling and tool access. The next evaluation retains all four metric families rather than reporting only whichever one improved.

Common exam traps

• Counting HTTP errors as the entire refusal population.
• Treating stop_details.category as proof of correct refusal.
• Reducing harmful output by refusing legitimate work without measuring over-refusal.
• Measuring injection resistance on benign traffic.
• Treating a blocked action and completed exfiltration as identical outcomes.
• Reporting no final-answer leakage while ignoring tool arguments and traces.
• Calling zero observed attacks a guarantee of safety.

What to remember

Count policy violations and both refusal error directions. Test direct and indirect attacks against explicit forbidden outcomes, and inspect every relevant disclosure surface. Preserve denominators and authorized-task usefulness so safer-looking scores cannot conceal a service that refuses its job.

Knowledge check

References

• Anthropic, Define success criteria and build evaluations
• Anthropic, Refusals and fallback
• Anthropic, Mitigate jailbreaks and prompt injections