AnthropicClaude Certified Architect – Professional2026-105.1 Implement guardrails and safety controls
Place controls before each harmful effect
Completed by the knowledge check
Learning objective
Design layered controls — input screening, output screening, and tool-call authorization — placed by impact and trust boundary
Core idea
A validated answer and an authorized tool call solve different problems. Neither establishes that the content entering the conversation was safe to process. The architectural task is to identify each transition from untrusted material to a consequential use, then put a control before that transition.
The same workflow can need several controls because it has several effects. A customer message enters the model; a retrieved document enters its context; a response reaches a person; a tool changes a record. Input screening, output screening, and tool authorization protect different boundaries. Anthropic recommends chaining safeguards and treating third-party content as untrusted even when the user is legitimate.
Start with impact, rather than adding a classifier wherever integration is convenient. For every harm, identify the last point where the system can prevent it and the team that owns that point. A daily review of disclosures cannot substitute for a check before disclosure.
How it works
Screen what enters from each source
Anthropic’s jailbreak guidance recommends a lightweight model to pre-screen user input, with output_config.format constraining the screening response. The page’s example uses an application-defined is_harmful boolean. A screening result is a signal your application consumes; it is not an authorization issued by the user.
Separate the signed-in user’s request from material supplied by someone else. A trusted employee can retrieve an attacker-controlled document. Screening only the initial message leaves that second entry point uncovered. Place a screen for that material before context assembly; see 5.1.4 for its implementation.
Decide what the screen protects. An input screen before inference can reduce exposure to adversarial content, but it cannot inspect an answer that has not been generated. A screen after retrieval can inspect the retrieved passage, but cannot retroactively prevent the retrieval service from reading an unauthorized record. Resource access belongs upstream of that read; see 3.2.2.
Hold output before releasing it
Use a separate release decision for model-generated text. Anthropic’s content-moderation guide demonstrates semantic classification and recommends monitoring precision and recall. Apply that pattern to the application’s defined output policy: the moderation rubric must describe the content that would be unacceptable to release, rather than merely asking whether the answer is helpful.
Keep structural validation and semantic screening distinct. output_config.format specifies a supported JSON output structure. It does not establish that a claim is grounded or that a paragraph follows a business policy. The output screen therefore evaluates the proposed response, while deterministic validators retain their own responsibility for checkable relationships.
A release check must precede display on every response path. If the application streams unchecked text to a customer and moderates the completed message afterwards, the check is detective for the text already shown. For an all-or-nothing release policy, buffer the draft, screen it, and publish only an accepted result. Buffering adds visible wait time; record that cost as part of the design rather than calling the system streaming-safe.
Authorize effects at execution
The execution layer independently checks the proposed operation. In an Agent SDK deployment, PreToolUse is the interception point; hookSpecificOutput.permissionDecision: "deny" blocks a tool call. This lesson uses that mechanism as one layer; 5.1.3 owns approval handling and unavailable-check behavior.
An answer screen cannot undo a tool that already sent a message or changed a record. Likewise, permission to call a tool does not establish that its natural-language result is safe to show. Treat the tool’s response as another content source with its own trust level, even when execution was authorized.
Use a compact placement table for the design review. These are illustrative application policies:
| Harm | Boundary that can prevent it | Additional layer |
|---|---|---|
| An adversarial customer request redirects the conversation | Input screen before the main model | Prompt scope and downstream authorization |
| A retrieved attachment introduces instructions | Tool-output screen before context assembly | Source isolation; see 5.1.4 |
| A generated explanation violates release policy | Output screen before display | Review of accepted and rejected examples |
| A proposed account change exceeds the employee’s authority | Authorization before the tool handler executes | Audit of the backend outcome |
Assign an owner to every row. The retrieval team owns source handling, the application team owns release, and the operation’s service owner owns its authorization. A shared safety classifier does not transfer those responsibilities to the classifier team.
Key distinctions and trade-offs
Prevention versus detection: A control is preventive only for effects that have not happened yet. Sampling released answers remains useful for measuring misses, but it cannot erase a disclosure.
Independent responsibilities versus repeated judgments: Two classifiers using the same rubric on the same text may share failures. Layering is stronger when the controls address different questions: content policy, source provenance, and operation authority. Do not describe their error rates as independent without measurements.
Coverage versus latency: Screening inputs, retrieved material, and outputs adds calls and waiting. Place semantic screens where the threat crosses a boundary, and keep checkable authorization in execution code. Measure the combined path, including rejected requests, rather than reporting only the main model’s latency.
Release versus recovery: Define a withheld-output response and an unavailable-operation response separately. See 5.1.2 for screening errors and 5.1.3 for execution checks. A fallback path is another path that must respect the same boundary.
Architecture / decision scenario
A healthcare administrator deploys an assistant that summarizes patient correspondence and proposes appointment changes. The business requires no unchecked generated text to reach patients and no appointment change outside the logged-in scheduler’s authority. A prototype responds in one second; buffering and screening raises measured patient-response latency to 1.8 seconds, within the agreed two-second target. These figures are scenario measurements.
The team proposes one final-answer screen, measured at 1.3 seconds end to end, to minimize integration work. The architect rejects that placement as the whole safety design: it cannot prevent an appointment tool from changing the wrong schedule, and it never screens a malicious attachment before the main model consumes it.
The chosen design screens incoming messages and retrieved attachments, buffers and screens patient-facing drafts, and authorizes appointment changes in the execution path. The scheduling service owner signs off on operation checks; the clinical communications owner approves the release rubric. They accept the measured latency increase because each requirement now has a preventive control. Sampled production review measures misses without being presented as the primary barrier.
Common exam traps
- Placing every control after the final model response.
- Screening the user’s message while trusting retrieved content automatically.
- Treating schema validity as semantic release approval.
- Calling a post-display screen preventive.
- Assuming a successful input screen authorizes every subsequent action.
- Counting several classifiers as independent protection without testing shared failures.
What to remember
Place each control before the effect it governs. Screen each untrusted entry, hold output until its release decision, and authorize tool effects at execution. State the remaining risk, latency cost, and owner at every boundary.
Knowledge check
An insurer’s claims assistant streams each answer to policyholders as it is generated and runs a release-policy screen on the completed message, retracting any message the screen rejects. The business requires that no unchecked generated text reach a policyholder, and buffering keeps measured latency within the agreed target. What should the architect change?
output_config.formatA procurement agent serves signed-in employees and reads supplier-submitted attachments through an internal retrieval tool. The design screens each employee message before inference and each final answer before display. A red-team test shows instructions planted in an attachment redirecting the agent’s plan. Which control covers the entry point the design leaves open?
A bank’s operations assistant proposes account-limit changes through a tool. Every request passes an input screen for adversarial content, and the team proposes letting the tool execute whenever that screen passes. A requirement says no change may exceed the requesting employee’s approval authority. Where should that requirement be enforced?
References
- Anthropic, Mitigate jailbreaks and prompt injections
- Anthropic, Content moderation
- Anthropic, Structured outputs
- Anthropic, Intercept and control agent behavior with hooks