Trace attacks from content to capability

Learning objective

Identify security and privacy risks: prompt injection, data exfiltration, sensitive-data exposure, and over-privileged access.

Core idea

For an architect reviewing a Claude system, the threat model must connect three things: what an adversary can influence, what information the agent can read, and what the system can send or change. A warning about malicious text is incomplete until it names the data or authority that text could reach.

Anthropic distinguishes direct prompt injection by an adversarial application user from indirect injection in third-party content processed for a trusted user. The delivery channel does not determine the author’s trustworthiness. A document returned by your own retrieval service can still contain instructions written by an attacker.

How it works

Identify the instruction source and trust crossing

Prompt injection attempts to redirect the model away from the application’s or user’s instructions. Direct injection enters through the user’s request. Indirect injection can enter through a retrieved document, inbound email, webpage, uploaded-file extraction, or tool result. The trusted employee asking for a summary may be the victim rather than the attacker.

Write the threat as an explicit path: an external supplier controls a document; the document reaches the invoice-analysis agent; embedded instructions request access to another supplier’s records. That description identifies the point where source content is being promoted into authority. A generic “malicious prompt” label does not tell the implementation team which boundary to inspect.

Anthropic recommends delivering third-party content inside tool_result blocks, identifying its nature and source in the tool description or result structure, and JSON-encoding untrusted strings. Keep the application’s instructions in the instruction channel rather than embedding them in those results. These mechanisms help preserve the distinction between data and instructions; they do not establish that every injected instruction will be ignored.

The risk register should preserve the attack path even when a tool-output screen returns injection_suspected: true for the test payload. Anthropic documents this classifier verdict for screening tool output before returning a tool_result. Detection of one payload is evidence about that test, not proof that an attacker cannot vary the document or its wording; see 3.2.5 for injection testing and access trails.

Distinguish exfiltration from accidental disclosure

Data exfiltration is an unauthorized extraction path. An injection might persuade an agent to read confidential data and then place it in an outbound request, email, or another externally visible result. The meaningful question is whether sensitive information can reach an unauthorized destination, not whether the output contains an obviously malicious phrase.

Examine the combination of untrusted input, sensitive access, and an outbound channel. A document reader with no secrets may still produce a corrupted summary. A reader that can access customer records and send arbitrary HTTP requests creates a different consequence: an attacker can try to move those records outside the approved boundary.

Anthropic’s secure-deployment guidance recommends network restrictions through a proxy, narrow filesystem access, and credentials injected outside the agent’s environment; see 3.2.3 for credential custody. These break parts of that path even if the model is redirected. An endpoint allowlist still needs scrutiny: an allowed service might accept an attacker-selected recipient or expose a public artifact. Domain approval alone does not define the business authorization of every request.

Sensitive-data exposure does not require an attacker or an injection. A generated explanation might include another person’s account details; an overly broad retrieved passage might put unrelated confidential material in the agent’s context. OWASP’s sensitive-information-disclosure category includes personal, financial, health, business, and credential data, and identifies error-message leakage as another concern.

Include operational copies in the data-flow review: debugging output, incident attachments, evaluation cases, and retained tool results. For each copy, name the intended audience and the information necessary for its purpose. This is risk identification, not a claim that any particular provider trains on, retains, or deletes that copy; vendor-specific data-handling and compliance decisions belong in 5.4.

Identify the authority that exceeds the task

Over-privileged access means the agent or its tool principal can do more than the business assignment needs. OWASP distinguishes excessive functionality, permissions, and autonomy. A broad shell interface is a functionality concern; an account that can read every tenant is a permission concern; unattended execution of consequential writes adds an autonomy concern.

Inspect effective downstream permissions, not only the tools listed in a prompt. A narrowly named lookup tool can still run under a principal (the identity under which a tool runs) with access to all customers; see 3.2.1 for principal selection and 3.2.2 for resource-scoped execution. Conversely, removing write tools does not remove confidentiality risk when the reader can access secrets. Anthropic explicitly warns that read-only code mounts can expose .env, credential files, and private keys.

Make the review artifact concrete: a capability inventory containing each tool, the principal it uses, reachable resources, readable fields, writable actions, and permitted destinations. Identify a risk when one of these exceeds the task’s intended scope. The inventory is an architectural artifact derived from the documented least-privilege principle, not an Anthropic API response.

Key distinctions and trade-offs

The categories describe different parts of a threat path:

One incident can involve all four. Keeping them distinct helps you choose controls later: a better input classifier does not narrow a service account, and a narrower service account does not sanitize diagnostic output. Isolation can increase integration effort and restrict useful workflows; that cost must be compared with the reachable harm.

Architecture / decision scenario

A technology company processes supplier invoices within a four-hour settlement window. Its six-person operations team wants one agent to read supplier PDFs, query the full vendor database, email vendors, and open arbitrary support URLs. A prototype also writes complete tool results into broadly accessible incident attachments.

A supplier-controlled PDF asks the agent to fetch a different vendor’s banking record and email it to a “verification” address. The tempting design retains every capability and adds a system instruction never to disclose secrets. The architect identifies indirect prompt injection at the PDF boundary, exfiltration through email or HTTP, excess database privilege, and an independent disclosure path through incident attachments.

The security owner records each path and its reachable data. The integration owner proposes scoped database access and a separate sending service that checks authorized recipients; the platform owner evaluates proxy-enforced destinations and credentials outside the reader. The operations owner removes unnecessary fields from incident material. These are candidate treatments, with acceptance decided in 5.2.5, rather than a claim that a successful screening run closes the risks.

The team can still meet its four-hour window with a restricted reader and authorized sending step. Giving the same reader unrestricted retrieval and outbound authority offers less integration work but makes one malicious document capable of crossing several boundaries.

Common exam traps

• Trusting a retrieved document because the retrieval service is internal.
• Treating a trusted user’s session as proof that every document author is trusted.
• Calling every disclosure an injection attack.
• Equating read-only access with an absence of privacy risk.
• Auditing tool names while ignoring their principals and destinations.
• Treating JSON encoding or a system instruction as complete attack prevention.

What to remember

Follow the path from attacker-controlled content through readable data to outputs and actions. Identify injection, exfiltration, accidental exposure, and excessive authority separately. A useful threat model names the boundary, reachable information, destination, and owner who can change the capability.

Knowledge check

References

• Anthropic, Mitigate jailbreaks and prompt injections
• Anthropic, Securely deploying AI agents
• OWASP, LLM02 Sensitive Information Disclosure
• OWASP, LLM06 Excessive Agency