AnthropicClaude Certified Architect – Professional2026-104.4 Diagnose system issues (prompt failure, hallucinations, model mismatch)
Prove which instruction causes the defect
Completed by the knowledge check
Learning objective
Diagnose prompt failures from ambiguous instructions, conflicting rules, and missing or misleading examples.
Core idea
Explicit criteria and contrasting examples establish intended behavior. At the architectural level, the additional task is proving that the assembled prompt caused a production failure rather than blaming it for missing evidence, a tool defect, or application parsing.
A prompt diagnosis needs a reproducible violation and a comparison that changes the relevant instruction while preserving the task’s facts. Anthropic recommends clear instructions, relevant and diverse examples, and inspecting performance through evaluations. Apply those mechanisms as controlled interventions: remove ambiguity, resolve a conflict, or repair examples separately, then measure which change helps.
How it works
Examine the assembled prompt, not its source template
Read the actual system and messages sent for a failed request. Include instructions assembled from templates, retrieved material, examples, and tool results. A correct template can produce a defective request if a conditional clause is omitted or an example for another workflow is inserted.
Compare the expected answer with the raw model response before blaming wording. If the raw answer meets the requirement but a parser discards a field, the prompt is not the first repair. If the deciding fact is absent, investigate context delivery through 4.4.3. The triage method in 4.4.1 keeps these alternatives distinct.
Write a short defect hypothesis tied to a visible instruction. “The exception rule leaves precedence unresolved” is testable. “The prompt is too weak” does not identify what to change or what a passing comparison would demonstrate.
Distinguish ambiguity from conflict
An ambiguous instruction admits several reasonable interpretations. “Prioritize urgent requests” leaves urgency, ranking, and ties undefined. Compare the same request under two explicit interpretations. If subject-matter reviewers also disagree about the expected answer, the product owner must settle the requirement before a prompt owner can repair it.
A conflict instead supplies requirements that cannot both hold for a case. “Never reject a request” and “Reject requests outside the service area” conflict on an out-of-area address. Adding emphasis to both makes neither a consistent rule. Remove the contradiction or state the approved exception and its scope. Anthropic’s Prompting best practices page notes that Claude Opus 4.5 and Claude Opus 4.6 can overtrigger on aggressive tool or skill instructions and recommends dialing back that language. Stronger emphasis is therefore a behavior change to test, not a substitute for resolving the conflict.
Anthropic’s clarity guidance recommends specific output constraints and explaining the reason behind a requirement. Use that guidance to express the resolved business rule. XML tags such as <instructions>, <context>, and <input> help distinguish sections, but they are organizational markup, not application-enforced precedence. A team still has to decide which business rule wins.
Test whether examples teach an unintended rule
Missing examples are a hypothesis when the requirement is explicit but its application to boundary inputs remains unstable. Misleading examples are a different hypothesis: the demonstrations consistently exhibit the behavior the prose forbids. Removing them can be a useful diagnostic comparison before writing replacements.
See 2.3.1 for establishing the zero-shot baseline that separates instruction defects from missing context. Use a small comparison matrix rather than replacing the whole prompt at once:
| Candidate | Change from baseline | Diagnostic question |
|---|---|---|
| Clarified rule | Define the ambiguous condition only | Does uncertainty about the condition explain the failure? |
| Resolved conflict | State the approved exception only | Do competing instructions explain the inconsistent verdicts? |
| Examples removed | Remove demonstrations but retain rules | Were examples reinforcing the wrong behavior? |
| Examples repaired | Replace demonstrations while retaining the resolved rules | Does a representative example set improve boundary decisions? |
Wrap demonstrations in <examples> and <example> tags, as Anthropic documents. Choose relevant, varied examples rather than several surface variations of the same positive case. See 2.3.2 for curating the example set. Then evaluate on different cases. A prompt that reproduces its demonstrations but fails unseen inputs has not established the intended distinction.
A missing example and a missing rule should not be repaired interchangeably. If the business policy never defines an exception, an example silently creates one. Make the rule explicit first so the owner can approve its meaning and the evaluation can check it.
Check context-dependent failures
A rule that works on short requests but fails on long ones deserves an assembled-context comparison. Preserve the rule and vary irrelevant material or its placement while keeping the deciding facts. This tests whether context arrangement contributes, rather than assuming every long-input failure is model incapacity.
Anthropic’s long-context guidance places long documents near the top and the query and instructions after them. Its XML guidance separates document content and metadata. Use those documented arrangements as candidates when length-specific failures implicate prompt structure. Do not claim that a tag or placement guarantees adherence.
After isolated comparisons, test the combined repair. Two changes can interact: removing misleading examples might help only after a conflicting rule is resolved. Record that interaction instead of assigning all improvement to whichever edit was tested last.
Key distinctions and trade-offs
Ambiguity versus policy disagreement: Better wording can express a settled rule. It cannot decide an unresolved business priority on behalf of its owner.
Missing examples versus misleading examples: The first leaves a boundary under-demonstrated; the second actively demonstrates an unwanted boundary. Their diagnostic comparisons differ.
Prompt defect versus shape enforcement: A coherent instruction can still produce occasional format errors. Conversely, a structurally valid result can apply a contradictory business rule. Measure the business verdict separately from formatting.
More examples consume context and create more material to maintain. Adopt them when held-out comparisons show a benefit; an example collection is not a substitute for a rule reviewers can explain. Keep the prompt revision and the cases that justify it together so a later edit can be tested against the same failure.
Architecture / decision scenario
A university routes student requests to specialist queues with a two-second service target. Review finds inconsistent handling of requests mentioning both a scholarship deadline and a disability accommodation. The operations lead proposes routing every such request to two human queues, doubling handling effort.
The architect freezes 80 reviewed requests and runs five trials per configuration. Twelve cases expose an undefined “urgent” label; ten expose a conflict between “one queue only” and “send every accommodation request to accessibility”; six copy examples that always choose financial aid. These groups can overlap, so the team reports both per-condition and overall results.
The student-services owner defines urgency and approves accessibility as the primary queue for the overlapping case, with a financial-aid notification handled downstream. The prompt owner expresses that precedence explicitly. Removing the financial-aid-only demonstrations isolates their contribution; replacing them with varied, tagged boundary examples tests the repair on unseen requests.
The architect chooses the combined prompt repair over universal dual routing because it preserves the service target without duplicating every case. Release requires correct primary routing and required notifications, including previously passing single-topic requests. Failures where the accommodation text never entered messages go to the ingestion owner rather than being counted as prompt defects.
Common exam traps
- Diagnosing from the template while ignoring the assembled request.
- Repeating contradictory instructions more forcefully.
- Letting an example settle a business exception the policy never defines.
- Adding examples when existing examples are causing the failure.
- Testing only the demonstrations used in the prompt.
- Calling every missing field a wording failure before inspecting context and parsing.
What to remember
State a prompt-specific hypothesis, preserve the facts, and compare isolated edits. Ambiguity needs a defined interpretation, conflict needs approved precedence, and examples need an intentional boundary. Validate the combined repair on unseen and previously passing cases.
Knowledge check
A field-service scheduler’s prompt contains “Never reject a booking request” and, later, “Reject requests outside the service area.” Verdicts on out-of-area requests vary across runs. An engineer proposes rewriting both rules in capitals with “IMPORTANT”. What is the right repair?
A triage prompt says “Prioritize urgent requests.” Failures cluster on requests whose urgency is debatable, and the subject-matter reviewers also disagree about the expected answer for those requests. What must happen before a prompt owner can repair the instruction?
An insurer’s claims router has a rule that sends claims mentioning both water damage and mold to the specialist queue. All three demonstrations in the prompt route such claims to general property, and the model often does the same. Which comparison first isolates the examples’ contribution?
A loan-servicing assistant ignores a hardship rule on some requests. The prompt template contains the rule inside a conditional clause, but the logged system content of the failing requests does not contain it. Which change is the first repair?
References
- Anthropic, Prompting best practices
- Anthropic, Demystifying evals for AI agents