AnthropicClaude Certified Architect – Professional2026-101.2 Design end-to-end architectures (input → processing → output → feedback loops)
Turn verified feedback into tested releases
Completed by the knowledge check
Learning objective
Design feedback loops that turn user corrections, review outcomes, and production signals into prompt, retrieval, and evaluation improvements.
Core idea
For an architect responsible for a deployed Claude service, feedback must change a component you can evaluate and release. Diagnose the failing stage, test a targeted change, and observe its released behavior.
Assume that a single response can be refined and that reviewed findings retain their diagnostic labels. The Professional delta is the operating loop across many requests and versions. Anthropic’s evaluation guidance connects measurable success criteria to iterative testing and final validation; its evaluator-optimizer pattern applies feedback when the criteria are clear enough to judge improvement. Extend that discipline to the system’s prompt, evidence selection, and evaluation artifacts.
See 1.2.1 for the stage map and 1.2.3 for evidence-based acceptance. This Bite owns how outcomes from those stages become improvement work.
How it works
Capture three kinds of evidence together
User corrections describe a mismatch between a candidate response and the user’s intended or supported outcome. Retain the original response, corrected passage or value, stated reason, and relevant evidence. A user replacing a date with another date supplies feedback, but not necessarily a verified label. Check it against the authorized source before using it as evaluation truth.
Review outcomes describe an explicit acceptance decision. Preserve confirmed, corrected, and unresolved cases separately as application statuses. A verified source conflict is a useful detection; an unsupported conflict claim is an extraction error. Joining both to one “review failure” count would encourage the wrong change.
Production signals describe what occurred around the response: stage duration, retries, dependency results, acceptance decisions, and delivery outcomes. For Python Claude calls, record the SDK’s public _request_id, which comes from the request-id response header. It identifies the API request for investigation; pair it with your business case identifier rather than treating it as the identity of the entire workflow.
Attach the prompt version, retrieval configuration and source snapshot, selected model, and application rule version to the evidence record. These application-managed fields locate the responsible stage.
Diagnose the component that can repair the defect
Compare the accepted source evidence, the context actually supplied, the candidate, and the verified outcome. Use this sequence to locate the cause before changing a prompt. An answer cannot cite an applicable paragraph that retrieval never supplied. Conversely, retrieving it again adds little if it was present and the response ignored it.
The following application diagnostic table turns a signal into bounded work:
| Verified observation | Candidate intervention | Evidence for evaluation |
|---|---|---|
| Required evidence is present but omitted from the answer | Revise task instructions or examples | Cases requiring that evidence in the answer |
| Exact product identifiers fail to retrieve relevant passages | Test lexical retrieval alongside embeddings | Queries with known relevant chunks |
| A retrieved chunk loses the document’s qualifying context | Test contextualized chunks | Queries whose answer depends on those qualifiers |
| Review rejects semantically incorrect accepted records | Repair validator or routing rule | Verified false acceptances and valid controls |
| Good answers arrive after the service target | Inspect stage calls and retrieval work | Quality and elapsed-time measurements together |
Anthropic’s Contextual Retrieval article describes why lexical BM25 can help exact identifiers that semantic embeddings miss. It also describes adding document-specific context to chunks before embedding and indexing them. These are candidate retrieval interventions with a documented mechanism, not universal fixes for every wrong answer.
For prompt changes, edit the actual system instructions or task content in messages, with examples that express the missing distinction. Define the expected change in behavior before testing. “Be more accurate” does not identify which source selection or reasoning error the examples should repair.
Convert verified outcomes into evaluation cases
Build each regression case from the input, available evidence, expected outcome, and grading criteria. Include the failure that motivated the change and a valid case the change might damage. Anthropic recommends task-specific evaluations that reflect real-world distributions and ambiguous edge cases; a suite of only corrected failures measures repair coverage, not overall service quality.
Use an exact-match or code grader for facts that have a defined answer, such as a normalized product identifier or a required disposition. Use an explicit expert rubric or LLM-based grader where the response needs qualitative judgment. Distinguish factual support, coverage, and tone in the rubric, and check important grader decisions against reviewed labels.
Evaluate retrieval and generation separately enough to diagnose regressions. A query with known relevant chunks measures whether retrieval selected the evidence. A drafting case supplied with those chunks measures whether the model used it correctly. An end-to-end case then checks that the composition still succeeds.
Keep a held-out set apart from the examples used to edit the prompt. Record performance by meaningful workload segment and retain the denominators. Review-selected corrections overrepresent difficult cases; add accepted-case audits and representative traffic so the report does not confuse queue quality with automated quality.
Release a tested component, then check its outcomes
Give the candidate a specific change artifact: a prompt diff, retrieval configuration and index revision, or acceptance-rule revision. Assign the component owner to that artifact and name the evaluation cases that justify it. The architect reviews the trade-off across quality, elapsed time, cost, and review coverage, following Anthropic’s multidimensional success criteria.
Compare the candidate with the baseline on the same evaluation inputs. Require the motivating defect to improve without violating another required dimension. Check latency alongside retrieval quality, and useful coverage alongside complaint counts.
After release, join observations to the version that produced them and check the business outcome. More unedited approvals might reflect better drafts, a different case mix, or rushed review. Verify the interpretation before counting it as improvement. If the released version fails an acceptance criterion, return to the accepted baseline and investigate the affected component; these are application release decisions, not a model-managed learning protocol.
For a Claude API workflow, the memory tool is a tempting alternative: {"type": "memory_20250818", "name": "memory"} lets Claude request persistent file operations under /memories, which your application executes client-side. The page documents support on Claude 4 and later models and says Claude automatically checks its memory directory before starting a task. If your handler accepts every correction into a shared store immediately, later responses can change without a versioned, evaluated release; the prompt version alone no longer explains their inputs. Keep such writes out of the shared policy assistant, or subject the memory snapshot to the same approval and evaluation as other context changes.
Key distinctions and trade-offs
Feedback versus a verified label: An edit records someone’s preference or claim. Evidence and authorized review establish whether it is suitable as evaluation truth.
Retrieval repair versus prompt repair: Missing context calls for evidence-selection work. Misuse of available context calls for instructions, examples, or acceptance changes.
Regression cases versus representative evaluation: Regression cases protect known failures. Representative and held-out cases test whether the repair preserves broader service performance.
Retain version and source identity to reproduce the accepted test.
Architecture / decision scenario
An education provider offers staff a course-policy assistant with a ten-second response target. Staff corrections reveal that it sometimes applies a general attendance rule to learners covered by a specific exception. Reviewers verify the exception against the approved policy.
The evidence record joins the correction to the prompt version, retrieved chunks, and Python _request_id. The exception paragraph was absent from the supplied context for queries containing an exact course code. The retrieval owner tests BM25 alongside embedding retrieval rather than first demanding stricter drafting instructions. A separate case contained the exception but omitted it from the answer; the prompt owner adds a targeted example for that distinction.
The evaluation owner adds both failures, valid general-rule cases, and ambiguous-policy cases. Retrieval tests check the selected chunks; drafting tests provide the correct evidence; end-to-end tests enforce the response target and review outcome. The architect releases only a combination that meets those criteria and declines automatic shared-memory writes because a single staff correction must not change other staff members’ guidance before evaluation. Production corrections and accepted-case audits then test the released versions, closing the loop with observed results rather than a collection of unused edits.
Common exam traps
- Treating every user correction as verified evaluation truth.
- Changing a prompt when retrieval never supplied the required evidence.
- Counting genuine source conflicts as extraction mistakes.
- Testing only the failures used to construct the repair.
- Measuring success only on cases already selected for review.
- Letting unreviewed
memory_20250818writes change shared guidance outside the evaluated release.
What to remember
Join corrections, review outcomes, and production evidence to the version that produced them. Diagnose the failing component, turn verified outcomes into regression and representative cases, and release a measured change. The loop closes when observed results inform the next accepted version.
Knowledge check
A product-support assistant answers wrongly for queries containing exact part numbers such as HX-4402-B. Logs show that the relevant manual sections were not among the retrieved chunks for those queries, while plain-language queries retrieve well. Retrieval uses embeddings only. What should you test first?
max_tokens so answers can cover more manual sectionsA policy assistant retrieves the correct chunk for refund-exception questions, but the chunk reads “The above does not apply to annual plans,” and the answer applies it to the wrong policy because nothing in the chunk says which section “the above” means. Which intervention should you test?
Staff using a benefits assistant can edit any answer, and 600 edits arrived last month. The team plans to add every edited answer to the evaluation suite as ground truth. What should happen first?
A course-policy assistant shared by all staff runs on the Claude API with the memory tool enabled, and the handler writes every staff correction into one /memories directory. Every other behavior change ships as an evaluated, versioned release. Which change keeps the assistant’s behavior tied to evaluated releases?
References
- Anthropic, Define success criteria and build evaluations
- Anthropic, Introducing Contextual Retrieval
- Anthropic, Building effective agents
- Anthropic, Python SDK
- Anthropic, Working with Messages
- Anthropic, Create a Message
- Anthropic, Memory tool