AnthropicClaude Certified Architect – Foundations2026-105.5 Design human review workflows and confidence calibration
Expose weak fields before reducing review
Completed by the knowledge check
Learning objective
Measure extraction quality by document type and individual field instead of overall accuracy alone.
Core idea
For an architect deciding whether an extraction pipeline can bypass human review, the useful question is which results meet the quality requirement. An overall accuracy figure answers a different question: how often the pipeline is right across the mix you measured. Common, easy documents can dominate that figure while a rare document type or an important field fails repeatedly.
Task statement 5.5 in the exam guide explicitly requires accuracy analysis by document type and field before reducing review. Treat that as an eligibility decision for each segment, rather than approval for the entire pipeline. See 5.5.2 for calibrating individual confidence scores and 5.5.3 for auditing accepted results after deployment.
How it works
Define a correct field before counting it
Start with labeled documents: the original source paired with the accepted answer for each requested field. Anthropic’s evaluation guidance recommends task-specific tests that reflect the real task distribution and include difficult or ambiguous cases. Its exact-match example compares outputs with predefined correct answers. For extraction, adapt that comparison to the meaning of each field.
An invoice identifier might require exact character equality. An amount might require equality after an agreed currency and numeric normalization. A missing purchase-order number can be correctly extracted as absent; a guessed number is wrong even if it has the expected format. Declare the comparison rule before scoring, so a favorable metric cannot come from changing the definition after seeing errors.
Keep semantic correctness separate from structural validity. A date can fit the output contract while belonging to the shipment rather than the invoice. See 4.3.6 for semantic checks and 4.3.5 for source normalization. This lesson uses their outcomes to measure quality; it does not replace them with another schema test.
Keep the denominator visible
For each field, compute field accuracy as correctly extracted values divided by evaluated values. For each document type, compute the same measure over its evaluated fields. Then inspect document-type-by-field intersections. The intersection can reveal a failure that neither marginal average makes obvious.
Use counts beside percentages. A result of 100% on four handwritten totals is four successes, not evidence comparable to 100% on four thousand printed totals. A segment with insufficient labeled coverage should retain review while you collect evidence. There is no universal sample count in this objective that automatically authorizes automation.
Do not remove missing or unresolved outputs from the denominator to make the rate rise. Decide how your rubric handles genuinely absent information, ambiguous source evidence, and extraction failures. Report unresolved cases separately when they cannot receive a single factual label, and expose their frequency. Otherwise, the pipeline that declines every difficult field can appear flawless.
Inspect both dimensions and their intersections
The following illustrative results all use the same field-level correctness rule:
| Segment | Correct / evaluated fields | Field accuracy | Decision implication |
|---|---|---|---|
| Typed invoices | 4,950 / 5,000 | 99% | Candidate for further automation validation |
| Scanned invoices | 900 / 1,000 | 90% | Keep review for this document type |
| All invoices | 5,850 / 6,000 | 97.5% | Describes the mix, not each segment |
| Due dates across both types | 920 / 1,000 | 92% | Investigate this field separately |
The due-date row overlaps the document-type rows; do not add it to their totals. It asks a separate question about the same evaluations. Next, split due-date accuracy between typed and scanned invoices to locate the failure precisely.
A document-level measure can also be useful: the proportion of records with every required field correct. Label it separately. Six mostly accurate fields do not establish that most six-field documents are entirely correct, and the errors can cluster in the same documents. Measure complete-record correctness directly if a downstream consumer requires a fully correct record.
Key distinctions and trade-offs
A production-weighted aggregate describes the expected workload mix. A segmented report exposes where performance differs. Keep both: the first supports overall planning, while the second controls which work loses review. Giving rare segments extra validation examples improves their visibility, but their enlarged validation share should not be mistaken for their production share.
Document type and individual field are complementary dimensions. “Scans are weak” identifies an input class; “due dates are weak” identifies an output responsibility. A typed document can still contain an ambiguous date, and a scan can still yield a correct total. The intersection tells you whether the repair should target a source layout, a field instruction, or both.
Measure the proposed accepted subset as well as the full segment. A confidence filter might improve accepted scan accuracy, or it might miss confidently misread digits. Before reducing review, validate the exact combination of document type, field, and acceptance rule that will run. Confidence calibration belongs to 5.5.2; this lesson establishes the segmented evidence that rule must satisfy.
Architecture / decision scenario
A distributor extracts invoice numbers, totals, and due dates with Claude. Management sees 97.5% field accuracy and proposes accepting every result without review. The architect asks for a report built from labeled invoices rather than approving the headline number.
The report shows the typed-versus-scanned split in the table and a weak due-date field across both types. Looking at the intersections reveals two causes: scans lose digits in totals, while typed invoices often print payment terms instead of an explicit due date. Neither defect is solved by observing that most invoice numbers are correct.
The team retains review for scanned totals and unresolved due dates. It evaluates the proposed acceptance rule for typed invoice numbers and totals against labeled examples, including uncommon layouts. Segments with adequate evidence and acceptable error rates can qualify independently; one successful segment does not authorize the others.
After a prompt change, the team reruns the same field comparators across all affected segments. If the due-date repair harms invoice-number extraction, the segmented report exposes the regression. The acceptance decision follows measured results for the revised pipeline rather than carrying forward its predecessor’s overall score.
Common exam traps
- Approving automation because overall accuracy exceeds a target, without checking document types and fields.
- Reporting document-type accuracy while overlooking a weak field shared across those types.
- Treating four correct rare examples as evidence equivalent to thousands of common examples.
- Counting well-formed JSON as a correct extraction.
- Excluding unresolved outputs without reporting how many cases disappear.
- Assuming confidence filtering fixes a segment without evaluating its accepted subset.
What to remember
Define correctness, retain denominators, and inspect document type, field, and their intersections. Overall accuracy describes a mixture; it cannot establish that every part of that mixture is safe to automate. Reduce review only where the proposed acceptance rule has adequate labeled evidence of acceptable quality.
Knowledge check
An invoice-extraction pipeline scores 98.4% field accuracy on 3,000 labeled invoices, above the team’s 98% target. Most labeled invoices are typed; scanned invoices and handwritten totals are rare. The team proposes to stop human review of every extraction. What should the architect require before any review is removed?
Every one of 1,000 labeled scanned invoices contains a due date. The extractor returned a value for 700 of them and marked the other 300 unresolved; 693 of the 700 values match the labels. The team reports 99% accuracy and proposes that scanned due dates bypass review. What should the evaluation report show?
A labeled set holds 4,000 printed totals at 99.2% accuracy and 4 handwritten totals, all 4 correct. The team wants to auto-accept handwritten totals too, because their measured accuracy is 100%. What is the appropriate decision?
An extractor returns six fields per invoice, and on labeled invoices each field scores between 97% and 99% accuracy. A downstream payment system needs records with all six fields correct. How should the team estimate how often it receives one?