Audit accepted results across every stratum

Learning objective

Sample apparently high-confidence extractions with stratified random auditing.

Core idea

For an architect reducing extraction review, the most important unseen errors are the ones the acceptance rule lets through. Reviewing low-confidence fields tells you about flagged work. It does not measure the correctness of apparently high-confidence results that bypass people.

The exam guide names stratified random sampling for measuring errors in high-confidence extractions and detecting unfamiliar patterns. Split the accepted population into relevant groups, then select randomly within each group. This preserves visibility into small groups that a single workload-wide random sample can miss. See 5.5.2 for initial confidence calibration; this lesson adds continuing evidence about the work those thresholds accept.

How it works

Define the population being audited

Start with accepted extractions from a stated production period. Retain the document, extracted fields, confidence scores, document type, extraction configuration, and field-level acceptance decisions needed to reproduce the quality comparison. Audit results should describe that accepted population, not a mixture of accepted work and known failures already sent to reviewers.

A low-confidence review queue is a selected set of difficult cases. Its corrections are useful, but its error rate cannot stand in for the error rate of accepted results. Conversely, a high-confidence sample cannot describe all submitted documents without accounting for the work excluded by the acceptance rule. Name the population whenever you report a percentage.

Use the same field correctness rubric as the validation evaluation in 5.5.1. Anthropic’s evaluation guidance calls for measurable task-specific criteria and edge cases. Consistent grading makes an audit result comparable with its validation baseline; changing the comparator can produce an apparent regression or improvement that is only a measurement change.

Partition the accepted population into strata

A stratum is one mutually exclusive group within the population. Useful extraction strata reflect plausible differences in failure behavior: document type, supplier template, supplier volume, or time since the supplier or template first appeared. Choose the partition before examining individual sampled outputs.

Avoid overlapping buckets that count one document twice. For example, define groups as high-volume suppliers with typed layouts, high-volume suppliers with scanned layouts, and low-volume suppliers of either format. If you need format detail within the third group, subdivide it explicitly. The categories should cover all accepted records, including a place for records with missing classification metadata.

A stratum with no accepted records has no accepted work to sample. If all scanned invoices remain under full review, their review outcomes belong in the reviewed population rather than an invented high-confidence audit stratum. When scans become eligible for automatic acceptance, they enter the audit population too.

Draw randomly within each group

The sampling unit is an invoice with at least one auto-accepted field; compute audit error rates only over its fields that bypassed review, excluding fields already corrected by reviewers. In this illustrative month, that population contains 10,000 invoices:

Select records randomly without replacement within each stratum. The counts illustrate allocation; they are not an Anthropic requirement or proof of a particular precision. Choose counts from the evidence quality and error sensitivity the application needs. No fixed percentage makes every rare group adequately measured.

Choosing the easiest records, the first records of the day, or whichever outputs look suspicious is not random auditing. Suspicious outputs can receive additional targeted review, but keep those findings separate from the random sample used to estimate rates. Otherwise, the measurement silently changes into a search for defects.

Measure errors and classify what failed

Review each auto-accepted field in each sampled invoice against its source, using its original extraction before any audit correction. Report errors divided by evaluated values within each stratum and field, with counts. Also record the failure pattern: a due date confused with a delivery date, a handwritten digit misread, or a field omitted after a template rearrangement.

Oversampling rare strata changes their share in the sample. For this illustrative calculation, suppose every invoice in the table has an auto-accepted due date. If the three strata have due-date error rates of 1%, 4%, and 10%, the production-weighted error rate is 0.80 × 1% + 0.15 × 4% + 0.05 × 10% = 1.9%. For each field, weight by its auto-accepted value counts in the population, not the audit draw shares; those counts equal the invoice counts in this due-date example. Keep the stratum rates visible even when reporting this aggregate; the rare group’s 10% remains actionable.

The audit draws only from the accepted population, randomly within each stratum, and weights its summary by population counts rather than draw shares.

An error rate and an error pattern serve different purposes. The rate estimates how often accepted work fails under the rubric. The pattern identifies a cause that the initial validation set may not have represented. Neither a high score nor repeated correct examples of another layout rules out that cause.

Key distinctions and trade-offs

Stratification determines who receives sampling attention; randomness determines which records within a group receive that attention. Dropping either changes the result. A global random draw can underrepresent small groups, while handpicked records within strata cannot support an ordinary error-rate estimate for those groups.

More audit attention improves visibility but consumes reviewer capacity. Spend that capacity across groups deliberately rather than assuming proportional allocation is always sufficient. A rare layout may deserve a larger sampling fraction because its individual templates have less validation evidence or its errors matter more.

Finding zero errors is an observation about the sample, not proof that the population has none. Keep denominators and the audit period visible. Continue auditing after threshold validation, particularly when documents or extraction configuration change; one predeployment measurement cannot reveal every later failure pattern.

Architecture / decision scenario

An invoice extractor accepts due dates above their calibrated threshold. Most accepted records use established typed layouts. The operations team proposes auditing 1% of all accepted documents without grouping them.

The architect instead defines the three strata in the table and selects randomly within each. Reviewers find several high-confidence due-date errors among low-volume suppliers: suppliers moved the delivery date into the position previously occupied by the payment deadline. Those results passed the confidence rule because the model extracted a clear date from the wrong location.

The team retains the source, predicted value, corrected label, and pattern description. It routes the affected layouts to review, adds representative failures to the evaluation corpus, and tests a targeted extraction repair. See 4.4.4 for diagnostic pattern fields and 5.5.2 for revalidating thresholds.

The audit report presents field error rates by stratum and a population-weighted summary. It does not average the 200 sampled invoices as if their layout proportions matched production. Auditing continues for accepted strata while the affected layouts receive targeted review, so reducing automation in one group does not stop measurement in the others.

Common exam traps

• Auditing only low-confidence outputs and calling the result an accepted-set error rate.
• Drawing a global random percentage and assuming every rare document type is represented adequately.
• Picking suspicious records within each stratum instead of selecting randomly.
• Averaging oversampled strata without accounting for production proportions.
• Treating zero observed errors as proof of zero population errors.
• Collecting error counts without preserving the unfamiliar patterns needed to repair extraction.

What to remember

Audit the accepted population, partition it into defined strata, and draw randomly within each. Keep field-level counts and patterns, and weight aggregate estimates by the population being described. Continuing samples reveal confident errors that initial calibration and low-confidence review cannot see.

Knowledge check

References

• Anthropic, Define success criteria and build evaluations
• Anthropic, Claude Certified Architect – Foundations Exam Guide