Turn confidence scores into tested thresholds

Learning objective

Calibrate field-level confidence and review thresholds against labeled validation data.

Core idea

A field’s confidence score is a proposed signal for review, not proof that the value is correct. Calibration asks what that signal means empirically: among values assigned similar scores, how often does the extraction match the labeled answer? An architect should choose review thresholds from those outcomes rather than treating a score such as 0.95 as a measured 95% probability.

The exam guide asks for model-produced field-level scores and thresholds calibrated on labeled validation sets. The field level matters: an invoice number can be clear while its due date is ambiguous. See 5.5.1 for segment eligibility and 5.5.4 for evidence conflicts that require review even when the score clears its threshold.

How it works

Request one signal per extracted field

Extend your extraction contract with a score beside each value. In an illustrative application contract, each field contains value, confidence, and an evidence note. These are application-defined properties, not built-in Claude response metadata. Define confidence consistently, such as a number from 0 to 1 describing confidence that the value matches the supplied source.

Describe the signal in the prompt and the field description. Tell Claude to acknowledge uncertainty when the source is unclear or does not support a value. Anthropic’s hallucination guidance explicitly recommends permitting uncertain answers and restricting factual answers to supplied documents. That guidance improves the conditions for useful scoring; it does not establish the score’s empirical reliability.

Use the existing extraction interface from 4.3.1 or 4.3.2. For a client tool, the application-defined properties live in input_schema; native JSON outputs use output_config.format. Avoid one document-wide score: it cannot say which field needs review, and it makes reliable values share the fate of one doubtful value.

Structured-output enforcement has a relevant limit. JSON outputs and strict tool use share the same schema limits: numeric constraints such as minimum and maximum are not supported, and sending a schema that contains them returns a 400 error. Most SDK helpers strip such constraints from the sent schema and append them to the field’s description; a helper that validates responses still checks them. State the score range in its description and validate the returned range in application code. In the TypeScript SDK, jsonSchemaOutputFormat() transforms unsupported constraints by default but does not validate the response against them, so retain the local range check. Schema validity, numeric range validity, and calibration are three separate checks.

Compare scores with labeled answers

Use documents whose field values people have verified. Include the document types and hard cases identified in 5.5.1. Keep the final evaluation documents separate from prompt examples and threshold tuning so the result tests generalization rather than recognition of the development set.

For each field, record the predicted value, its raw confidence, and whether the comparator accepts it against the label. Group scores into bands and compute empirical accuracy within each band, the same calibration method 4.6.3 applies to review findings. A band scored near 0.90 but correct only 80% of the time is overconfident for that field. A band scored near 0.80 but correct 95% of the time is underconfident. Neither direction can be inferred from the score alone.

Keep counts with the bands and inspect relevant document-type slices. A score might be informative on typed invoices and misleading on scans. Sparse evidence remains sparse even when a band has no observed errors; do not promote it to a reliable probability estimate by rounding to 100%.

Select a threshold from the accepted subset

Suppose the application requires at least 99.5% measured accuracy among automatically accepted due dates. These cumulative validation results are illustrative, not Anthropic limits:

The lowest tested cutoff satisfying this measured target is 0.98, with half the fields accepted. The table supports a candidate rule, not a guarantee about future invoices. Confirm it on held-out data and examine segment counts before removing review. A different field can require a different cutoff.

A calibration table describes the relationship between score and correctness. A threshold table measures the entire set accepted by a proposed rule. Inspect both. One good score band does not justify accepting every higher score if those higher bands contain a distinct failure pattern.

Key distinctions and trade-offs

Calibration and threshold selection answer different questions. Calibration tests the meaning of confidence; threshold selection chooses how much work to accept under a quality requirement. You can use raw-score cutoffs without converting each score into a revised probability, provided you validate the resulting accepted set rather than advertising the raw score as probability.

Increasing the cutoff often reduces acceptance coverage, but better accuracy must be measured. Confident misreadings can survive a strict cutoff. If no cutoff yields acceptable accuracy for a segment, retain review for that segment rather than forcing an automation target. Do not borrow a total-field threshold for due dates because both fields return numeric scores.

Calibration belongs to the extraction configuration and document population you evaluated. A changed prompt, model, or document mix calls for renewed validation. See 5.5.3 for ongoing samples that reveal when accepted results no longer match the validation evidence.

Architecture / decision scenario

A finance team proposes reviewing only invoice fields below 0.90 confidence. The architect collects labeled typed and scanned invoices, preserves a held-out evaluation set, and computes score bands separately for invoice number, total, and due date.

The due-date results match the table: 0.90 accepts too many incorrect values, while 0.98 meets the illustrative measured target with lower coverage. Totals satisfy the same quality target at 0.95 on typed invoices. Scanned totals do not have enough trustworthy accepted examples to establish eligibility, so they retain review.

The application stores field-specific cutoffs and applies them only to validated segments. It sends below-cutoff values to review while retaining their proposed values and evidence. An invalid score such as 95 on a 0-to-1 scale fails local validation rather than silently becoming a highly confident result.

The team tests the selected rules on held-out labeled documents before reducing review. It then samples accepted work as described in 5.5.3. The resulting workflow saves reviewer attention where the scores are useful without assuming that every field shares one meaning for 0.95.

Common exam traps

• Interpreting a self-reported score as measured probability before calibration.
• Choosing 0.90 because it sounds conservative rather than testing accepted accuracy.
• Using one document score or one cutoff for fields with different error behavior.
• Sending minimum and maximum directly in a strict wire schema to enforce confidence bounds.
• Confusing valid score format with reliable score meaning.
• Keeping thresholds unchanged after altering the configuration that produced their evidence.

What to remember

Request field-level scores, validate their range, compare them with labeled correctness, and evaluate the full accepted subset for each proposed cutoff. Choose thresholds by field and validated segment, with coverage and counts visible. Scores direct review only after evidence establishes their usefulness.

Knowledge check

References

• Anthropic, Structured outputs
• Anthropic, Reduce hallucinations
• Anthropic, Define success criteria and build evaluations
• Anthropic, Claude Certified Architect – Foundations Exam Guide