Extract evidence across layouts and omissions

Learning objective

Use extraction examples spanning different document structures and missing or informal values to reduce fabricated and empty fields.

Core idea

An empty extraction field has two possible causes: the source does not contain the value, or the extractor overlooked it. A fabricated value creates a third problem by making missing evidence look complete. Examples should demonstrate how to distinguish all three outcomes.

Anthropic recommends diverse examples that cover edge cases and avoid accidental patterns. For extraction, vary where the evidence appears and how it is expressed. A bibliography and an inline citation can both supply citation evidence; a sample size can appear under Methodology or inside a results paragraph.

Also demonstrate a legitimate unknown. Anthropic’s hallucination guidance recommends explicitly permitting uncertainty, grounding responses in quotes, and restricting claims to provided information. A worked extraction with an honest null teaches that the absence of evidence is a permitted output, rather than a blank the model must fill from general knowledge.

How it works

Vary the location of the same information

Suppose a research-extraction application wants citations, sample_size, and a latency measurement. Define what each field means before constructing examples. In this application, sample_size counts timed runs rather than users, records, or warm-up iterations. Without that definition, an example can demonstrate a number’s location while teaching the wrong number to select.

Use examples that separate layout from meaning:

These field names and interpretations are application-defined. In this contract, citations is a list of reference strings copied from the supplied text.

Preserve what an informal value actually says

Choose an explicit normalization policy. For this application’s prompt, a fractional time expression can be converted when its unit and quantity are clear, but approximation must remain visible. “Roughly a fifth of a second” supports an approximate 200 milliseconds; it does not support a precisely measured 187 milliseconds or an invented error interval.

A finished target record demonstrates the policy:

{
  "sample_size": null,
  "citations": [],
  "latency": {
    "value_ms": 200,
    "approximate": true,
    "source_text": "roughly a fifth of a second, give or take"
  }
}

The source excerpt contains no timed-run count and no references, so those fields do not borrow values from another example. source_text preserves the evidence for the conversion. The output does not claim that “give or take” identifies a measured range.

If the source instead says “roughly 200” without a unit, this application’s prompt leaves value_ms unknown and preserves the wording. The numeric appearance alone does not establish milliseconds. If the source says only “felt faster,” there is no numeric latency to normalize at all.

The example must reflect the chosen output contract. Do not demonstrate an absent value as an empty string in one example, null in another, and zero in a third unless the instructions explicitly distinguish those meanings. See 0.3 for property presence and nullability; here the deciding issue is whether evidence supports a value, not how to define a schema.

Make evidence recovery different from guessing

For documents with embedded details, the example should show the supporting passage and extracted field together. For long documents (more than 20k tokens), Anthropic recommends extracting relevant word-for-word quotes before performing the task to ground the response in the supplied text. Its long-context prompting guidance, for documents or data-rich inputs of 20k+ tokens, also recommends wrappers with <document>, <source>, and <document_content> when supplying multiple documents.

Use those wrappers to preserve which document supports each field. A benchmark with no sample size must not inherit 200 from a neighboring benchmark merely because both occur in one prompt. Separate the worked examples from the live documents as taught in 4.2.1.

An inline reference such as Lee (2024) supports that reference text, not a guessed paper title, URL, or publisher. The same principle applies to narrative measurements: extract the supplied evidence, apply only declared conversions, and preserve qualifiers. Explicitly instruct Claude to use the supplied documents and admit insufficient information when the field cannot be supported.

Key distinctions and trade-offs

Absent evidence versus unfamiliar layout: A value in a results paragraph should not become null because the example set only used Methodology headings. A truly absent value should not become a guess because a downstream record expects the field.

Normalization versus fabrication: A declared conversion can change representation while preserving meaning. Removing “roughly,” inventing precision, or choosing an unstated unit changes the claim. The examples must demonstrate that boundary.

Empty collection versus unknown scalar: In this application’s contract, no cited sources produces citations: []; an unstated run count produces sample_size: null. Neither output is an extraction failure when the supplied document genuinely lacks the evidence.

Examples versus guaranteed accuracy: Examples improve consistency and clarify handling, but Anthropic states that hallucination-reduction techniques do not eliminate hallucinations entirely. Keep evidence attached and validate important extracted information; see 4.4 for validation and feedback mechanisms.

Architecture / decision scenario

A multi-agent research workflow extracts records from benchmark reports. The existing prompt works for a Methodology section and bibliography but returns empty fields for narrative reports. It also invents precise latency values from informal descriptions.

The architect adds the four examples in the table, with the same output contract throughout. The first establishes the easy layout. The second moves the same facts into narrative text and inline references. The third demonstrates approximate conversion and a genuinely missing count. The fourth demonstrates that qualitative language supplies no quantitative value.

An evaluation report mentions 320 timed runs inside a caption, cites a method inline, and describes latency as about half a second. The expected output recovers 320, preserves the supplied reference text, and reports approximately 500 milliseconds with its source wording. A second report omits the count entirely; it retains sample_size: null rather than inheriting 320.

If a count remains empty, inspect whether the prompt examples teach only a heading location. If a precise number appears without support, inspect the normalization policy and missing-value demonstration. Neither making every field optional nor ordering Claude to always populate every value repairs the underlying evidence distinction.

Common exam traps

• Treating a missing Methodology heading as proof that sample size is absent.
• Requiring a bibliography before recognizing an inline citation.
• Expanding an incomplete reference with a guessed title or URL.
• Converting informal wording into unsupported precision or an unstated unit.
• Treating every honest null as failure, or every populated field as success.
• Copying values from the examples or neighboring documents into the live record.

What to remember

Show different layouts, informal values, and true absences in the example set. Recover evidence wherever it appears, normalize only under an explicit policy, preserve approximation and source wording, and allow honest unknowns. The useful result is supported information rather than a record that merely looks complete.

Knowledge check

References

• Anthropic, Prompting best practices
• Anthropic, Reduce hallucinations