AnthropicClaude Certified Architect – Professional2026-103.5 Design a RAG pipeline with appropriate chunking and indexing strategies
Preserve meaning and permissions during ingestion
Completed by the knowledge check
Learning objective
Design ingestion that parses, normalizes, and tags documents with metadata and access permissions.
Core idea
An ingestion pipeline produces the units your retrieval service can safely select. Parsing preserves meaning, normalization makes equivalent source forms comparable, and metadata records facts that similarity cannot establish: identity, version, effective dates, and access. If ingestion destroys a table’s headers or broadens a document’s permissions, a better generation prompt cannot restore the lost information.
Treat publication into the searchable corpus as an explicit decision. The content owner must recognize the parsed document, the security owner must approve how source entitlements become retrieval permissions, and the ingestion owner must detect failures before malformed or restricted material reaches Claude.
How it works
Parse the document as a source of meaning
Start with the source of record and its version, not a detached text export whose origin nobody can reconstruct. Separate obtaining bytes from extracting usable content. For scanned documents, optical character recognition (OCR) may produce text, but successful extraction does not prove that columns, footnotes, or minus signs survived.
Preserve headings, section order, list membership, table captions, column headers, and the relationships between footnotes and their targets. A rate table with values detached from their units or applicability dates becomes dangerous even when every word is present. Retain page or section locators so reviewers can compare extracted content with the original and retrieval can identify where a passage belongs.
Anthropic’s contextual retrieval article explains why isolated chunks can lose the information required to understand them. Ingestion supplies the structure that the chunking stage needs to avoid that loss; see 3.5.3. Do not ask an enrichment model to repair an unreadable table by guessing what the missing header probably meant.
Normalize representation without rewriting authority
Normalization should remove accidental representation differences while preserving source claims. Convert text encoding consistently, remove repeated navigation furniture when safe, and standardize whitespace. Keep semantically meaningful punctuation, section identifiers, units, and dates intact. A normalization that merges distinct product codes improves cosmetic uniformity while breaking exact-term retrieval.
Store the source artifact separately from its normalized representation. Record an application-defined source_hash, parser version, and normalization version in the ingestion record. These are your pipeline’s fields, not Claude API parameters. They let the ingestion owner identify whether a changed result came from a source revision or a processing revision.
Define format-specific acceptance checks. A policy parser can require a title and section hierarchy; a table extractor can verify expected headers and row counts against a reviewed fixture. Quarantine an extraction that fails its checks rather than publishing partial text as a complete version. Keeping the previous approved version available is a business decision, not an automatic substitute for a withdrawn document.
Attach metadata to every searchable unit
Use an application schema that includes document_id, version, chunk_id, section_path, source_uri, effective_from, effective_to, and the relevant tenant or audience fields. Define what each date means: ingestion time is not the date a policy takes effect. Distinguish an archived version that remains valid for historical questions from a withdrawn version that must no longer be served.
Keep stable document identity separate from chunk identity. A parser revision can change chunk boundaries without changing which source document you ingested. A unique combination of document version and chunk locator helps find every derivative record during replacement or deletion; see 3.5.4 for that lifecycle.
Map approved provenance to search_result.source and a readable name to search_result.title; see 2.2.4 for the answer interface. For charts and scanned pages whose meaning cannot be preserved in text, evaluate multimodal retrieval: Anthropic’s embedding guide describes Voyage models called with multimodal_embed() that vectorize screenshots of PDFs, slides, tables, and figures. Keep the visual source for inspection and measure whether this path retrieves the required information; do not claim a text extract covers a dropped chart.
Carry permissions through every transformation
Propagate the authorization policy from 3.2.2 through ingestion. When one document contains sections with different access rules, split on those boundaries before making independently retrievable chunks. A generated context derived from a restricted parent can disclose information even if the underlying chunk appears public.
As one concrete implementation, Elasticsearch document-level security places a Query DSL query in an index permission entry. Its documentation demonstrates a term condition on department_id. This restricts document-based read access; Elastic describes this security mechanism for read-only privileged accounts. Give the ingestion writer a separate identity rather than treating a restricted reader role as a general-purpose writing identity.
Elastic warns that document-level security does not prevent every form of aggregate information disclosure. The ingestion owner must therefore assess which derived fields expose restricted content, including indexed terms and generated context, rather than assuming correct chunk labels protect every query surface.
Key distinctions and trade-offs
Source metadata versus generated context: Metadata establishes trusted identity, dates, and access. Generated context improves findability. Do not let the second rewrite the first.
Deduplication versus authorization: Two documents with identical text can belong to different tenants or have different retention rules. Sharing their extracted representation requires preserving the separate provenance and permission records; deleting one must not silently affect the other.
Fail closed versus service continuity: Quarantining malformed input protects correctness but may leave coverage gaps. Track rejected versions and expose missing coverage to the content owner. A pipeline that reports only successful jobs can look healthy while serving obsolete guidance.
Choose metadata that the actual query and lifecycle paths need. Every field has an owner and a definition. Unused labels add complexity, while missing tenant, source version, or effective-date labels can force a complete re-ingestion after rollout.
Architecture / decision scenario
A healthcare network ingests 8,000 procedures from HTML and scanned PDFs. General staff may read clinical procedures; an incident-review appendix is limited to an investigation group. Revised procedures must appear within one hour, and the six-person platform team cannot manually inspect every daily import.
A proposed flatten-and-embed job discards headings and assigns each PDF one broad staff label. It meets the throughput target but leaks the restricted appendix and separates dosage tables from their units. A prompt asking Claude to avoid restricted passages cannot undo their inclusion in context.
The architect chooses format-specific parsers, reviewed fixtures for table extraction, and quarantine for missing units or section structure. The security owner supplies trusted section entitlements, which propagate to chunks and enrichment inputs. Normalization standardizes encoding and whitespace while preserving dosage units, and each revision carries its source-defined effective_from, separately from ingestion time. The platform team records parser versions and source hashes; the clinical content owner reviews quarantined changes. Retrieval uses a restricted read identity and approved query interface. An hourly freshness report counts accepted, rejected, and pending source versions, so a failed parser cannot masquerade as a successful update.
Common exam traps
- Treating successful text extraction as proof that structure and numeric meaning survived.
- Using ingestion time as the policy’s effective date.
- Applying one permission label to mixed-access sections.
- Deriving authorization from a model-generated summary.
- Deduplicating identical text while discarding separate tenant ownership.
What to remember
Preserve source meaning, provenance, and trusted permissions before indexing. Normalize representation without changing claims, make rejected versions visible, and give every derivative unit enough identity to update or remove it. Authorization applies to enrichment inputs as well as to retrieved outputs.
Knowledge check
A manufacturer ingests its HR handbook as one PDF. General staff may read every chapter, but one appendix describes open investigations and is limited to the investigations group. The pipeline labels every chunk of the PDF for all staff, and Claude generates each chunk’s contextual prefix from the whole document. General staff must keep access to the rest of the handbook, and the investigations group must still be able to search the appendix. What should the ingestion owner change?
A bank’s ingestion job sets each policy version’s effective_from field to the time the job ran. A new rate policy is approved on 1 March, ingested on 2 March, and takes effect on 1 April. Staff asking in mid-March about the current rate receive the April rate. Staff must also be able to ask about rates already announced for later dates. What should the ingestion owner change?
effective_from from the effective date the source statesA healthcare network re-ingests a revised dosage procedure from a scanned PDF. Extraction reports success, but review finds that the dosage table’s column headers and units are missing. Revised procedures must appear within one hour, and the clinical content owner reviews rejected imports. What should the pipeline do with this version?
References
- Anthropic, Introducing Contextual Retrieval
- Anthropic, Search results
- Anthropic, Embeddings
- Elastic, Document and field level security