AWSAWS Certified Generative AI Developer - Professional2026-101.4 Design and implement vector store solutions
Design provenance that survives document chunking
Completed by the knowledge check
Learning objective
Design provenance metadata for authorship, timestamps, domain, tenant, and document access permissions.
Core idea
Provenance connects retrieved evidence to the document and governance decision that produced it. A useful schema answers who authored the source, which revision you indexed, when it was valid, which domain it covers, which tenant owns it, and which access policy applies. Those answers must survive chunking.
A source URL alone does not establish all of that information. Neither does an author’s name embedded in prose. Preserve structured, typed attributes alongside the chunks, and distinguish authoritative metadata from the copy stored in the vector index. See 1.4.4 for the storage and consistency architecture; this lesson defines the contract that architecture carries.
Metadata supports traceability and candidate selection. It does not grant access by itself. The application enforces the access decision described in 1.4.6 before evidence enters a foundation model (FM) request.
How it works
Define identity before descriptive fields
Give each document an immutable application identifier. Pair it with a content revision and a source locator. For versioned S3 content, retain the intended version ID in the authoritative record. See 1.4.4 for source-version behavior. A locator without a revision can point to different evidence after a source update.
Make every derived chunk carry the document and revision identities. A chunk identifier distinguishes separate evidence units from the same revision. If the parser or chunking policy changes, the chunk set can change while the document identity remains stable. Record the processing release in your ingestion records so you can explain which transformation produced a disputed passage.
Use explicit field meanings rather than overloading one value. The following are example application fields, not AWS-reserved metadata keys:
| Field | Meaning | Design choice |
|---|---|---|
document_id, content_revision | Stable source identity and accepted revision | Preserve through every derived chunk. |
author_id | Author or owning organization identity | Use a controlled identifier; keep a display name separately if needed. |
created_at, updated_at | Source lifecycle times | Define the event and time convention for each. |
effective_at, expires_at | Business validity interval | Do not substitute ingestion time for policy validity. |
domain | Controlled subject category | Use a vocabulary the filter can match exactly. |
tenant_id | Owning tenant | Derive from trusted publication context. |
acl_policy_id, acl_revision | Access-policy identity and revision | Resolve against authoritative permissions. |
When authorship is unknown, represent that uncertainty through your publication workflow. Do not silently attribute a generated summary to the author of one source when the summary combines several sources. Preserve the source relationships in the authoritative record and attach the retrieval fields that the backend can support.
Use types that support the intended comparisons
Amazon Bedrock documents metadata attribute types STRING, NUMBER, BOOLEAN, and STRING_LIST. Match the type to the operation. A numeric epoch timestamp supports a range filter such as greaterThanOrEquals; a date rendered for display is a different concern. State whether time values use seconds or milliseconds and keep that convention across producers and query builders.
Separate document creation, source modification, ingestion, and policy-effective time. A document indexed today might describe a policy that expired last year. If the application asks for policy valid at a particular time, filtering on ingestion time selects the wrong evidence even when the filter is syntactically valid.
For domain and tenant, use exact canonical values. A broad category such as finance is useful only if ingestion and retrieval agree on its meaning. Do not derive tenant ownership from model interpretation of the document title. Obtain it from the trusted publisher or metadata authority, then validate it before indexing.
Backend capabilities constrain the schema. Neptune Analytics GraphRAG metadata supports string, number, and boolean properties, but not list-valued document metadata. Its in filter can match a scalar property against a list of candidate values; that is different from storing a STRING_LIST property. With Bedrock Retrieve and RetrieveAndGenerate, only string candidates are supported in in and notIn lists. Keep acl_policy_id a string pointing to the authoritative membership record instead of copying a list of principals into each chunk.
Project the schema into Knowledge Bases
For an S3 data source, create a sidecar whose name is the document’s complete filename with .metadata.json appended. Store it in the same folder as the document. AWS limits that file to 10 KB. The simplified metadataAttributes form stores values for filtering without including them in the embedding.
For example, a sidecar for policy.pdf can contain:
{
"metadataAttributes": {
"document_id": "policy-42",
"content_revision": "rev-7",
"author_id": "policy-office",
"effective_at": 1791586800,
"domain": "procurement",
"tenant_id": "tenant-a",
"acl_policy_id": "policy-readers",
"acl_revision": 3
}
}These values illustrate an application contract; they do not define AWS authorization. The timestamp uses epoch seconds. An ingestion validator should reject missing tenant, revision, or policy information required by your publication rules before the document becomes searchable.
The extended sidecar format supports includeForEmbedding. When true, the attribute’s key-value pair is concatenated to the chunk text before embedding; when false, it remains available for filtering without influencing the vector. Use this deliberately for retrieval-relevant context. Tenant and access-policy fields should remain filtering attributes rather than relying on similarity to implement their meaning.
CSV metadata needs a separate type review. AWS states that metadata values taken from CSV columns are stored as strings, even when the general metadata schema supports numbers. A numeric range policy must not assume that a CSV-derived timestamp has the same type as the sidecar’s numeric attribute.
Verify the retrieved provenance
The Retrieve response includes retrievalResults with content, location, metadata, and score. Check that returned chunks carry the identifiers and fields your application needs for attribution and policy lookup. A relevance score is not a provenance validation result.
Use ingestion fixtures that cover multiple authors, revisions, domains, tenants, and policy changes. Verify that each retrieved chunk resolves to its accepted source revision. Reject a candidate if its required identity is missing rather than manufacturing a citation from the nearest-looking filename.
Keep permission changes traceable through acl_revision. When that revision changes in the metadata authority, the indexed copy can lag. Comparing revisions gives the application a concrete reason to recheck or exclude evidence. It also helps distinguish a stale permission projection from poor retrieval quality.
Key distinctions and trade-offs
Authorship versus ingestion actor: The person or organization that created the source is not necessarily the worker that uploaded or parsed it. Preserve both in their appropriate records.
Validity versus freshness: Fresh ingestion does not imply a document is business-valid. Use the time attribute that matches the question.
Policy identifier versus permission grant: A policy label supports lookup and filtering. Authorization depends on the caller and authoritative policy, not possession of the label.
Rich provenance increases auditability but also raises metadata size and maintenance costs. Copy only the fields needed for retrieval and attribution; retain detailed histories and large permission memberships in the metadata authority.
Architecture / decision scenario
A procurement assistant serves two tenants and answers both present-day and historical policy questions. Documents have different authors, overlapping effective intervals, and changing access policies.
The publisher assigns document revisions, canonical tenant and domain values, numeric validity timestamps, and policy revisions before ingestion. The assistant reads returned provenance, checks the accepted revision, and uses the requested date against business-validity fields rather than ingestion time.
When evaluating Neptune GraphRAG, the team replaces a proposed list of allowed principals with a scalar policy identifier and authoritative membership lookup. For a CSV import, it validates timestamp types before enabling numeric filters. Every returned passage must identify its source revision and pass the access check before generation.
Common exam traps
- Treating a citation URL as a complete provenance record.
- Using ingestion time as the policy’s effective date.
- Deriving tenant identity from model-generated metadata.
- Assuming every backend supports list-valued metadata.
- Assuming numeric-looking CSV metadata is a numeric attribute.
- Treating
includeForEmbeddingas an authorization setting.
What to remember
- Preserve identity, revision, authorship, time, domain, tenant, and policy meaning.
- Define typed fields and shared value conventions before ingestion.
- Keep large histories and access memberships in an authoritative store.
- Validate provenance in returned chunks, not only in source files.
- Metadata describes access; trusted application logic enforces it.
Knowledge check
A procurement policy was ingested this morning, but it expired last year. A user asks which rule applies today. Which metadata should the retrieval filter compare with today’s date?
updated_at time recorded when the source file was last editedeffective_at and expires_at interval of the policy itselfcreated_at time recorded when the document was first writtenChunks from three revisions of a contract share one S3 URL. A reviewer must identify the exact passage behind a disputed answer. Which fields should every chunk carry?
A publisher sets includeForEmbedding: true on tenant_id in each .metadata.json sidecar, expecting it to keep tenants’ results apart. How should the tenant field be used instead?
includeForEmbedding only concatenates the attribute to the chunk text before embedding, which shifts similarity; it never enforces scope. Tenant identity comes from the trusted publisher and is applied as an exact filter, with the application’s access check deciding what reaches the model.A team imports CSV records into a knowledge base and maps a published_at column of epoch seconds as metadata. Its query builder applies greaterThanOrEquals with a number. What should the team check first?
NUMBER . A numeric range filter cannot assume a CSV-derived timestamp behaves like a numeric sidecar attribute; the sidecar format supports the full set of types.A team evaluating Neptune Analytics GraphRAG proposes storing each document’s allowed principals as a list attribute on its chunks. How should the access-policy field be modeled?
STRING_LIST of principals, matched with an in filteracl_policy_id , resolved against an authoritative membership recordin filterin filter matches a scalar property against candidate values. A scalar policy identifier pointing at authoritative membership keeps large, changing permission lists out of every chunk.