AWSAWS Certified Generative AI Developer - Professional2026-103.3 Implement AI governance and compliance mechanisms
Carry source lineage into answer evidence
Completed by the knowledge check
Learning objective
Catalog source lineage with AWS Glue Data Catalog and tag evidence for response attribution.
Core idea
Source lineage and response attribution answer related but different questions. Lineage explains where an asset originated and what transformations produced it. Attribution connects a generated answer to the evidence used for that particular response.
AWS Glue Data Catalog is a metadata repository for data locations, schemas, and properties, organized into databases and tables. Use it to identify and describe source assets. Then preserve explicit lineage records through extraction, transformation, chunking, and retrieval. A catalog entry alone does not identify the exact evidence behind every generated sentence.
For response attribution, Amazon Bedrock Knowledge Bases exposes source metadata and citation structures. Connect those fields to the cataloged asset and immutable source revision. A useful evidence chain lets an investigator move from an answer to its cited passage, from that passage to the derived artifact, and from the artifact to the original source.
How it works
Catalog the asset and its revision context
Glue crawlers discover data and populate catalog metadata. The Data Catalog stores location, schema, and source properties. It also provides lineage information about transformations and operations. Treat that catalog as part of your evidence system, with a clear contract for what your pipeline records.
The Glue Table structure includes DatabaseName, Name, StorageDescriptor, Parameters, and VersionId. StorageDescriptor.Location describes physical storage. Parameters is a string-to-string map for properties associated with the table. VersionId identifies a catalog table version.
Record the catalog database and table identity alongside each released dataset or document collection. Use table parameters for appropriate asset-level properties, such as an owner identifier or a reference to the transformation manifest. These property names are organization-defined, not reserved Glue fields. Do not store secrets or unnecessary personal data in discoverable metadata.
A table version is a version of metadata, not automatically the version of every source object or row it describes. A table location can identify a collection that changes independently. Preserve the source revision separately so “the table existed” cannot be mistaken for “this answer used this exact source content.”
Preserve transformation lineage explicitly
Define a lineage record for each derived artifact. Suggested application fields include document_id, content_revision, source_uri, source_version_id, catalog_table, catalog_version, transformation_run_id, and transformation_revision. These are a proposed evidence contract, not an AWS-generated schema.
Tie the record to the immutable input and output of a transformation. If a pipeline extracts a PDF into text, preserve the original document identity and the extraction run. If another stage normalizes tables, record that stage separately. A page number or chunk offset is meaningful only with the revision of the content to which it refers.
See 1.3.5 for multimodal preprocessing provenance and 1.4.5 for document metadata design. Here the additional concern is the join from those processing records to a cataloged source and then to an answer. Avoid rebuilding a second, inconsistent identity scheme for governance.
Validate required lineage attributes before publishing evidence to retrieval. If the pipeline cannot resolve an output to an input revision, quarantine that output in your application workflow. A plausible passage without a resolvable source may be useful for exploration, but it does not meet a traceable-response contract.
Project evidence attributes into retrieval
For an S3 data source, Bedrock supports a sidecar .metadata.json file for a document. Put the evidence attributes needed at retrieval into the metadata projection; preserve the fuller lineage record separately. AWS also documents document-level metadataAttributes in CSV metadata configuration, applied to the resulting chunks.
Project a stable document identity and content revision, plus a reference to the lineage record. Do not assume every source attribute belongs in the embedding or that copying metadata creates an authorization decision. Metadata selection and access-filtered retrieval are separate concerns; see 1.4.6.
“Tag evidence” here means assigning provenance attributes that survive retrieval. Do not confuse these attributes with AWS resource tags on a bucket, a knowledge base, or a catalog resource. Resource tags classify infrastructure; they do not automatically become cited-document metadata in a response.
Test the projection with representative files and transformed content. Inspect the returned attributes, not only the ingestion result. Successful ingestion proves that processing completed under its contract; it does not prove that every field needed for attribution reached the response.
Preserve the answer-to-source mapping
The Bedrock Citation structure used in RetrieveAndGenerate and InvokeAgent responses contains generatedResponsePart and retrievedReferences. Each RetrievedReference can contain content, location, and metadata. Preserve these returned structures when they are present, rather than reducing a citation to a display URL.
The source location tells you where evidence came from. Metadata lets your application resolve its stable identity and revision. The cited content provides the passage associated with the answer segment. Together they support attribution; none alone proves that the generated claim is correct.
If you use an application-owned retrieval-and-generation path, record the selected evidence and the resulting attribution mapping yourself. Do not assume an arbitrary model invocation produces the Knowledge Bases citation contract. Preserve the evidence actually admitted to the prompt, not only a list of search candidates.
Validate that each required citation resolves to the retained source revision. If the location now points to changed content, use your revision record to retrieve the original evidence. See 3.1.6 for verifying that answer claims are supported by the cited passage.
Key distinctions and trade-offs
Catalog metadata versus document evidence: A Glue table describes an asset or collection. Response evidence identifies the particular passage and revision used.
Catalog version versus content revision: Table.VersionId versions catalog metadata. Preserve the underlying source revision independently.
Resource tags versus retrieval metadata: They are different metadata surfaces. Your ingestion pipeline must explicitly project evidence attributes.
Lineage versus factual support: A complete transformation history explains provenance. It does not establish that a generated conclusion follows from the source.
AWS documents a quota on document metadata fields/attributes per chunk. Keep the projection within that constraint by storing the identifiers needed to resolve evidence and a reference to the fuller lineage record, rather than copying the entire lineage graph into every chunk. The referenced records must remain available for the investigation period.
Architecture / decision scenario
A warranty assistant answers from product manuals and a structured claims dataset. Glue Data Catalog identifies the claims table, its storage location, schema, and table version. The publication pipeline records the dataset revision and transformation run separately.
The manual pipeline assigns each document a stable identity and content revision, then carries those attributes into S3 sidecar metadata. Both pipelines retain records that resolve derived passages or records to their original source revision.
For each generated response, the application preserves returned citation content, locations, and metadata. An investigator can identify which manual edition and claims extract supported the answer. If a citation lacks the required revision attribute, the application withholds that answer from the workflow requiring complete attribution.
A subsequent crawler update changes catalog metadata, and a manual is replaced at the same location. Neither change rewrites the retained evidence for earlier answers. Investigation follows the stored revisions rather than reading the source location as it exists today.
Common exam traps
- Treating a crawler or catalog table as a complete answer-level lineage record.
- Equating
Table.VersionIdwith an S3 object or dataset content revision. - Expecting AWS resource tags to appear automatically in response citations.
- Preserving a source URL while discarding cited content and metadata.
- Recording all retrieved candidates as if the model used all of them.
- Treating traceable evidence as proof of factual correctness.
What to remember
Catalog the source, preserve transformation and content revisions, project evidence attributes into retrieval, and retain the answer-to-source mapping. Attribution remains useful only when it resolves to the evidence that existed at response time.
Knowledge check
A warranty assistant’s evidence pipeline stores, for each answer, the Glue database, table name, and Table.VersionId of the manuals collection. Manuals are replaced at the same S3 location. An investigator must identify the manual edition behind a disputed answer. What is missing?
StorageDescriptor.Location for the manuals collectionTable.VersionId versions catalog metadata, not the objects at the table’s location, which can change independently. Only a separately recorded content revision, such as the object version and document revision, ties an answer to the exact edition. The location and owner describe the collection as it is now.Knowledge base citations must carry each manual’s document ID and revision. The team adds those values as AWS resource tags on the source bucket, but retrieved references show neither. What should the team do?
.metadata.json sidecar fileAn application stores only the display URL of each citation returned by RetrieveAndGenerate. Months later, the manual at that URL has been revised. Which retained record would let an investigator reconstruct the answer’s evidence?
generatedResponsePart with its references’ content, location, and metadataA team replaces RetrieveAndGenerate with its own retrieval-and-generation code that reranks 20 candidates and puts the top 5 into the prompt. What must the application record for attribution?
An answer cites a passage whose source revision and transformation run both resolve correctly, yet the answer misstates an exception that the passage makes. Which check catches this?