Keep metadata authority separate from vectors

Learning objective

Combine Amazon RDS or DynamoDB metadata with Amazon S3 documents and a separate vector index.

Core idea

A document retrieval system often needs three representations of the same business object. S3 holds the document, RDS or DynamoDB holds authoritative business metadata, and the vector index holds a derived search representation. Connect them through stable identifiers and explicit versions rather than trying to make any one representation serve every responsibility.

The database can decide that a document is approved, expired, or inaccessible. The index can find semantically relevant chunks. Those are different decisions. A high similarity score cannot override the database’s document state, and a successful database write does not prove that the corresponding search projection has caught up.

This is an application architecture, not an automatic three-service join provided by Knowledge Bases. Define who assembles the projection and who verifies candidates before generation. See 1.4.5 for the provenance schema and 1.4.6 for permission enforcement.

How it works

Establish the metadata authority

Choose RDS when your metadata access pattern needs relational structure and application-managed SQL queries. For example, document records can refer to policy owners, approval records, and entitlement tables. Aurora PostgreSQL’s documented Knowledge Bases setup uses a dedicated schema, role, and chunk table, illustrating that retrieval storage has its own database objects. Do not confuse those objects with your application’s business records.

Choose DynamoDB when the application can address metadata through designed item keys. A document item might use the tenant and immutable document identifier as its key and carry the active content revision, S3 location, publication state, and permission revision. Design that key from the lookup the application actually performs after retrieval, rather than storing metadata that requires an unbounded scan to find each candidate.

Either choice can remain separate from the vector index. The distinction is logical: even if an Aurora cluster stores both business tables and a vector table, the index remains a derived projection with its own update and retrieval behavior. Here, a separate OpenSearch index makes the physical boundary explicit.

Define application fields such as document_id, content_revision, acl_revision, and index_state as your own contract. They are not reserved AWS parameters. Keep metadata authority behind the application API so an ingestion worker cannot declare a document approved merely by writing a searchable chunk.

Bind the document to a specific source revision

S3 Versioning preserves multiple versions of an object. In a versioning-enabled bucket, overwriting a document creates another version; a normal deletion inserts a delete marker rather than permanently erasing previous versions. Record the intended version ID with the document’s metadata when your architecture needs reproducible evidence.

A bucket and object key alone are insufficient for that purpose. If the key is overwritten, the same location can refer to different content. Your custom ingestion worker should process the designated revision and attach the corresponding revision identity to every derived chunk. Treat immutable revision keys as another possible application convention, with a separate pointer to the active revision.

Do not assume a Knowledge Bases S3 connector selects the historical S3 version recorded in an unrelated database. Its source configuration identifies a bucket and inclusion prefix; see 1.4.2. The application owns publishing database-derived metadata and synchronizing that projection.

Publish a searchable projection

Build each vector record from an accepted source revision and an accepted metadata snapshot. Include the document identifier, content revision, chunk identifier, tenant, source location, and the fields needed for retrieval filtering. Keep the authoritative record available for checks that cannot safely rely on the projection.

For a Knowledge Bases S3 source, publish the database snapshot as a .metadata.json sidecar; see 1.4.5. For a custom OpenSearch path, the ingestion application writes those fields into its own index mapping. These are two implementations of the same projection responsibility, not two mandatory ingestion steps.

DynamoDB Streams provides a concrete change signal for the DynamoDB option. It captures item-level modifications in order for each item and retains stream records for up to 24 hours. StreamViewType selects KEYS_ONLY, NEW_IMAGE, OLD_IMAGE, or NEW_AND_OLD_IMAGES for the change payload. A consumer can use those records to schedule projection updates. The stream itself does not write the separate vector index.

For an OpenSearch target, DynamoDB zero-ETL integration with Amazon OpenSearch Service provides a managed alternative: an OpenSearch Ingestion pipeline with a DynamoDB source streams creates, updates, and deletes to a domain or Serverless collection. It requires Streams with NEW_IMAGE or NEW_AND_OLD_IMAGES; an initial full snapshot also requires point-in-time recovery (PITR). This projects table records, not the S3 document body: the application still supplies accepted document text to the embedding path described in 1.4.3.

For the RDS option, require the publishing application to create a durable indexing work record in the same database transaction as the metadata change. This outbox is an application design choice, not an automatic Knowledge Bases feature. An ingestion worker processes the pending records and marks progress only after the intended projection is verified.

Reconcile failures and query safely

A separate index introduces a period in which the metadata authority and search results can disagree. Make that state visible. Record the revision projected successfully, compare it with the active revision, and preserve pending work for retry. A recovery process should find approved records whose index revision is absent or behind, rather than depending solely on an event that might no longer be retained.

Do not describe DynamoDB Streams’ record ordering as a cross-service transaction. Its 24-hour retention also means it cannot be your only repair history. Retain enough authoritative state to rebuild the index from S3 documents and database metadata.

At query time, retrieve candidates with the required tenant and metadata scope, then resolve their document identifiers against authoritative metadata. With DynamoDB, GetItem with ConsistentRead: true reads the most up-to-date table item reflecting prior successful writes. Strongly consistent reads are not supported on global secondary indexes. This matters when a recently revoked document might still exist in the search projection.

Reject mismatched, inactive, or unauthorized revisions before their text enters the generation request. A database read is a point-in-time check, not a lock on future permission changes; define the access semantics your application promises. For RDS, perform the equivalent authoritative lookup through the application’s reviewed SQL path.

Key distinctions and trade-offs

The three stores have different jobs:

Copying filter fields into the index improves candidate selection but creates synchronization work. Checking authority after retrieval adds a database lookup but closes a stale-projection gap. Use both when the application’s access requirements justify them, and measure how selective filtering changes the available authorized evidence.

Architecture / decision scenario

A contract assistant stores PDFs in versioned S3. One organization uses RDS for contracts and approval relationships; another uses DynamoDB items keyed by tenant and contract identifier. Both use a separate vector index for clause search.

The RDS publisher writes pending indexing work alongside the approval change. The DynamoDB implementation uses OpenSearch Ingestion with a PITR snapshot and Streams for the metadata projection, then reconciles active document revisions against projected revisions. Its document worker supplies the corresponding S3 text to the embedding pipeline. Each worker indexes the accepted content and metadata snapshot, then records the projected revision.

A contract is revoked before index cleanup finishes. Retrieval still finds its chunks, but the application checks the RDS approval row or strongly consistent DynamoDB table item and excludes them before generation. Rebuilding the index later uses the source revision and authoritative metadata, not whatever happened to survive in search.

Common exam traps

• Treating an S3 upload, metadata write, and vector update as one atomic transaction.
• Assuming a vector score establishes approval or access rights.
• Using only an object key when evidence must identify a content revision.
• Expecting Knowledge Bases to join external metadata automatically.
• Assuming a DynamoDB global secondary index supports strongly consistent reads.
• Depending only on a 24-hour stream to repair indefinite ingestion outages.

What to remember

• Use stable document identifiers across all three representations.
• Bind chunks to accepted content and metadata revisions.
• Publish the search projection through explicit, recoverable work.
• Recheck authoritative state before using sensitive candidates.
• Keep the vector index rebuildable from the source and metadata authority.

Knowledge check

References

• AWS, Retaining multiple versions of objects with S3 Versioning
• AWS, Using Aurora PostgreSQL as a Knowledge Base for Amazon Bedrock
• AWS, Change data capture for DynamoDB Streams
• AWS, DynamoDB read consistency
• AWS, Using an OpenSearch Ingestion pipeline with Amazon DynamoDB