AWSAWS Certified Generative AI Developer - Professional2026-101.4 Design and implement vector store solutions
Separate ingestion resources from document storage
Completed by the knowledge check
Learning objective
Configure Amazon Bedrock Knowledge Bases ingestion and data-source hierarchy while separating source documents from the configured vector store.
Core idea
In a self-managed vector knowledge base, the knowledge base defines the embedding and storage destination. A data source belongs to that knowledge base and defines which source content to ingest and how to process it. An ingestion job synchronizes that data source with the destination. These resources have different identifiers and lifecycles.
An S3 bucket containing PDFs is not the configured vector index. The ingestion process reads documents, parses and chunks them, generates embeddings, and writes searchable records to a vector store. The source remains the document authority; the index is a derived representation. Mistaking those boundaries leads to configuration errors and stale retrieval after source changes.
AWS also documents managed knowledge bases, created with managedKnowledgeBaseConfiguration, that need no vector-store configuration. This lesson addresses the self-managed vector-store path named in the objective, rather than applying its storage requirements to every Knowledge Bases mode.
How it works
Define the destination at the knowledge-base level
CreateKnowledgeBase defines knowledgeBaseConfiguration, the service roleArn, and, for the self-managed path, storageConfiguration. A vector knowledge base specifies its embeddingModelArn within vectorKnowledgeBaseConfiguration. The storage configuration identifies the backend and its fields; the source bucket does not belong in place of those backend settings.
For example, opensearchServerlessConfiguration names a collectionArn, vectorIndexName, and fieldMapping. The mapping includes vectorField, textField, and metadataField. An Aurora destination instead uses rdsConfiguration with its database, table, credentials secret, and mappings. See 1.4.1 for backend eligibility and prerequisites.
Plan the vector dimensionality together with the embedding model. The destination must be able to store what the model produces. Separately check that the service role can read the source and write the destination. Successful resource creation is not proof that document ingestion can cross both permission boundaries.
Attach the source and its processing policy
After creating the knowledge base, call CreateDataSource with knowledgeBaseId. The same knowledge base → data source relationship also supports connector types such as Confluence, Microsoft SharePoint, Microsoft OneDrive, and Web Crawler; a data source is not synonymous with an S3 prefix. For an S3 source, dataSourceConfiguration.type is S3, and s3Configuration identifies bucketArn, optional bucket-owner information, and inclusionPrefixes. The source bucket must be in the same Region as the knowledge base. An inclusion prefix selects source objects; it does not create a vector namespace or a tenant authorization policy.
vectorIngestionConfiguration belongs to the data-source configuration. It controls processing such as chunkingConfiguration. For fixed-size text chunking, chunkingStrategy: FIXED_SIZE works with maxTokens and overlapPercentage. AWS states that you cannot change the chunking strategy after connecting the data source. Treat a strategy change as a planned source reconfiguration, not a query-time setting.
Distinguish the resource hierarchy from hierarchical chunking. The resource hierarchy is knowledge base → data source → ingestion job. Hierarchical text chunking creates parent and child chunks within documents. Retrieval initially finds child chunks and replaces them with broader parents, so the number of returned results can be smaller than requested. Parent chunks do not become independent data sources.
Choose chunking from the evidence unit the reader needs. A definition might fit a small chunk; a procedure might require its prerequisites and exception together. Evaluate returned context rather than accepting a token count because it is convenient. Keep extraction quality gates separate; see 1.3.1 and 1.3.5.
Synchronize and inspect the job
StartIngestionJob requires both knowledgeBaseId and dataSourceId. Its response returns an ingestionJobId. Use all three identifiers with GetIngestionJob to track the specific run. Do not interpret the initial API response as a completed or queryable index.
On completion, the job reports status: COMPLETE. Inspect statistics, including numberOfDocumentsFailed, numberOfNewDocumentsIndexed, numberOfModifiedDocumentsIndexed, and numberOfDocumentsDeleted, rather than checking the status alone. failureReasons provides additional failure information. Reconcile these outputs against the expected source change before declaring the corpus ready.
S3 synchronization is incremental: it processes additions, modifications, and deletions since the previous sync. Source changes therefore need a synchronization step; uploading a document does not itself establish that retrieval uses it. AWS also documents metadata-only optimization when only metadata files change, content is not CSV, and no custom transformation Lambda is configured. Do not promise that every metadata edit avoids re-embedding.
AWS notes that newly ingested embeddings can take a few minutes to become available for querying in stores other than Aurora. A practical release gate therefore combines job inspection with a retrieval probe for the changed documents. Keep the previous accepted corpus available until the new evidence passes the application’s readiness checks.
A started job becomes a ready corpus only after job inspection and a retrieval probe both pass.
Define deletion deliberately
CreateDataSource.dataDeletionPolicy distinguishes DELETE and RETAIN. The setting controls the ingested vector data when a knowledge-base or data-source resource is deleted. Neither value deletes the vector store itself. This is separate from synchronizing a document deletion from the source.
Record which resource owns cleanup. Deleting a connector, deleting an S3 object, and deleting a vector collection are different actions. A retention decision must include the derived text and metadata as well as the embeddings; otherwise an application can retire its source while leaving searchable evidence behind.
Key distinctions and trade-offs
Use the following boundaries when reviewing configuration:
| Resource | Owns | Does not establish |
|---|---|---|
| Knowledge base | Embedding model and configured destination | Which S3 documents have been successfully synchronized |
| Data source | Source connection, inclusion scope, ingestion policy | An independent vector database or permission boundary |
| Ingestion job | A synchronization attempt and its results | Immediate availability of every changed chunk |
| Source document | Original content | A searchable embedding without successful ingestion |
Multiple source connections can separate ingestion responsibilities within the same knowledge base, but they still target its configured storage path. Do not use connector separation as a substitute for retrieval authorization; see 1.4.6.
Architecture / decision scenario
A policy assistant reads approved PDFs under an S3 prefix. Its knowledge base targets an OpenSearch Serverless index. The team configures the collection, index fields, and embedding model at the knowledge-base level, then attaches the S3 prefix through CreateDataSource with a reviewed text-chunking policy.
After publishing a replacement policy and deleting its superseded file, the release process calls StartIngestionJob. It checks the indexed and deleted document counts with GetIngestionJob, investigates any failed documents, and probes retrieval for the replacement. The release stays pending if the job finishes but the new policy is not yet returned.
Later, the team retires the connector with RETAIN for an approved recovery exercise. It records that the vectors remain and that the OpenSearch collection still exists. Connector retirement is not reported as corpus erasure.
Common exam traps
- Configuring the S3 source bucket as though it were the vector-store destination.
- Treating data-source hierarchy and parent-child chunking as the same concept.
- Calling an ingestion API without the data-source identifier.
- Assuming
COMPLETEproves zero document failures or immediate query visibility. - Expecting source edits to appear without synchronization.
- Assuming
DELETEremoves the vector-store infrastructure.
What to remember
- Configure storage on the knowledge base and ingestion on its data source.
- Track each sync with its three resource identifiers.
- Gate releases on job statistics and retrieval evidence.
- Separate source deletion, connector deletion policy, and store cleanup.
- Keep authorization independent of ingestion scope.
Knowledge check
A team builds a self-managed vector knowledge base over approved PDFs in an S3 bucket, with an OpenSearch Serverless index as the store. Where does each part of the configuration belong?
storageConfiguration , the index in the data sourcestorageConfiguration , the bucket in the data sourcestorageConfigurations3ConfigurationCreateDataSource owns the source connection — the bucket and its inclusion prefixes — and the ingestion policy. The source bucket is the document authority, never the vector store.An editor uploads a replacement policy PDF under the knowledge base’s S3 inclusion prefix. An hour later, retrieval still returns only the old policy. What should the release process do?
StartIngestionJob takes both the knowledge base and data source identifiers and returns an ingestionJobId , which GetIngestionJob uses to follow that specific run. Sync is incremental, so only the changed documents are processed.After a policy update, an ingestion job reports status: COMPLETE. What should gate declaring the updated corpus ready?
COMPLETE status, because it means that no document failedStartIngestionJob response, plus the uploaded file countCOMPLETE status, plus the knowledge base’s configured field namesCOMPLETE does not mean zero failures: statistics reports failed, new, modified and deleted documents, which should match the intended change. Embeddings can also take a few minutes to become queryable in stores other than Aurora, so a retrieval probe confirms the new evidence is actually returned.A data source uses hierarchical chunking. A query asks for 10 results, and only 6 come back, although more than 10 child chunks matched. What explains this?
A team retires an S3 connector by deleting its data source, with dataDeletionPolicy set to RETAIN. What is the state afterwards?
RETAIN keeps the data converted into vector embeddings when the data source is deleted, and neither RETAIN nor DELETE deletes the vector store itself. Source documents are a separate authority with their own cleanup, so retiring a connector is not corpus erasure.