AWSAWS Certified Generative AI Developer - Professional2026-101.4 Design and implement vector store solutions
Detect changes before scheduling index work
Completed by the knowledge check
Learning objective
Detect source changes using DynamoDB Streams for DynamoDB records and scheduled refresh for other repositories.
Core idea
Change detection identifies which source records require work. It does not prove that extraction, embedding, indexing, or deletion has completed. Use DynamoDB Streams for item-level changes in DynamoDB tables, and use scheduled repository refresh where the source does not supply the change stream your architecture consumes. Both paths need a durable source identity and a measurable freshness boundary.
The system should distinguish “change observed,” “index work accepted,” and “revision retrievable.” Otherwise, a healthy detector can conceal a stalled downstream worker. This lesson owns the detection and triggering contract; 1.4.11 owns reconciliation when revisions, deletions, and ingestion failures leave the vector projection inconsistent.
How it works
Capture DynamoDB item changes
DynamoDB Streams captures a time-ordered sequence of item-level table modifications and retains the records for up to 24 hours. Stream records include the primary keys of changed items. Configure StreamViewType to choose the additional evidence: KEYS_ONLY, NEW_IMAGE, OLD_IMAGE, or NEW_AND_OLD_IMAGES.
Choose the view from the decision the consumer must make. KEYS_ONLY identifies the item but requires another source of content or revision evidence. NEW_IMAGE carries the post-change item. OLD_IMAGE carries the preceding item. NEW_AND_OLD_IMAGES lets a consumer compare relevant fields without treating every metadata edit as a content change. The image configuration determines available evidence; it does not define which application fields deserve re-embedding.
Use an application revision or content fingerprint to identify meaningful updates. For example, a change to a display counter might require no retrieval work, while a change to approved explanatory text does. This classification is your domain rule. Preserve separate access-policy revisions where permission changes need immediate handling; see 1.4.5 and 1.4.6.
DynamoDB documents that each record appears exactly once in the stream and that changes to an individual item appear in modification order. That statement does not imply exactly-once side effects in a consumer. Stream storage and event processing have different delivery contracts.
Consume the stream through Lambda
A Lambda event source mapping polls DynamoDB stream shards and invokes a function with batches. AWS states that event source mappings process events at least once, so duplicate processing can occur. Make the change-to-work conversion idempotent: the same source identifier and revision should not create distinct logical index updates merely because Lambda retries the batch.
Use the record’s item key and your revision as the application work identity. Preserve dynamodb.SequenceNumber for processing diagnostics, but do not invent a global ordering across unrelated items. DynamoDB Streams guarantees ordering per item. A distributed consumer must still reject an obsolete revision if asynchronous downstream work completes out of order.
Configure partial batch reporting through FunctionResponseTypes with ReportBatchItemFailures. The function returns batchItemFailures entries whose itemIdentifier is the failed record’s sequence number. Lambda uses the lowest reported sequence number as the retry checkpoint. Lambda then retries all records starting from that checkpoint, so records that succeeded after it run again and the idempotency rule remains necessary. Returning that response without enabling the feature does not activate partial success handling.
Monitor Lambda’s IteratorAge to detect a consumer falling behind. AWS defines it as the age of the last record in the event, in milliseconds: the time between the stream receiving the record and the event source mapping sending the event to the function. AWS documents ParallelizationFactor from 1 to 10 for concurrent batches per shard while preserving per-item ordering. Increasing it can help throughput, but do not use it as a substitute for bounding downstream embedding capacity. Compare processing lag against the 24-hour stream retention boundary and keep a source-inventory recovery path for gaps beyond retention.
Refresh repositories on a schedule
A repository without this DynamoDB stream requires another documented change-discovery mechanism. For a connected Knowledge Bases source, invoke StartIngestionJob with knowledgeBaseId and dataSourceId. The sync is incremental, as 1.4.2 describes. Poll GetIngestionJob with the returned ingestionJobId to distinguish an accepted trigger from a finished job.
Prefer a native schedule when one exists and its granularity meets the freshness requirement. A managed knowledge base connector’s syncSchedule runs daily, weekly, or monthly; omitting it leaves the source on demand. An existing Kendra data source takes a cron Schedule (see 1.4.9). A customer-managed knowledge base data source, such as an S3 source, has no schedule field in CreateDataSource, so something else must call StartIngestionJob. Build your own trigger in that case, or when the requirement is finer than a day.
Amazon EventBridge Scheduler is that trigger. It can invoke a Lambda refresh worker through a universal target. Its target configuration contains RoleArn, an Arn in the form arn:aws:scheduler:::aws-sdk:service:apiAction, and JSON Input matching the operation’s request. The documented Lambda target is arn:aws:scheduler:::aws-sdk:lambda:invoke. Give the execution role permission to invoke that worker, and let the worker perform source-specific discovery or initiate ingestion.
Keep the schedule separate from the source connector’s capabilities. A scheduler does not give a repository an incremental API, deletion feed, or stable revision field. The worker must use the source’s supported inventory or sync behavior. Connector-specific choices for Kendra, AppFlow, and managed knowledge bases belong to 1.4.9.
Select refresh frequency from the freshness requirement and measured job duration. Prevent uncontrolled overlapping runs for the same source in the worker’s coordination logic. Record the last successfully reconciled source boundary rather than advancing it merely because the schedule fired. A run that scans only part of the inventory must not authorize deletion of all unseen documents.
Key distinctions and trade-offs
Event-driven detection versus periodic discovery: Streams reduce the delay between a DynamoDB modification and observed work. A schedule provides bounded opportunities to discover repository changes. Neither path guarantees immediate searchable visibility.
Record uniqueness versus consumer idempotency: A unique stream record can be delivered to a Lambda handler more than once. Deduplicate logical work by source identity and revision, not by assuming a single invocation.
Low detection lag versus low retrieval staleness: IteratorAge measures stream processing lag. A downstream ingestion job can still be slow or fail after the detector catches up. Measure source-to-retrievable delay independently.
Retention versus recovery: A 24-hour stream is a change log with finite history. It is not a permanent inventory. A repair scan must compare the authoritative source with the projection when the consumer cannot recover the missing interval.
Architecture / decision scenario
A product assistant uses DynamoDB product explanations and approved documents in a separate repository. The team selects NEW_AND_OLD_IMAGES and compares the explanatory-text revision before creating embedding work. Repeated Lambda delivery produces the same logical work identity, and partial failures use ReportBatchItemFailures.
The repository is an S3 data source on a customer-managed knowledge base and must be fresh within an hour, which no native schedule offers. EventBridge Scheduler therefore invokes a Lambda worker hourly that starts and tracks the ingestion job. The application measures detection lag and searchable revision lag separately. During a consumer outage beyond stream retention, it performs an authoritative repair inventory; during a failed scheduled scan, it retains the last successful boundary and avoids interpreting missing observations as source deletions.
Common exam traps
- Using DynamoDB Streams to claim change detection for unrelated repositories.
- Treating exactly-once stream records as exactly-once Lambda processing.
- Assuming order across all table items rather than per item.
- Returning partial failures without enabling
ReportBatchItemFailures. - Treating a schedule firing or
StartIngestionJobreturning as completed indexing. - Building a Scheduler loop for a managed connector whose daily
syncSchedulealready meets the freshness requirement. - Expecting a 24-hour stream to support an arbitrarily old replay.
What to remember
Choose the detector by source, preserve identity and revision, and make repeated delivery safe. Name StreamViewType, ReportBatchItemFailures, IteratorAge, and the scheduled target that carry the design. Track searchable freshness beyond detection and maintain a recovery inventory for missing history.
Knowledge check
A Lambda function consumes a DynamoDB stream and creates an embedding job for each changed product description. After a retried batch, two jobs exist for the same item and revision. How should the function identify index work?
A DynamoDB stream consumer shows an IteratorAge of a few seconds, but the downstream embedding worker has been stalled for two hours. What can the team conclude about freshness?
IteratorAge includes downstream processing timeIteratorAge measures the time between the stream receiving a record and the event source mapping sending it to the function. It says nothing about extraction, embedding or indexing afterwards, so searchable freshness needs its own measurement.An S3 data source on a customer-managed knowledge base must reflect changes within an hour. How should refreshes be triggered?
syncSchedule on the data source, set to run every hour on the hourStartIngestionJob and stops theresyncSchedule offers only daily, weekly or monthly runs. A scheduled worker starts StartIngestionJob and follows it with GetIngestionJob , because a fired schedule or an accepted start is not completed indexing. DynamoDB Streams covers DynamoDB tables, not buckets.A stream handler returns batchItemFailures naming the one record that failed, yet Lambda keeps retrying the entire batch. What is the cause?
KEYS_ONLY , which disables partial batch responsesReportBatchItemFailures is not enabled in the mapping’s FunctionResponseTypesParallelizationFactor is 1, which forces whole-batch retriesReportBatchItemFailures is enabled for the event source mapping. Once enabled, Lambda checkpoints at the lowest reported sequence number and retries from there, so records after it can still run again.A product table’s items hold an explanatory text field and a frequently updated view counter. Only text changes should create embedding work, and the consumer must decide from the stream record alone. Which StreamViewType fits?
KEYS_ONLY , because the key is enough to schedule embedding workNEW_AND_OLD_IMAGES , so the consumer can compare the text revisionNEW_IMAGE , because the new image shows which fields were changedOLD_IMAGE , because the old image shows which fields were changedKEYS_ONLY identifies the item without content, and a single image cannot show what changed; which fields deserve re-embedding remains an application rule.