AWSAWS Certified Generative AI Developer - Professional2026-102.3 Design and implement enterprise integration architectures
Match source synchronization to change semantics
Completed by the knowledge check
Learning objective
Synchronize enterprise knowledge sources using change data capture, DataSync, or Transfer Family
Core idea
Enterprise knowledge does not arrive through one universal synchronization mechanism. Databases expose committed row changes, file shares expose files and metadata, and partners deliver packages through transfer protocols. Choose the mechanism that matches how the source expresses change and who initiates movement.
A successful copy is evidence of data movement, not proof that the assistant can safely answer from the result. You still need a source identity, revision, deletion policy, accepted publication state, and downstream ingestion state. Keep synchronization separate from source-connector selection in 1.4.9, change detection in 1.4.10, and vector reconciliation in 1.4.11. This lesson owns the enterprise-to-staging boundary.
How it works
Capture database changes through CDC
AWS Database Migration Service (AWS DMS) supports ongoing replication through change data capture (CDC). It reads changes using the database engine’s native log APIs. A task can capture ongoing changes during a full-load migration or after an initial full load. The database needs the source-specific configuration that exposes usable changes; for example, MySQL requires row-based binary logging.
Choose the initial snapshot and CDC start position as one recovery design. Starting a change stream after loading a snapshot can miss changes if the two boundaries do not cover the same interval. AWS DMS supports CDC recovery checkpoints; DescribeReplicationTasks exposes task checkpoint information. Preserve the recovery information before deleting a task, because the task’s checkpoint information is lost on deletion.
CDC is not a real-time guarantee. AWS states that DMS CDC has no latency SLA, and replication lag can grow with workload, network conditions, replication resources, and target capacity. Define how the application determines source freshness and when it must consult the authoritative system rather than answer from a delayed projection.
When Amazon S3 is the DMS target, full-load and CDC data use CSV by default, with Parquet available as an alternative. CDC rows identify changes with operation indicators: I for insert, U for update, and D for delete. A knowledge pipeline must interpret those records and construct document state. An append-only folder of change files is not itself a replacement snapshot of the source.
By default, DMS S3 output stores changes by table without preserving transaction order. If preserving transaction order is necessary, set the S3 endpoint PreserveTransactions to true and set CdcPath for the transaction files. Transaction-order capture always emits CSV regardless of DataFormat; do not combine these settings with DatePartitionEnabled or addColumnName. Select the output contract before writing a consumer that assumes file layout and transaction ordering.
A delete record must remove or tombstone the corresponding derived document. Otherwise, a source record can disappear while its extracted text remains available for retrieval. Carry stable source identifiers through transformation so that deletion handling does not depend on matching titles or regenerated summaries.
Synchronize files and objects through DataSync
AWS DataSync moves data between supported source and destination locations. For an on-premises storage path, the documented architecture uses a DataSync agent. A task identifies the locations and transfer options; a task execution performs the run. Basic mode prepares, transfers, and verifies sequentially, while Enhanced mode performs these stages in parallel.
Configure TransferMode: CHANGED when subsequent runs should copy data and metadata that differ between source and destination. TransferMode: ALL copies everything without comparing differences. A changed-data run is a synchronization comparison, not database CDC: it does not emit every intermediate file revision that existed between runs.
Make deletion and overwrite behavior part of the knowledge contract. PreserveDeletedFiles determines whether destination files remain when they no longer exist at the source. OverwriteMode determines whether changed source data overwrites the destination. A destination configured to preserve deleted files can retain obsolete knowledge even when the transfer succeeds.
DataSync cannot both delete destination data absent from the source and use transfer-all mode, because transfer-all does not scan the destination to determine what should be deleted. Treat this as a deciding constraint when choosing between a mirror and an append-oriented landing area.
DataSync performs integrity checks during transfer and can perform additional end-of-transfer verification. It compares checksums and metadata; differences can produce verification errors. Enhanced mode verifies transferred data, while Basic mode’s final verification depends on the configured options. This verifies transfer integrity, not document correctness, access-policy correctness, or embedding readiness.
Files can change while a transfer runs. Agree on a source publication convention, such as finalized files in a designated directory, and accept a run only under that convention. Schedule recurrence from the freshness budget and storage workload. A completed run does not imply every file was readable or that every source change since the last accepted snapshot is represented.
Receive partner files through Transfer Family
AWS Transfer Family supports managed file transfer using protocols including SFTP, FTPS, FTP, and AS2, with AWS storage backends including Amazon S3 and Amazon EFS. Choose the documented protocol and backend combination that the partner’s delivery contract requires. Transfer Family servers accept partner-initiated delivery using established transfer clients. SFTP connectors instead initiate pull or push transfers between Amazon S3 and a partner-owned remote SFTP server. To pull files, call StartFileTransfer with ConnectorId, RetrieveFilePaths, and the S3 destination LocalDirectoryPath; use SendFilePaths for the outbound source files.
For server uploads, use managed workflows to process files before publication. Workflows support actions such as copying, tagging, and custom Lambda processing. Configure distinct paths for complete and partial uploads through OnUpload and OnPartialUpload. A file left by a prematurely disconnected session should not enter the same accepted-content path as a completed package.
Custom processing can validate the package’s expected structure and source revision. If validation fails, the custom step reports Status="FAILURE" to trigger configured exception handling. Use that outcome to keep invalid input outside the accepted knowledge area. Validation rules remain your application’s responsibility; Transfer Family does not know which supplier document or policy version your assistant should trust.
Key distinctions and trade-offs
The source’s change model determines the transport:
| Mechanism | Source contract | Evidence to preserve |
|---|---|---|
| DMS CDC | Database log changes after a defined start position | Checkpoint, row identity, and insert/update/delete semantics |
| DataSync | Comparison and transfer between storage locations | Task execution, overwrite/deletion policy, and verification outcome |
| Transfer Family | Server uploads or connector-initiated SFTP transfers | Transfer outcome; upload completeness and workflow validation for servers |
CDC preserves row-change meaning but needs source-engine preparation and materialization. DataSync handles storage synchronization but can miss intermediate revisions between runs. Transfer Family preserves partner transfer interfaces but needs an explicit package and publication contract. None independently grants retrieval authorization or proves the vector index has caught up.
Architecture / decision scenario
A compliance assistant uses policy records in a relational database, approved PDFs on an internal file share, and supplier declarations delivered over SFTP. The team uses DMS full load plus CDC for the policy records, materializes I, U, and D changes into document state, and records the recovery checkpoint.
The file share uses DataSync with TransferMode: CHANGED. The team chooses overwrite and deletion behavior to match its approved-document mirror and checks transfer verification before accepting the run. Supplier declarations arrive through Transfer Family servers; partners hosting their own SFTP endpoint instead use connector pulls through StartFileTransfer. Complete uploads enter a validation workflow; partial uploads take a separate exception path. A custom Lambda step reports failure for an invalid package rather than publishing it.
All three paths produce accepted source revisions for the downstream ingestion process. The assistant tracks source acceptance and index readiness separately. If a policy is deleted or superseded, the pipeline updates authoritative document state and reconciles its retrieval projection; it does not treat the continued presence of an S3 object as proof that the policy remains valid.
Common exam traps
- Treating DMS CDC as instantaneous replication with a latency SLA.
- Assuming default DMS S3 output preserves transaction order.
- Ignoring
Drecords when materializing knowledge documents. - Treating DataSync changed-data comparisons as a complete change log.
- Preserving destination deletions accidentally while claiming a mirror.
- Ingesting partial uploads as completed supplier packages.
- Treating verified transfer integrity as completed retrieval ingestion.
What to remember
Use CDC for database changes, DataSync for storage synchronization, and Transfer Family servers for partner uploads or SFTP connectors for AWS-initiated partner transfers. Preserve the source’s change semantics and the mechanism’s acceptance evidence. Decide how deletions, overwrites, partial uploads, and recovery checkpoints affect publication before handing accepted revisions to the knowledge pipeline.
Knowledge check
A supplier publishes its declarations on an SFTP server that it hosts and will not push files to you. The declarations must land in Amazon S3 on a schedule you control. Which mechanism fits?
StartFileTransfer to retrieve the filesRetrieveFilePaths into an S3 LocalDirectoryPath . DataSync would need an agent in the supplier’s storage environment, and DMS captures database changes, not files.DMS replicates a policy table to Amazon S3 with full load plus CDC, and a pipeline turns the output into retrieval documents. A policy deleted in the source database is still being cited by the assistant days later. What is the most likely defect?
D for that policyI , U or D , and the consumer must materialise document state from them; a delete must remove or tombstone the derived document, keyed by a stable source identifier. Output format does not change that duty, DMS CDC has no latency SLA to exceed, and CDC change files are not replacement snapshots.A consumer of DMS CDC files in Amazon S3 must apply related row changes from one transaction together and in commit order. The target endpoint uses default settings. What must change?
DataFormat to ParquetDatePartitionEnabled to partition the CDC files by datePreserveTransactions to true and set a CdcPathaddColumnName so each row carries its column namesPreserveTransactions with a CdcPath writes transaction-ordered CDC files, always as CSV, and it cannot be combined with DatePartitionEnabled or addColumnName . Choose this output contract before writing the consumer.An approved-document mirror is synchronised from a file share by DataSync with TransferMode: ALL. Runs succeed, but PDFs withdrawn at the source remain in the destination and keep being retrieved. Which configuration makes the destination a mirror?
TransferMode: CHANGED , with PreserveDeletedFiles set to remove files absent from the sourceTransferMode: ALL , with PreserveDeletedFiles set to remove files absent from the sourceTransferMode: ALL , with end-of-transfer verification of every file enabledTransferMode: CHANGED , with OverwriteMode set to overwrite changed filesPreserveDeletedFiles says so, and it cannot do that in transfer-all mode, because it does not scan the destination to learn what to delete. Verification checks transfer integrity, and overwriting handles changed files, not withdrawn ones.Suppliers upload declaration packages to a Transfer Family SFTP server. A session that disconnects early can leave a partial package behind. How do you keep partial packages out of the accepted knowledge area?
OnUpload workflow for every file and check sizes in itOnPartialUpload workflow that keeps them apartStatus="FAILURE" for invalid input; transfer integrity says nothing about business completeness.References
- AWS, Creating tasks for ongoing replication using AWS DMS
- AWS, Using Amazon S3 as a target for AWS Database Migration Service
- AWS, How AWS DataSync works
- AWS, Configuring how AWS DataSync transfers data
- AWS, What is AWS Transfer Family? and AWS Transfer Family SFTP connectors
- AWS, AWS Transfer Family managed workflows and StartFileTransfer