Skip to lesson

AWSAWS Certified Generative AI Developer - Professional2026-101.4 Design and implement vector store solutions

Match source ingestion to sync ownership

Completed by the knowledge check

Learning objective

Ingest documents, wikis, and SaaS data with Knowledge Bases connectors, Amazon Kendra, or Amazon AppFlow based on source type and sync needs.

Core idea

Choose the ingestion path from the source contract and the system that must become searchable. A Knowledge Bases connector connects a repository to knowledge-base ingestion. A Kendra data source synchronizes a repository into a Kendra index. AppFlow transfers records between supported applications and destinations. These mechanisms do not have interchangeable outputs.

The deciding question is not which service recognizes the source’s brand. Ask who discovers changes, what gets copied or indexed, how deletion is represented, and which operation makes the result retrievable. A successful source transfer is not proof that a knowledge base contains the corresponding evidence. Resource hierarchy and vector-store configuration belong to 1.4.2; this lesson chooses how source data reaches that hierarchy.

How it works

Connect document repositories to Knowledge Bases

For Knowledge Bases, CreateDataSource identifies the knowledgeBaseId, name, and dataSourceConfiguration. The configuration’s type and service-specific fields identify the source. Amazon S3 provides a path for source files; custom data sources provide another path when you own the document delivery mechanism. Preserve source identity and provenance through preparation, as described in 1.4.5.

AWS changed the connector creation boundary on September 30, 2026. Customer-managed knowledge bases no longer support creating connectors for Confluence, Microsoft SharePoint, Salesforce, or Web Crawler. Existing connectors of those types continue to support ingestion and retrieval. AWS recommends Amazon Bedrock Managed Knowledge Base for applications requiring these connectors.

The managed path does not replace every restricted connector. Its connectorParameters.type values are S3, ONEDRIVE, CONFLUENCE, SHAREPOINT, WEB_CRAWLER, and GOOGLE_DRIVE; there is no managed Salesforce type. For a source the managed knowledge base does not support, AWS recommends exporting content to Amazon S3 and configuring an S3 data source, which is where AppFlow enters the design.

For a managed Confluence source, the documented CreateDataSource configuration uses type: MANAGED_KNOWLEDGE_BASE_CONNECTOR and managedKnowledgeBaseConnectorConfiguration.connectorParameters with type: CONFLUENCE. Connection configuration includes the host and authentication information. The managed connector documentation names AWS Secrets Manager for credentials and provides crawl scope and inclusion controls. CreateDataSource is asynchronous for managed knowledge bases: the source transitions from CREATING to AVAILABLE.

The managed configuration also owns its sync trigger. The optional syncSchedule takes exactly one of daily (an empty object; AWS picks an off-peak time), weekly (dayOfWeek), or monthly (dayOfMonth, either a dayNumber from 1 to 28 or lastDayOfMonth). Omit it and the source syncs on demand only. Deletion safeguards in the same configuration belong to 1.4.11.

Set Confluence aclEnabled explicitly. AWS documents authentication-dependent defaults and states that the ACL configuration is permanent after creation. The access policy still needs the design and verification from 1.4.6; authentication to a repository and authorization to individual documents are different decisions.

Connector support also has a content boundary, and it depends on the path. For customer-managed knowledge bases, AWS says multimodal content is supported only with S3 and custom data sources; other source types skip images, audio, and video during ingestion. Managed connectors instead offer mediaExtractionConfiguration, with independent imageExtractionConfiguration, audioExtractionConfiguration, and videoExtractionConfiguration (.mp4, .mov, .m4v). A wiki with video attachments therefore needs that setting enabled on the managed path, or a separate plan for the attachments on an existing customer-managed connector. A connector that authenticates successfully does not guarantee ingestion of every object it can enumerate.

Keep existing Kendra synchronization

Start with the service’s status. AWS put Amazon Kendra into Maintenance Mode on June 30, 2026, with no new feature development, and closed it to new customers on July 30, 2026. Existing customers keep full support with bug fixes and security updates. AWS recommends Amazon Bedrock Managed Knowledge Base for migrations and for new search applications. Kendra is therefore a path for an account that already runs it, not a choice for a new account’s design.

For those existing deployments, Kendra data source connectors connect repositories, including Microsoft SharePoint and Google Drive, to a Kendra index, and a periodic scan updates that index.

Kendra’s Schedule parameter on data source creation or update accepts a cron expression. An empty string indicates on-demand updates. Kendra supports cron expressions, not rate expressions, for this parameter. StartDataSourceSyncJob starts a manual synchronization.

Set a schedule from the freshness requirement and the repository’s change pattern. A nightly scan might satisfy an archive but fail a policy assistant whose documents change during the workday. Test changed, removed, and inaccessible documents through the chosen connector.

Transfer SaaS records with AppFlow

Amazon AppFlow supports on-demand, scheduled, and event-triggered flows. Event triggers are available only for SaaS applications that provide change events; frequency for scheduled flows depends on the source application. Check the specific connector rather than promising event-driven behavior for every SaaS source.

Scheduled flows can use full or incremental transfer. Full transfer copies a snapshot of records at the flow run. Incremental transfer copies records added or changed since the last successful flow run, using a selected timestamp field. Choosing a creation timestamp can capture new records while missing updates. The timestamp choice is therefore part of the correctness contract, not a cosmetic mapping preference.

AWS documents that the first scheduled flow pulls 30 days of past records. Do not mistake that initial window for a full historical backfill. Establish the required baseline separately when older records belong in the corpus, then validate subsequent incremental coverage.

AppFlow can use S3 as a destination and export JSONL (the default), CSV, or Parquet. Knowledge Bases accepts .txt, .md, .html, .doc/.docx, .csv, .xls/.xlsx, and .pdf files, each within a 50 MB quota. CSV is on that list; JSONL and Parquet are not. Either choose CSV deliberately or add a preparation step that selects useful text, preserves record identifiers and revision information, and writes a supported format before ingestion.

Key distinctions and trade-offs

Compare the target and sync responsibility explicitly:

PathTarget maintainedSync triggerResponsibility to resolve
Managed knowledge base connectorKnowledge-base source ingestionsyncSchedule or on demandSupported connector type, scope, and permissions
Existing Kendra connectorKendra search indexCron Schedule or StartDataSourceSyncJobMaintenance Mode status and migration plan
AppFlow to S3Transferred source recordsFlow on demand, on schedule, or on eventSupported format, ingestion, and deletion reconciliation

An incremental record transfer is not a documented universal deletion feed. Establish how the chosen source exposes removals or compare against an authoritative inventory. Reconciliation and stale-vector handling belong to 1.4.11. Likewise, avoid expanding a connector’s crawl scope to compensate for poor retrieval before checking whether the content is eligible and authorized.

Architecture / decision scenario

An enterprise assistant needs approved S3 documents, Confluence wiki pages, and SaaS case records. It connects the document collection to a Knowledge Bases source. For a Confluence connection created after September 30, 2026, it uses the managed connector with a daily syncSchedule instead of attempting an unsupported customer-managed connector creation.

Another business unit, an existing Kendra customer, already serves its wiki through Kendra. It keeps that index for now with a cron Schedule, and records the migration to the managed knowledge base that AWS recommends. The case records live in a SaaS application with no managed connector type, so a connector is not an option. The pipeline uses AppFlow scheduled incremental transfer with an update-sensitive timestamp, lands CSV records in S3, and ingests them through an S3 data source. Acceptance checks cover all three paths, including baseline history, edited wiki pages, and removed case records.

Common exam traps

  • Treating Knowledge Bases, Kendra, and AppFlow as interchangeable vector-index loaders.
  • Creating a restricted customer-managed connector after the documented cutoff.
  • Interpreting the cutoff as termination of all existing connectors.
  • Choosing Kendra for a new account’s ingestion design.
  • Expecting a managed connector for every restricted customer-managed type, such as Salesforce.
  • Expecting Kendra’s Schedule to accept a rate expression.
  • Selecting a creation timestamp while expecting incremental updates.
  • Treating an AppFlow transfer as completed downstream indexing or a universal deletion feed.

What to remember

Choose by source type, searchable target, and sync ownership. Verify connector creation eligibility before designing the connection. Name the trigger and inspect the target it maintains: Kendra synchronization, Knowledge Bases ingestion, and AppFlow record transfer are different stages with different acceptance evidence.

Knowledge check

References