NVIDIANVIDIA-Certified Professional: Agentic AI2026-106.3 Build extract, transform, and load (ETL) pipelines to integrate enterprise or client data sources.
Choose ingestion cadence from freshness targets
Completed by the knowledge check
Learning objective
Design batch versus incremental, event-driven ingestion.
Core idea
Separate two decisions: how much source data to process and what starts processing. A batch can rebuild the whole corpus or process only changed documents. An event can trigger an incremental update, but does not make extraction or indexing instantaneous.
For an enterprise assistant, choose the ingestion design from the allowed delay between a source change and a searchable result. Include source detection, queueing, download, extraction, embedding, and index publication in that delay. See 6.3.1 for extraction choices and 6.3.3 for the identity, permissions, and failure records that travel through the pipeline.
How it works
Use the ingestor as the processing boundary
The NVIDIA RAG Blueprint ingestion API exposes POST /documents for upload, PATCH /documents for replacement when a document already exists, GET /documents for listing, and DELETE /documents for deletion. Its OpenAPI server base is /v1; resolve these paths against the deployed ingestor base URL.
Uploads carry multipart documents and a data string containing JSON settings. The documented request example includes collection_name, blocking, split_options, and custom_metadata. With asynchronous submission, retain the returned task_id and poll GET /status with that identifier. The batch scheduler and event consumer can use the same processing boundary instead of implementing different extraction semantics.
Distinguish acceptance from completion. A nonblocking submission gives your controller a task to observe. Publish the controller’s successful source-version record only after the result has been checked, including failed_documents and validation_errors. See 6.3.3 for interpreting those fields. Queue submission alone cannot establish freshness.
Bound batch work by its delivery window
Use a full-corpus batch for an initial load or a deliberate reprocessing campaign. A parser or transformation change can justify re-extracting unchanged source bytes because the derived content changes. A source change alone usually needs a smaller update set.
For scheduled incremental batches, make the scheduler produce a change manifest: source object, source version, intended action, target collection, and completion record. These are application-defined fields, not an NVIDIA request schema. Resolve each entry to a document upload, replacement, or deletion through the ingestor API.
Advance a source’s processed-version record after confirmed success. Keep failed entries eligible for repair without replaying every successful file. A listing-based reconciliation can compare the desired source set with GET /documents, but account for the API’s response cap: its max_results defaults to 1000 and accepts up to 1000000. A capped listing is not proof that the remaining documents were deleted or never indexed.
Scheduling every five minutes does not imply five-minute freshness. A document arriving just after a scan waits almost an entire interval before processing begins. If processing then takes four minutes, the freshness delay can approach nine minutes before additional queueing. Choose the interval and processing capacity together.
Route storage events to incremental work
NVIDIA’s continuous-ingestion reference connects MinIO notifications to a Kafka topic. A Kafka consumer reads an event, downloads the corresponding object, and sends the file to the RAG ingestor for indexing. The deployment example is examples/rag_event_ingest/deploy/docker-compose.yaml on the RAG stack’s nvidia-rag Docker network.
The documented consumer settings include KAFKA_BOOTSTRAP_SERVERS, KAFKA_TOPIC, MINIO_ENDPOINT, INGESTOR_SERVER_URL, and COLLECTION_NAME. They identify the event stream, source storage, processing service, and target collection. The reference stack lists two RTX PRO 6000 Blackwell GPUs or two H100 GPUs as its hardware requirement.
The reference indexer uploads put-event documents with POST /documents; routing revisions through PATCH /documents is an adaptation the adopting team must implement. The reference consumer routes delete events to a document deletion path, but the bucket must publish delete notifications; otherwise, reconciliation is the only removal path for withdrawn documents. Its document indexer checks failed_documents and validation_errors when parsing a finished ingestion result. Use that distinction when adapting the example: an event’s arrival, a successful HTTP submission, and a successful indexed update are different observations.
Make replay and replacement explicit
Use the application change manifest as a processing ledger for both scheduled and event-triggered updates. Key a processing decision by source identity and version, rather than merely by event arrival time. If a repeated notification refers to a version already confirmed successful, reconcile its recorded result instead of automatically paying for extraction again.
When a source revision changes, deliberately choose replacement through PATCH /documents; when the source is removed, propagate deletion through DELETE /documents. Serialize conflicting operations for one source in the controller so an unfinished replacement does not publish content after a deletion. The controller owns this ordering policy; the API operation names do not establish an end-to-end ordering guarantee.
The published example consumer configures Kafka with enable_auto_commit=True. Treat its delivery behavior as a reference implementation to review, rather than evidence that source events and vector writes commit atomically. Keep the processing ledger and reconciliation path so you can identify a source version whose notification was consumed but whose ingestion never completed. Do not infer exactly-once indexing from the presence of Kafka.
Key distinctions and trade-offs
Compare scope and trigger independently:
| Design | Deciding condition | Mechanism | Cost or operational burden |
|---|---|---|---|
| Full batch | Initial corpus load or changed extraction logic | Change manifest covering every source; document API | Re-extraction and embedding for unchanged files |
| Scheduled incremental batch | Changes can wait for the scan interval | Version comparison plus PATCH /documents and deletion | Scan cost and interval-dependent staleness |
| Event-driven incremental | Changes need prompt detection | MinIO notifications, Kafka consumer, ingestor | Event infrastructure, replay, ordering, and reconciliation |
An event-driven path can reduce detection delay while still losing its freshness target to extraction backlog. A scheduled path can be adequate when a bounded incremental batch finishes comfortably within its window. Choose the simplest trigger that meets the measured end-to-end requirement, then retain a reconciliation process for missing or failed updates.
Architecture / decision scenario
A retail merchandising research assistant indexes 80,000 supplier documents. Merchandisers need price corrections searchable within ten minutes. During business hours, 600 documents change per hour. A measured incremental workload processes 1,200 documents per hour on the assigned ingestion deployment. Under normal load, the team measures four minutes from source change to searchable content, including detection, queueing, and processing. The initial corpus load is complete.
One proposal rebuilds the corpus each hour to avoid operating event infrastructure. At the measured rate, one full load needs about 66.7 hours, so this cannot meet either the hourly schedule or the freshness target. A second proposal scans every fifteen minutes; it can miss the ten-minute target before extraction starts.
The data-platform owner chooses event-driven incremental ingestion using the Blueprint’s MinIO-to-Kafka-to-ingestor pattern. PATCH /documents handles revisions, while the deletion path removes withdrawn documents. The normal arrival rate uses half the measured processing capacity, leaving room for variation. A reconciliation job compares source versions with the controller’s completed-version ledger and repairs entries without confirmed success.
A promotion burst raises arrivals to 1,800 per hour for thirty minutes. That creates roughly 300 documents of backlog above the 1,200-per-hour processing rate. An arrival behind that backlog waits about fifteen minutes at 1,200 documents per hour before its own processing, already exceeding the ten-minute freshness target. With arrivals back at 600 per hour, draining the backlog takes about thirty minutes. The owner therefore cannot promise the ten-minute target from event delivery alone; the release decision needs measured burst capacity or a narrower priority workload that fits the target.
Common exam traps
- Treating “batch” as necessarily a full-corpus reload.
- Treating an event-driven pipeline as instantaneous indexing.
- Advancing a processed-version record when upload is merely accepted.
- Replaying a whole corpus after one failed document.
- Inferring exactly-once vector writes from Kafka notifications.
- Ignoring deletes or using an incomplete document listing for reconciliation.
What to remember
Choose processing scope separately from its trigger. Use incremental updates when the source change set is small, events when detection delay matters, and full batches when the whole derived corpus needs rebuilding. Measure freshness through confirmed indexing and budget for bursts, replay, replacement, and deletion.
Knowledge check
A team changes its table extraction configuration so that cells keep their units. No source document has changed since the last run. Which ingestion scope brings the indexed content in line with the new configuration?
Price corrections must be searchable within eight minutes of a source change. A proposal scans for changes every five minutes, and processing one change takes a measured four minutes. Which assessment is correct?
An event consumer submits each changed document to the ingestor with blocking: false and receives a task_id. When should the controller advance that source’s processed-version record?
task_idfailed_documents or validation_errorsGET /status returns a finished result whose per-document error lists are empty. An accepted upload, a consumed notification and a completed download are earlier observations, and advancing the record on any of them can hide a document that never reached the index.A nightly job compares the 80,000 expected source documents with the response of GET /documents called with default parameters, then schedules re-ingestion for every expected document missing from that response. An independent check confirms that all 80,000 remain indexed, yet the job schedules about 79,000 re-ingestions. What is the most likely cause?
max_resultsmax_results defaults to 1000 , so the default listing returns at most 1,000 of the 80,000 indexed documents; it accepts values up to 1000000 . A capped listing is not evidence that the rest are missing. The independent check confirms that every document remains indexed, which rules out lost events and rejected uploads as the source of the gap.