Document Continuous Ingestion from Object Storage

Why it matters

Event-driven ingestion replaces a schedule with a reaction to each change, and its acknowledgment lives in a different place from its result. This notebook wires the whole path, so you can follow one document from upload to answer.

What to practise

Deploy the RAG Blueprint (NIMs, Elasticsearch, ingestor and RAG server), then the continuous ingestion stack: Kafka, MinIO and the kafka-consumer container. Upload a sample PDF to the aidp-bucket bucket and watch the consumer index it into the aidp_bucket collection without any call from you. Read the consumer log’s SUMMARY line for the file, collection, duration and status, then query the collection through collection_names. Note that the upload succeeding says nothing yet about the index.

How it fits

This is 6.3.2’s event-driven ingestion; compare it with that Bite’s batch and incremental options for the same freshness target. The gap between the upload and the consumer’s status is 6.3.3’s distinction between an accepted job and a complete result.