NVIDIANVIDIA-Certified Professional: Agentic AI2026-106.3 Build extract, transform, and load (ETL) pipelines to integrate enterprise or client data sources.
Join source identity, permissions, and ingestion outcomes
Completed by the knowledge check
Learning objective
Track source identity, access controls, and ingestion failures.
Core idea
An ingestion pipeline produces derived content that still belongs to a source and its access policy. You need to answer three questions for every searchable chunk: where did it come from, who may retrieve it, and did processing complete for the intended source version?
Keep these questions connected through a processing record. A document’s filename is useful for display and API operations, but it is not enough to distinguish the same filename in two repositories or to explain which revision failed. See 6.3.1 for extraction and 6.3.2 for ingestion cadence. Here, the concern is whether you can authorize, trace, and repair the resulting content without mixing sources or hiding failures.
How it works
Preserve identity across transformations
NeMo Retriever’s canonical SourceMetadataSchema contains source_name, source_id, source_location, and source_type, with optional context including collection_id, last_modified, and access_level. ContentMetadataSchema includes page_number and structural hierarchy. Together, source and content metadata connect a derived region to its origin.
Respect the stage boundary. The library constructs the canonical nested MetadataSchema during .vdb_upload(). Extraction-only DataFrame rows have extractor-specific flat columns: PDF page rows include source_id, but Markdown, plain-text, and HTML chunks do not. Do not write a transformation that assumes every extraction result already contains source_metadata.source_id.
At the connector boundary, assign an application source key from the authoritative repository identity and object identity. Retain its source version, target collection, and document name in the processing ledger introduced in 6.3.2. These are application-defined fields; adapt them to the library’s output and the Blueprint’s API rather than assuming identical schemas.
The RAG Blueprint accepts per-file custom_metadata entries with filename and a metadata object. Its collection metadata_schema defines field names, types, and whether each field is required. Define fields such as tenant_id, source_key, source_version, and allowed_groups as your application schema. Use a string array for allowed_groups, with array_type: "string", and mark identity and authorization fields required for this design.
The names type, subtype, and location are reserved and cannot be custom schema fields. If ingestion reports an unknown field, align the submitted metadata with the collection schema. If it reports a missing required field, repair the connector’s metadata rather than weakening the requirement to admit an unclassified document.
Enforce permissions before retrieval supplies context
OWASP’s vector and embedding guidance calls for permission-aware stores and strict logical and access partitioning. Carry the source’s access classification into derived content, then make the retrieval boundary apply it before the generator receives chunks. A prompt telling the model not to reveal another tenant’s information acts too late if those chunks already entered its context.
For a Milvus-backed Blueprint deployment, filter_expr carries a string boolean expression over metadata. A trusted application layer can construct a restriction such as content_metadata["tenant_id"] == "tenant-a" and combine it with array_contains_any(content_metadata["allowed_groups"], ["team-a", "team-b"]) to match any of the caller’s permitted groups. Obtain the tenant and groups from authenticated application context, not from a user’s question or arbitrary client-provided metadata.
Keep enable_filter_generator disabled on this authorization path and supply the required filter explicitly. When enabled, the generator can rewrite a supplied filter_expr to improve it, so even a trusted mandatory filter can change. The Blueprint documentation says LLM-generated filter failures can continue with no filtering; it also says user-supplied API filter validation errors are surfaced explicitly. Reject missing identity context and invalid mandatory filters in the application. Do not turn those errors into an unrestricted retry.
The identifier filter_expr does not make arbitrary filter text safe. The trusted service must control the permitted collection_names, build the mandatory restriction, and prevent a caller from bypassing that service with an unrestricted retrieval request. The application owner implements this authorization boundary; the metadata schema only validates stored values.
For multi-collection requests, metadata.allow_partial_filtering: false requires every collection to support the expression. With true, incompatible collections are skipped. Choose the strict mode when a missing authorization field signals a deployment defect that should fail the request. Treat an access-policy change as a publication change: exclude the affected source_key in the mandatory filter_expr until its derived content has the intended restriction or has been removed through DELETE /documents.
Reconcile job results with document outcomes
The ingestion API returns a task_id that you use with GET /status. The documented task states include PENDING, FINISHED, FAILED, and UNKNOWN. UNKNOWN means the task was not found or its state cannot be determined; it is not evidence of a successful upload.
Inspect the result’s total_documents, documents_completed, failed_documents, and validation_errors. Each failed-document entry can carry document_name and error_message. The nv_ingest_status object provides extraction_completed and document_wise_status, whose states include not_started, submitted, processing, completed, and failed.
Extraction progress and job completion measure different boundaries. A document marked completed in extraction status has passed that stage, not necessarily every later ingestion step. Likewise, inspect per-document errors even when the enclosing task reaches FINISHED: NVIDIA’s reference document indexer explicitly checks failed_documents and validation_errors before declaring success.
Record the join from source key and version to task_id, target collection, document name, and terminal result. Send schema validation defects back to the connector owner. Send extraction defects to the extraction owner with the source region and error message. If the outcome is unknown, reconcile the job and indexed document before scheduling another write; an ambiguous response does not establish that processing never happened. Replace or delete through the document API using the intended source-to-document mapping.
A FINISHED task still has its per-document errors inspected and sent to their owners, while an UNKNOWN one is reconciled before another write.
Key distinctions and trade-offs
Keep the three responsibilities explicit:
| Responsibility | Mechanism | What it establishes | What still needs a decision |
|---|---|---|---|
| Source identity | SourceMetadataSchema and application source/version record | Origin of derived content | Stable connector identity and revision mapping |
| Access control | Required custom metadata, trusted filter_expr, collection restrictions | Eligible retrieval content | Authenticated caller context and policy updates |
| Failure tracking | task_id, /status, document error lists | Processing progress and reported errors | Repair owner and reconciliation action |
Required metadata can reject an otherwise extractable document, but prevents unidentified or unclassified material from entering the intended publication path. Strict filtering can make a request fail when one collection is misconfigured, but exposes a schema defect instead of hiding it behind partial results. Choose those costs explicitly for sensitive corpora.
Architecture / decision scenario
A financial advisory firm builds an employee research assistant for 40 client teams. Its source repositories contain repeated filenames such as valuation.pdf. Every document must remain restricted to its client team, and revoked access must stop retrieval within two minutes. A nightly content ingestion takes four hours.
A developer proposes one shared collection with a natural-language filter generator that infers the client from each question. The security owner rejects it: a vague question or filter-generation failure cannot establish authorization. The ingestion owner defines required tenant_id, source_key, source_version, and allowed_groups metadata; the application builds filter_expr from authenticated team membership and restricts collection access.
A 200-document ingestion reports FINISHED, but failed_documents names two files and validation_errors identifies a third file with a missing access classification. The owner records 197 confirmed document results and sends the remaining three entries for repair. Replaying all 200 would obscure which revisions already succeeded.
An access revocation occurs during the four-hour nightly load. Waiting for that load would violate the two-minute requirement. The application owner blocks the affected source through its trusted retrieval policy immediately, while the ingestion owner removes or republishes the derived content and verifies the restriction before restoring retrieval. Source identity makes that targeted response possible even when other client teams use the same filename.
Common exam traps
- Assuming every extraction row already has the canonical nested source schema.
- Treating a display filename as a globally unique source identity.
- Confusing required metadata with enforcement of caller permissions.
- Depending on generated filters that can fall back to no filtering.
- Treating extraction completion,
FINISHED, orUNKNOWNas interchangeable outcomes. - Repairing content while leaving revoked access searchable.
What to remember
Join source identity and version to the ingestion task and indexed document. Require authorization metadata at ingestion and enforce it through a trusted retrieval path. Read per-document outcomes before declaring a job successful, and make permission changes effective within their own delivery target.
Knowledge check
A research assistant serves 40 client teams from one Milvus-backed collection, and each chunk carries required tenant_id and allowed_groups metadata. Callers reach retrieval only through the firm’s application service, and retrieval must return only chunks the caller’s team may see. Which design enforces that?
filter_expr from the caller’s authenticated tenant and groups, with the filter generator offfilter_expr from the caller’s authenticated tenant and groups, and let the generator refine itenable_filter_generator on, the generator can rewrite even a supplied filter_expr , and a failed generated filter can fall back to no filtering; inferring the team from the question cannot establish authorization at all. A prompt instruction acts after the other teams’ chunks have already entered the model’s context.A 200-document ingestion task reports FINISHED. Its failed_documents names two files, and validation_errors names a third that lacks a required access-classification field. What should the controller record?
FINISHED stateFINISHED while individual documents fail, so per-document results decide what succeeded. Replaying all 200 obscures which revisions already succeeded, and relaxing the requirement admits an unclassified document; the missing field is the connector’s metadata to repair.A controller polls GET /status for a submitted document replacement and receives UNKNOWN after a network interruption. What should it do before writing that document again?
UNKNOWN means the task was not found or its state cannot be determined: it is evidence neither of success nor of failure. Reconciling the task and the indexed document establishes what happened before another write; marking it complete may hide a failure, and resubmitting or deleting acts on an ambiguous response that does not show processing never happened.A client team’s access to one source is revoked while a four-hour nightly ingestion is running. Retrieval of that source must stop within two minutes. What should the application owner do first?
allowed_groups in tonight’s loadsource_key in the mandatory filter_exprmetadata_schema