NVIDIANVIDIA-Certified Professional: Agentic AI2026-102.2 Integrate generative and multimodal models (text, vision, audio).
Route visual questions to models or retrieval
Completed by the knowledge check
Learning objective
Integrate vision-language models and multimodal queries over images and documents.
Core idea
An image can be the material to inspect, a query used to find related documents, or retrieved context for an answer. Those roles require different integration paths. A vision-language model (VLM) interprets visual and textual input; a multimodal embedding model represents inputs for similarity search. Choose the path from the question’s information needs before choosing the endpoint.
For a supplied chart, a direct VLM request can ask what the chart contains. For a question about the company’s approved interpretation of that chart, connect the image query to an ingested document collection. Visual understanding alone does not supply the missing corporate knowledge. Retrieval alone does not ensure that the generator receives the visual detail needed to explain its result.
How it works
Build a typed image request
NVIDIA NIM for LLM and VLM documents multimodal input through its vLLM backend. The model must support the media type you send; a text-only model does not become a VLM because the request contains an image. Obtain the served model name from /v1/models and check the chosen model’s documented input capabilities.
Send POST /v1/chat/completions with a messages array. In a user message, make content an array containing a type: "text" item for the question and a type: "image_url" item for the image. The latter carries image_url.url. This preserves the distinction between instructions about the image and the image bytes themselves.
The URL can reference an image reachable from the NIM container, or contain inline bytes such as data:image/png;base64,ENCODED_IMAGE. Match the MIME type to the actual file. A URL accessible in an employee’s browser can still fail from the serving container’s network. Inline bytes remove the request-time remote download; they also put the encoded image in the request payload. Choose the transport with the data-handling owner, rather than assuming browser access proves server access.
For a document page that must be inspected visually, your application can render that page to an image and submit it through this interface. Keep the source document ID and page number alongside the image in application state. This is an application design, not a promise that every VLM accepts a PDF or preserves its page metadata. If the question spans pages, explicitly choose the pages needed and verify that the selected model supports the resulting request. For image resolution against GPU memory and latency, see 7.5.2.
Search an ingested visual collection
The NVIDIA RAG Blueprint’s multimodal-query path combines visual embeddings with VLM answer generation. Before querying, create a collection, ingest the documents, and select the collection. A query image does not replace those ingestion steps. The documented on-premises path requires an NVIDIA H100 or A100 GPU. Its Helm configuration replaces the default LLM NIM and text embedding NIM with VLM NIM and VLM embedding NIM. Set APP_EMBEDDINGS_MODELNAME="nvidia/llama-nemotron-embed-vl-1b-v2" for the documented visual embedding path, including ingestion.
The Blueprint’s documented image-ingestion settings include APP_NVINGEST_EXTRACTIMAGES="True" and APP_NVINGEST_IMAGE_ELEMENTS_MODALITY="image". Its example also sets APP_NVINGEST_STRUCTURED_ELEMENTS_MODALITY="". Treat these as settings for that documented visual ingestion configuration, rather than copying them over a different extraction design without checking what content it needs. For document extraction choices, see 6.3.1.
For questions about images inside ingested documents, consider ingestion-time image captioning. Enable APP_NVINGEST_EXTRACTIMAGES="True" and configure APP_NVINGEST_CAPTIONENDPOINTURL and APP_NVINGEST_CAPTIONMODELNAME for the captioning VLM. NVIDIA documents higher accuracy for image-related document questions at the cost of higher ingestion latency. This pays caption-generation work during ingestion; it serves a different input need from using an image as the query.
In the Python client, call rag.generate with a user message containing the mixed text/image content, use_knowledge_base=True, and explicit collection_names. Set enable_reranker=False for image queries. The documented path uses multimodal vector retrieval directly and bypasses reranking. Its deployment examples also disable reranking with ENABLE_RERANKER="false".
An image query reaches VLM answer generation through a collection created, ingested and selected before querying.
An image query retrieves content from only one page per document. It does not supply multi-page context from that document. If a question requires a diagram on page 4 and a qualifying procedure on page 9, design an additional retrieval step or select both pages explicitly for a compatible generator. Do not infer complete document coverage from a relevant first hit.
Preserve the question’s source scope
Keep the selected collection and source-page identity with the workflow result. When a query returns an irrelevant visual match, first check collection_names, whether the expected documents were ingested, and whether image extraction was enabled. Changing the answer prompt cannot recover material that never entered the searched collection.
Use an integration evaluation set containing direct image questions, document lookup questions, and questions whose answers require multiple pages. Record the expected source pages as well as acceptable answers. This tests the integration decision: whether the right material reaches generation. A fluent answer with the wrong source is a failed document query.
Key distinctions and trade-offs
Use the required information to choose between these paths:
| Path | Deciding condition | Mechanism | Cost or limitation |
|---|---|---|---|
| Direct visual inspection | The supplied image contains the needed information | Chat Completions content with text and image_url | Request-time visual processing; selected model must support the image input |
| Image-assisted document search | The image identifies related material in an enterprise corpus | rag.generate, use_knowledge_base=True, collection_names | Ingestion and embedding work; image-query reranking is bypassed |
| Ingestion-time image captions | Text questions concern images inside documents | APP_NVINGEST_CAPTIONENDPOINTURL, APP_NVINGEST_CAPTIONMODELNAME | Higher documented accuracy for those questions; higher ingestion latency |
| Cross-page document answer | The answer depends on separated pages | Additional retrieval or application-selected page images | More context and processing; one image-query hit per document cannot supply all pages |
A VLM and a visual embedding model serve different roles even when both accept images. Improving generation cannot repair a missing retrieval hit; improving similarity cannot make absent page context available. Assign ingestion coverage to the retrieval owner and input compatibility to the model-serving owner. The application owner decides which path each question uses and checks the end-to-end result.
Architecture / decision scenario
A manufacturing engineering assistant compares a photographed component with a 120-page maintenance manual. Company policy keeps photographs and manuals inside its on-premises network. In this illustrative evaluation, direct VLM inspection takes 0.8 seconds, while image-assisted retrieval plus generation takes 1.5 seconds against a 2-second response target.
The tempting proposal sends every request directly to the VLM: it is faster and can describe the component. The application architect rejects that path for questions such as “Which approved procedure applies?” because the photograph contains no approved procedure. The platform owner confirms an H100 or A100 is available and budgets the VLM and VL embedding NIMs in place of the default text services. The retrieval team ingests manual images with nvidia/llama-nemotron-embed-vl-1b-v2, and the workflow specifies the maintenance collection in collection_names with enable_reranker=False.
For “What marking is visible?”, direct inspection remains appropriate. For “Find the matching diagram”, the image-query path supplies document matches. For “Apply the diagram on page 4 with the warning on page 9”, the single-page restriction rules out treating that hit as sufficient. The architect adds a second page-selection step and measures its total latency; if it exceeds the target, the assistant returns the located material for review instead of fabricating a complete interpretation.
For repeated text questions about manual illustrations, the retrieval owner also evaluates ingestion-time captions, accepting extra ingestion latency only if the image-question accuracy improves enough. Captions do not replace the photograph-as-query path.
The serving team verifies container access to internal image URLs, or the application submits inline image bytes. The evaluation owner checks both source-page coverage and answer correctness before accepting the faster path.
Common exam traps
- Treating visual embeddings and VLM generation as interchangeable operations.
- Sending an image to a text-only model because the endpoint is OpenAI-compatible.
- Assuming a browser-reachable URL is reachable from the NIM container.
- Querying a document collection that was never ingested or selected.
- Enabling reranking for the Blueprint’s image-query path.
- Assuming a single-page image retrieval result supports a cross-page conclusion.
What to remember
Choose direct VLM inspection when the supplied image contains the answer. Choose multimodal retrieval when the image must locate enterprise knowledge. Preserve source identity, select the collection explicitly, and account for the Blueprint’s reranking and single-page restrictions before generating a document-wide answer.
Knowledge check
A field technician photographs a pump component and asks which approved repair procedure applies. The procedures exist only in maintenance manuals already ingested into a collection on the NVIDIA RAG Blueprint’s documented multimodal-query path. Which integration answers the question?
rag.generaterag.generate with use_knowledge_base=True and the manual collection in collection_names retrieves matching material for VLM answer generation. Direct VLM inspection fits when the supplied image holds the answer, a text-only model cannot process the image, and captioning enriches ingested documents rather than using a query image.A team runs image queries on the NVIDIA RAG Blueprint’s documented multimodal-query path and wants better ordering of the results. An engineer proposes enabling the reranker for those requests. What does the documented path specify?
collection_names lists several collectionsenable_reranker=False and the deployment examples set ENABLE_RERANKER to false. Neither an ingestion-time setting nor the number of selected collections is documented to lift that restriction.On the NVIDIA RAG Blueprint’s multimodal-query path, an engineer asks how to apply the diagram on page 4 of a manual together with the warning on page 9. The image query returns a strong match on page 4. What should the workflow do before generating the answer?
A VLM NIM request sets image_url.url to a link on the company’s internal file share. The link opens in an engineer’s browser, but the request fails at the NIM, and the container has no route to that share. Images may travel inside the request under the data-handling policy. Which change fits?
type: "text" content itemsystem message instead of the user messagedata:image/png;base64,… remove the request-time download, with the MIME type matched to the file. Base64 text in a text item is not an image input, a link in the question text reaches the model only as text, and moving the link to another message role leaves the same unreachable URL.Employees ask text questions about the wiring diagrams inside manuals already ingested into an NVIDIA RAG Blueprint collection, and answers often miss what the diagrams show. Retrieved passages contain surrounding prose but no descriptions of the diagrams. The deployment runs on H100 GPUs, users never upload images, and the team accepts slower ingestion but not slower answers. Which change targets the problem?
enable_reranker=FalseAPP_NVINGEST_CAPTIONENDPOINTURL and APP_NVINGEST_CAPTIONMODELNAME with image extraction enabled, is documented to give higher accuracy for questions about images in ingested documents at the cost of higher ingestion latency, which the team accepts. The image-query path needs a query image the users never supply, rendering pages for the VLM at query time adds visual processing to every answer, and a reranker only reorders retrieved text that still does not describe the diagrams.