NVIDIANVIDIA-Certified Professional: Agentic AI2026-107.5 Manage and optimize multimodal input pipelines on NVIDIA hardware.

Route media through GPU decode and extraction

Completed by the knowledge check

Learning objective

Route video, images, audio, and documents through GPU-accelerated decoding and extraction.

Core idea

An accelerated multimodal pipeline is a sequence of format-specific stages. Parsing a container, decoding samples, extracting text, and interpreting an image are different operations. A GPU-backed model does not make every preceding operation GPU-backed, and a video transcript does not describe silent visual events.

Choose the route from the information the agent needs and the execution support of each stage. Use NVIDIA DALI for explicit media preprocessing and NeMo Retriever Library for document and speech extraction where those outputs fit the task. See 6.3.1 for document extraction semantics and 2.2.1 for visual model queries. This lesson adds the hardware and dependency decisions before those representations reach the agent.

How it works

Choose an integrated video service or explicit preprocessing

The Video Search and Summarization (VSS) Real-Time VLM microservice provides a DeepStream-accelerated route from decoding to vision-language model inference. Its pipeline receives RTSP streams, decodes them, creates video segments, and runs VLM inference. Use POST /v1/files and POST /v1/generate_captions for file-based processing, with media_type: "video" or "image". It accepts JPEG/PNG images and H.264, H.265, AV1, VP8, and VP9 video. AV1 uses hardware decode by default; set FORCE_SW_AV1_DECODER=true on platforms without hardware AV1 decoding.

With an Omni model, VLM_MODEL_SUPPORTS_AUDIO=true enables native audio decoding and joint video/audio understanding. Use this integrated route when the model’s combined output fits the task; use separate extraction when you need independently searchable transcripts or page objects. NeMo Retriever’s Video and frame OCR path can combine speech processing with visual text extraction from frames or derived images.

Decode video into the frames you need

DALI’s nvidia.dali.fn.readers.video uses FFmpeg to parse containers and NVDECODE for hardware-accelerated video decoding. Its gpu backend returns batches shaped (N, F, H, W, C): batch, frames, height, width, and channels. sequence_length selects the frames per sequence; stride sets the distance between consecutive selected frames.

This reader supports constant-frame-rate video. Use skip_vfr_check only to suppress false positive detection, rather than as a general compatibility fix: route incompatible recordings through a suitable preparation path before using this reader. Retain the source interval alongside each extracted sequence so an analyst can recover the original segment.

A speech branch answers a different question. NeMo Retriever’s audio/video workflow uses the parakeet-1-1b-ctc-en-us ASR NIM for speech transcription. Send the visual track through frame processing when the task asks about actions, objects, or screen contents; use the transcription route for spoken statements. A meeting companion needs both kinds of information, whether an Omni model processes them jointly or separate branches produce them.

Select the image decoder by format

DALI’s nvidia.dali.fn.decoders.image offers cpu and mixed backends. The mixed path provides GPU-accelerated decoding for JPEG and JPEG2000; other supported formats use a CPU decoder. For JPEG, dedicated hardware decoding is used when available. Therefore, replacing JPEG inputs with PNG can change the bottleneck even when both formats are accepted.

A mixed operator still has CPU work. Reserve host processing capacity rather than interpreting an idle GPU as proof that decoding is cheap. If throughput falls after a format change, inspect the input distribution and decoder path before adding inference replicas. An image decoder produces pixels; optical character recognition (OCR) and image captioning are later extraction operations. On the mixed path, the format decides between GPU-accelerated decoding and a CPU decoder, and both produce pixels for later extraction operations.

Separate audio decoding from speech inference

DALI’s nvidia.dali.fn.decoders.audio decodes WAV, FLAC, and OGG on its cpu backend and returns waveform data plus sampling rates. Its output is not a transcript. For a custom signal-processing pipeline, nvidia.dali.fn.spectrogram supports a gpu backend and expects a single-channel float32 signal. Moving that transform to GPU accelerates a later stage; it does not change the audio decoder’s backend.

For NeMo Retriever speech ingestion, install the nemo-retriever[multimedia] extra and provide the system ffmpeg and ffprobe binaries. The Python wrapper does not install those executables. In Helm, enable nimOperator.audio.enabled and configure serviceConfig.nimEndpoints.audioGrpcEndpoint; enabling the NIM alone leaves the service’s audio endpoint unwired. In batch mode, pass the endpoint through ASRParams.audio_endpoints; graph ingest does not read the Helm service endpoint.

The documented self-hosted Parakeet route needs an additional dedicated GPU. Its support matrix excludes B200, other Blackwell GPUs, and H200 NVL for that transcription path, while core document extraction has different support. If the audio NIMService stays not Ready or enters CrashLoopBackOff during engine building on an excluded GPU, use a supported dedicated GPU such as H100 or A100. Repeated restarts do not change hardware support.

Route documents to extraction services

NeMo Retriever Library’s pdfium path combines document processing with page-layout and OCR stages; scanned or mixed pages can use pdfium_hybrid or ocr. Choose the extraction method for the content, as 6.3.1 explains, then provision the services it invokes. Native text extraction and GPU model inference are separate capacity consumers.

method="nemotron_parse" requires the nemotron-parse extra and does not produce chart modality rows. Keep chart detection on the documented pdfium layout/OCR path when downstream retrieval needs chart-classified objects. A visually rich document is not a reason to send every page through the same expensive parser.

Key distinctions and trade-offs

Compare the paths by their output and accelerator boundary:

Input	Deciding condition	Mechanism	Cost or failure boundary
Video	The question depends on visible events	VSS RT-VLM; DALI `readers.video`	VSS integrates decode and inference; DALI requires constant frame rate
Images	Pixels must reach visual inference or OCR	`decoders.image`, `device="mixed"`	JPEG/JPEG2000 acceleration; accepted formats can still decode on CPU
Audio	The question depends on speech or signal features	Parakeet ASR; alternatively `decoders.audio` then `spectrogram`	ASR needs its supported GPU allocation; DALI audio decoding remains CPU work
Documents	The agent needs text and structured page objects	`pdfium`, `pdfium_hybrid`, `ocr`, `nemotron_parse`	Extraction services consume GPU capacity; parser choice changes output modalities

NeMo Retriever’s throughput guidance recommends stage-specific measures — tables/sec, charts/sec, OCR pages/sec — paired with how many such elements each page holds. File size alone is a poor admission estimate: a small PDF with dense tables can cost more to extract than a larger native-text document.

Architecture / decision scenario

An insurance claims assistant receives 120 evidence bundles per hour. Each includes a short constant-frame-rate inspection video, JPEG or PNG photographs, an adjuster’s audio note, and a PDF estimate. The organization requires all processing on premises and a five-minute ingestion target. These are design assumptions, not vendor benchmarks.

The tempting proposal is to send everything through one visual model on the team’s B200 pool. An Omni model can combine speech and video, but it does not replace the required extraction of the estimate’s table objects. The pipeline owner chooses independently searchable speech and document outputs; the documented Retriever Parakeet path needs a supported GPU outside that B200 pool.

The media engineer selects VSS RT-VLM for inspection video and photographs; DALI remains an alternative for explicit preprocessing; on that path, PNG photographs decode on CPU. The document owner uses the pdfium layout path for the estimate’s tables and OCR. The platform owner assigns speech to a supported dedicated H100 and wires the audio endpoint. Video sampling density belongs to 7.5.2.

A trial records 90 seconds at p95 for native-text estimates but 240 seconds for scanned estimates. The team therefore measures OCR pages per second separately instead of pooling both classes under one files-per-second rate. Because the cluster blocks package downloads, it sets service.installFfmpeg=false and embeds FFmpeg in its service image, as the deployment instructions prescribe. Adding GPU capacity would not repair missing binaries or an unwired audio endpoint.

Common exam traps

Treating every operation in an accelerated pipeline as a GPU operation.
Confusing accepted image formats with GPU-accelerated image decoding formats.
Using a speech transcript as a substitute for visual video analysis.
Assuming a GPU supported for PDF extraction also supports the optional ASR NIM.
Enabling nimOperator.audio.enabled without configuring the audio endpoint.
Choosing nemotron_parse when the required output includes chart modality rows.

What to remember

Route by required information, then by format and hardware support. Video decode, image decode, audio decoding, speech recognition, and document extraction have distinct execution boundaries. Provision the CPU stages, GPU services, system binaries, and endpoint wiring that the selected route actually uses.