Building a Voice-Powered RAG Agent with NVIDIA Nemotron Models

Why it matters

A voice agent chains several models, and each one adds latency and a place to fail. Building the chain end to end shows where speech, retrieval and vision meet the agent loop.

What to practise

Transcribe audio with nemotron-speech-streaming-en-0.6b, embed text and images with llama-nemotron-embed-vl-1b-v2 into a FAISS index, and rerank results with llama-nemotron-rerank-vl-1b-v2. Wrap retrieval as a search_knowledge_base tool for a LangChain create_agent loop on nemotron-3-nano-30b-a3b, which calls nemotron-nano-12b-v2-vl only when a retrieved document is an image. Watch where Llama 3.1 Nemotron Safety Guard checks the transcript before the agent runs and the answer after it finishes. The notebook transcribes a finished recording and answers in text.

How it fits

The on-demand vision call is 2.2.1’s choice between sending an image to a model and retrieving over it. The notebook stops at a recorded clip, so 2.2.2’s streaming recognition, speech synthesis and turn budget are the parts to design beyond it.