NVIDIANVIDIA-Certified Professional: Agentic AI2026-102.5 Develop dynamic conversation flows with real-time streaming and feedback mechanisms.
Route streamed tokens and steps separately
Completed by the knowledge check
Learning objective
Stream tokens and intermediate steps over SSE or WebSocket.
Core idea
A conversation stream carries several kinds of information: answer fragments, workflow progress, interaction requests, and a terminal outcome. Your client must interpret each kind before displaying it. Receiving bytes quickly improves perceived responsiveness, but does not prove that a tool succeeded or that the answer is complete.
NeMo Agent Toolkit exposes HTTP streaming endpoints and a WebSocket message schema. Choose the transport by the exchange your application needs, then implement its message lifecycle. See 1.1.2 for choosing useful status text; this lesson covers the transport and parsing decisions that deliver that text correctly.
How it works
Select a workflow interface
Start the configured workflow with nat serve --config_file CONFIG_PATH, replacing CONFIG_PATH with its configuration file. The FastAPI front end exposes /v1/workflow/stream for the workflow-defined schema and /v1/chat/stream for chat-shaped input. Their legacy counterparts are /generate/stream and /chat/stream. The API reference marks the versioned paths experimental because of HTTP human-in-the-loop and OAuth support.
The /v1/chat/completions endpoint selects streaming through the request field stream: true. Do not confuse that switch with the separate /v1/chat/stream route. A client configured for one response shape needs an explicit adapter before consuming another. In particular, the toolkit’s chat-stream example contains intermediate_data alongside answer data; an adapter that assumes every object is an answer fragment can either crash or display internal execution details as prose.
With Server-Sent Events (SSE), the server sends events through an HTTP response. Interactive input travels through a separate request. NVIDIA documents interaction_required and oauth_required typed events for interactive HTTP streaming. The first includes execution_id, interaction_id, prompt, and response_url. Keep those events out of the answer buffer. See 2.5.3 for consuming the user’s response.
Parse SSE framing before its payload
The workflow streaming response separates data output from intermediate_data steps; see 1.1.2 for raw IntermediateStep objects, filtering, and safe presentation. Here the transport-specific job is to recognize an event before interpreting its payload.
NVIDIA’s interactive SSE examples put the type on an event: line and the JSON payload on a following data: line. For example, event: interaction_required precedes the object carrying the prompt and response URL; event: oauth_required precedes the authorization object. The Python client dispatches with line.startswith("event: interaction_required"), reads the following data line, and parses its JSON before making the response POST.
Preserve that distinction in your client adapter. A line naming an event is not JSON, and a JSON object describing a human prompt is not an answer token. The example handles typed interaction events before its ordinary data: chunk branch. If your client only parses lines beginning with data:, it loses the explicit event type that tells it whether to render text, ask a question, or start authorization.
The client adapter recognizes an event before interpreting its payload.
For /v1/chat/completions, set stream: true for output streaming and general.front_end.enable_interactive_extensions: true for interactive extensions. These are separate switches: enabling token streaming alone does not opt that endpoint into interaction events. The versioned workflow and chat endpoints have interactive extensions enabled by default.
Select the WebSocket response schema
See 1.1.2 for WebSocket message types, identity correlation, and completion status. Over this transport, a user_message also carries schema_type, which the schema reference defines as the response schema for a workflow.
Choose the response schema expected by your client per message; do not infer it from the fact that the connection is a WebSocket. Validate the adapter against that workflow’s documented response structure. A socket carries messages in both directions, but it does not turn different workflow response shapes into one interchangeable format.
For a browser client that cannot use header- or cookie-based authentication, the schema supplies auth_message. The server associates the resolved identity with the session and returns auth_response_message with status: "success" or status: "error". If your integration uses this authentication path, authenticate each replacement connection and require a successful response before sending workflow messages. Reconnecting restores transport access; recovering the interrupted turn remains the separate decision in 2.5.2.
Bound presentation work
Measure first visible output separately from final completion in the client integration. A tool-heavy workflow can emit a progress step before it can emit answer text. Count those as different milestones so that a fast progress indicator does not conceal slow answers.
Choose which events the client consumes rather than forwarding every diagnostic event to every viewer. filter_steps reduces the full-stream workload, while dispatch by message type keeps step rendering separate from answer rendering over WebSocket. These mechanisms reduce unnecessary presentation work; they do not make the underlying model generate fewer tokens. Repeatedly restarting generation to repair a display problem adds inference work instead of repairing the parser.
Key distinctions and trade-offs
The two transport choices divide by the required interaction pattern:
| Option | Deciding condition | Mechanism | Cost |
|---|---|---|---|
| HTTP SSE | Server updates fit an HTTP response and client input can use another request | /v1/chat/stream; typed interaction events and response_url | One long-lived response per active turn plus a separate POST per interaction |
| WebSocket | The session needs messages in both directions on one connection | system_response_message, system_interaction_message, user_interaction_message | One socket per connected session; replacement connections need authentication |
Streaming changes delivery timing. Background execution changes how a job outlives the request; see 1.8.3. Neither choice by itself supplies recoverable conversation state.
Architecture / decision scenario
An insurance research assistant serves 180 concurrent analysts through a network that permits HTTP streaming but blocks WebSocket upgrades. The product target is visible progress within 2 seconds and a complete research answer within 20 seconds. In a representative test, retrieval produces its first step after 0.7 seconds, while answer generation begins after 4 seconds.
The frontend owner chooses SSE through /v1/chat/stream, rather than the proposed WebSocket-only client. The network constraint rules out that alternative even though it supports bidirectional messages. The client routes intermediate steps to a status area, answer content to the response, and interaction_required to an input control. It reports 0.7 seconds to first progress and 4 seconds to first answer, so the team can distinguish the two targets.
With 180 active turns, the SSE design holds 180 streaming responses open; each human interaction adds a response POST. A WebSocket design would hold 180 session sockets and, for the message-authentication path, authenticate replacement connections with auth_message. The network constraint still decides the choice.
For internal diagnostics without HITL, the workflow owner can use legacy /generate/full?filter_steps=LLM_END,TOOL_END, which is compatible with the experimental versioned path for that use. If the answer stream fails after 90 words and before any interaction event, the client marks it incomplete and reconciles the durable workflow thread as described in 2.5.2. It has no returned interactive execution ID to query.
Common exam traps
- Treating every stream event as an answer token.
- Reading
LLM_ENDas completion of the whole workflow. - Assuming SSE prevents human input; interactive HTTP uses a response POST.
- Choosing WebSocket without checking the deployment network.
- Expecting interaction events from
/v1/chat/completionswithout enabling interactive extensions. - Equating an open streaming connection with durable execution state.
What to remember
Select the transport, select the endpoint schema, and route each event by its role and identity. Keep intermediate steps separate from answer content and preserve incomplete or failed outcomes. Measure first progress, first answer, and final completion independently.
Knowledge check
A claims-research assistant runs behind a corporate proxy that allows long-lived HTTP responses but blocks WebSocket upgrades. The workflow sometimes pauses to ask the analyst which policy period to use, and the analyst must see progress while it runs. Which client design delivers both the stream and the analyst’s answer?
user_interaction_message/v1/chat/completions with stream: true and default front-end settings/v1/chat/stream and POST the answer to the event’s response_url/generate/full and send the analyst’s answer as the next chat turnresponse_url that interaction_required carries. The WebSocket design is ruled out by the proxy, not by any lack of capability, and /v1/chat/completions emits no interaction events until interactive extensions are enabled.A client streams from /v1/chat/completions with stream: true. The team adds a step that prompts the user, but logging the raw response shows no event: lines at all, and the team must keep this endpoint. Which change makes the pause reach the client?
general.front_end.enable_interactive_extensions: truestream: false so the server can answer 202 Acceptedexecution_id with the request so the server can pause itfilter_steps to the request so interaction steps are keptstream: true only streams output, and enable_interactive_extensions defaults to false . The versioned workflow and chat routes have the extensions enabled by default, which is why the same workflow can pause there and not here.A custom SSE client reads only lines that begin with data: and appends every parsed JSON object to the answer. When the workflow pauses for input, users see an object containing a prompt and a URL inside the answer text. What should the client change?
intermediate_data fieldprompt and response_url fields before appendingevent: line before parsing the following dataevent: line before parsing the following data: payload. That separates an interaction prompt from answer content; parsing JSON alone is not enough if every object is then appended to the answer. Restarting generation adds inference work without repairing the parser, and stripping the fields hides the prompt while the workflow waits for an answer nobody is asked for.A tool-heavy research workflow emits a retrieval step after about a second but its first answer text several seconds later. The product sets one target for visible progress and another for a complete answer, and the client records a single “time to first stream event”. Which measurement supports both targets?
LLM_END step reaches the clientLLM_END marks the end of one model call, not of the workflow, and comparing transports measures delivery rather than the two targets.References
- NVIDIA, NVIDIA NeMo Agent Toolkit API Server Endpoints
- NVIDIA, WebSocket Message Schema
- NVIDIA, HTTP Interactive Execution