Synchronize speech within a turn budget

Learning objective

Build speech pipelines (ASR, agent, TTS) and synchronize streaming audio within latency budgets.

Core idea

A voice agent joins three different streams: automatic speech recognition (ASR) produces transcripts, the agent produces answer text, and text-to-speech (TTS) produces audio. Receiving something early from one stage does not establish that the next stage can safely consume it. A provisional transcript can change, a text fragment can end mid-phrase, and synthesized audio can remain queued before anyone hears it.

Budget the interval from the end of the speaker’s utterance to the first audible answer. Also measure completion and playback continuity. NVIDIA Speech NIM exposes the Riva ASR and TTS gRPC APIs; the application owns the ordering and timing between them. See 1.1.1 for choosing voice as a surface and correcting transcripts, 2.5.1 for general workflow streaming and 2.5.2 for dialog interruption and reconnection. Here the decision is when to move transcript, text, and audio across stage boundaries.

How it works

Recognize incrementally without committing tentative words

The ASR gRPC method StreamingRecognize accepts a stream of StreamingRecognizeRequest messages and returns streamed recognition responses. Send streaming_config first, without audio. Subsequent messages contain sequential audio_content chunks encoded as declared in RecognitionConfig. Set sample_rate_hertz to the actual input rate; changing the declaration does not change the captured bytes.

Set StreamingRecognitionConfig.interim_results=true to receive tentative hypotheses. A result with is_final=false can change. A result with is_final=true is final for that portion of audio, not necessarily the entire conversation. Assemble finalized portions in order, and use interim text for a replaceable transcript display. Do not append every hypothesis as if it were another spoken sentence.

The stability field estimates whether an interim hypothesis will change; it is not an authorization to execute the recognized instruction. Even a value of 1 is distinct from is_final=true. For an action-bearing turn, the orchestrator should wait for the finalized utterance before submitting its text to the agent.

Endpointing contributes directly to latency. RecognitionConfig.endpointing_config exposes stop_history, a detection window in milliseconds, and stop_threshold, the blank-token fraction used to detect the end of an utterance. Shortening that window trades waiting time against premature turn splits. Tune it with recordings that include hesitations and pauses, rather than treating silence detection as a free latency reduction.

For two-pass end of utterance, stop_history_eou and stop_threshold_eou trigger a first pass that produces a partial transcript with stability=1; stop_history_eou is shorter than stop_history. The orchestrator can use that first pass for speculative, read-only retrieval, discarding the result if the finalized utterance differs. Keep action-bearing calls behind is_final=true. This trades potentially wasted retrieval work for less delay after finalization, without treating the first pass as final input.

For an action-bearing turn, the agent receives the finalized utterance after is_final=true, while a tentative hypothesis and the first pass serve display and read-only retrieval.

Turn answer text into speakable segments

For a NIM-hosted agent model, POST /v1/chat/completions with stream:true streams generation. An adapter between the agent and TTS should accumulate answer text into speakable segments. Sending every token immediately minimizes buffering but can split words, punctuation, or a qualification that belongs with its sentence. Waiting for the entire answer sacrifices overlap between generation and synthesis.

Speech NIM offers gRPC SynthesizeOnline, which accepts streamed SynthesizeSpeechRequest messages and returns audio as it becomes available. Set text, language_code, voice_name, encoding, and sample_rate_hz for the intended voice and playback path. The request’s text limit is 2,000 characters after normalization, or 500 for Chatterbox TTS Multilingual. Split longer content at sentence or paragraph boundaries rather than retrying an oversized request unchanged.

For the TTS WebSocket realtime interface, send input_text.append to accumulate text, input_text.commit to commit the buffer, and input_text.done when input ends. The last event processes remaining text; it is not a discard command. The session’s custom_configuration offers flush, chunk_len_threshold, and max_chunk_threshold for plain-text flushing. The gRPC reference scopes these controls to Magpie: do not assume they apply to every TTS backend. Forced flushing reduces waiting but gives synthesis less surrounding text for prosody.

Keep audio order separate from arrival and playback

The realtime TTS server emits conversation.item.speech.data with base64-encoded audio, an item_id, a content_index, and an is_last_chunk marker. It also emits conversation.item.speech.completed. Associate those events with the application turn and play each item’s chunks in order. The completion event describes synthesis completion; record a separate client playback-complete time after the queued samples have played.

Use a bounded playback queue measured in milliseconds of audio, not the number of network messages. Chunk sizes can differ. In an illustrative client policy, pause additional text commits when queued audio exceeds 500 ms and resume below 200 ms. These are application watermarks, not Speech NIM defaults. They limit how far synthesis runs ahead of what the listener hears and reduce stale queued speech when a turn is replaced.

Match the playback decoder to the configured output format and sample rate. A documented failure on the HTTP streaming endpoint /v1/audio/synthesize_online is static or an unplayable file: its output is raw LPCM without a WAV header. Supply the correct format metadata to the player, or convert the raw output to WAV using the model’s sample rate. A transport chunk is not automatically a self-contained audio file.

Key distinctions and trade-offs

Choose each boundary independently:

An alternative accepts audio directly through an audio_url content item, a vLLM-specific extension requiring an audio-capable model. This bypasses separate ASR and its is_final and endpointing controls. For a text answer from that path, TTS and ordered playback still remain; choose the cascade when explicit transcript finality controls when the agent acts.

Measure endpoint delay, agent time to the first speakable segment, TTS time to first audio, transport, and client buffering. Their measured contributions locate the bottleneck. Track the joint distribution under concurrent sessions: adding independent component p95 values does not calculate an end-to-end p95. A fast first token does not compensate for a long TTS text buffer.

Architecture / decision scenario

A hotel concierge voice assistant must start an audible answer within 1,200 ms of a guest finishing a request. In an illustrative trace with 20 concurrent sessions, ASR endpointing and finalization take 300 ms, the agent’s first sentence takes 350 ms, TTS first audio takes 180 ms, and transport plus playback buffering take 150 ms. The 980 ms total leaves 220 ms against the target for that trace.

The proposed alternative waits for the complete agent answer before starting TTS. Measured full-answer generation takes 1,400 ms, making the same path total 2,030 ms. The voice application owner chooses progressive sentence synthesis and bounded playback buffering. The speech team tunes endpointing_config against guest pauses; it rejects triggering the agent from every interim transcript because a revised room number would launch a second, conflicting request.

The adapter commits speakable text, consumes audio events in order, and measures when playback actually starts. For a long concierge explanation, it chunks TTS input within the selected backend’s normalized-text limit. If concurrent-load traces miss the target, the team identifies whether endpointing, agent work, synthesis, or queued audio grew before changing a setting. It does not solve a TTS buffer delay by accepting unstable ASR input.

Common exam traps

• Treating stability=1 as identical to is_final=true.
• Sending audio in the first StreamingRecognizeRequest.
• Equating first agent token, first synthesized byte, and first audible sample.
• Assuming Magpie flush controls apply to every TTS model.
• Passing an oversized TTS request unchanged after a length rejection.
• Treating raw HTTP streaming audio as a complete WAV file.

What to remember

Finalize the utterance before committing its meaning, stream answer text in speakable segments, and preserve audio order through playback. Endpointing, text flushing, synthesis, and buffering all spend the same turn budget. Measure the complete audible path under load rather than substituting a model’s first-token latency for the voice experience.

Knowledge check

References

• NVIDIA, ASR gRPC API Reference
• NVIDIA, NIM API Reference
• NVIDIA, TTS gRPC API Reference
• NVIDIA, TTS Realtime API Reference
• NVIDIA, Troubleshooting NVIDIA TTS NIM Microservice Issues
• NVIDIA, Image, Audio, and Video Input