[RFC] Streaming Audio Output for WebSocket TTS

[lishunyang12]

Motivation

PR #1230 adds streaming text input via WebSocket — text arrives incrementally, audio is generated per sentence. However, each sentence's audio is returned as a single binary frame only after full synthesis completes.

For voice agents and real-time applications, users need streaming audio output — chunked PCM frames sent progressively as the model generates, enabling playback to start before synthesis finishes.

Use Cases

• Voice assistants: User hears the first audio chunk within ~200ms of sentence completion, rather than waiting 1-5s for full sentence synthesis
• Contact centers: Reduces perceived latency for callers, where every 100ms of silence increases drop-off rates
• Live translation/dubbing: Audio must stay synchronized with video — batch-per-sentence adds unacceptable delay for long sentences

Design Decisions

• Format: Raw PCM (16-bit signed, 24kHz mono). No WAV — headers require total file size upfront, which is incompatible with streaming. This matches OpenAI Realtime API and [Qwen3TTS][Feat] Streaming output #1438's approach.
• Chunk size: Follows model codec — one chunk per Code2Wav decode window (25 frames by default, configurable via Make chunk_size and left_context_size configurable via YAML for async chunking #1423). No arbitrary byte/duration splitting needed.
• Scope: Small follow-up to [Feature][TTS] Streaming Text Input for Qwen3-TTS via WebSocket #1230. Wire the streaming generation path from [Qwen3TTS][Feat] Streaming output #1438/support qwen3 tts streaming output #1189 into the WebSocket handler, replacing the blocking _generate_audio_bytes() call.

Protocol

// Client sends:
{"type": "session.config", "voice": "Vivian", "stream_audio": true}

// Server sends (per sentence):
{"type": "audio.start", "sentence_index": 0, "format": "pcm", "sample_rate": 24000}
<binary: audio chunk 1>   // one Code2Wav decode window
<binary: audio chunk 2>
...
<binary: audio chunk N>
{"type": "audio.done", "sentence_index": 0, "total_bytes": 96000}

stream_audio defaults to false for backward compatibility. When stream_audio: true, format must be pcm.

Implementation

1. In OmniStreamingSpeechHandler ([Feature][TTS] Streaming Text Input for Qwen3-TTS via WebSocket #1230), replace _generate_audio_bytes() with the streaming generation path
2. Each time Code2Wav produces a chunk, send it as a binary WebSocket frame
3. audio.start / audio.done framing stays the same — just multiple binary frames between them instead of one

Dependencies

• Blocked on [Qwen3TTS][Feat] Streaming output #1438 (REST streaming output) or support qwen3 tts streaming output #1189 (model-level streaming) merging first
• Builds on [Feature][TTS] Streaming Text Input for Qwen3-TTS via WebSocket #1230 (WebSocket endpoint)
• Chunk size configurability from Make chunk_size and left_context_size configurable via YAML for async chunking #1423

Related

• PR [Feature][TTS] Streaming Text Input for Qwen3-TTS via WebSocket #1230 — Streaming text input via WebSocket (foundation)
• PR support qwen3 tts streaming output #1189 — Token-level streaming in Talker (AsyncDecodingPipeline)
• PR [Qwen3TTS][Feat] Streaming output #1438 — REST streaming output (stream=true + PCM via SSE)
• PR Make chunk_size and left_context_size configurable via YAML for async chunking #1423 — Configurable chunk_size / left_context_size for Code2Wav

[Sy0307]

I have interest to get involved in this issue. Can u assign it to me?

[lishunyang12]

Thanks for the interest! Since we're both assigned now, let's split the work — I'll finalize the protocol design and you can pick up the implementation side (chunked binary frame logic in the WebSocket handler, stream_audio config plumbing, etc.).

Would be good to align on the open questions first, especially chunk size strategy and how to handle WAV headers. Any thoughts?

[lishunyang12]

After digging into the codebase and in-flight PRs, narrowing the scope here:

What already exists:

• PR support qwen3 tts streaming output #1189 adds token-level streaming at the model layer (AsyncDecodingPipeline, generate_streaming_iter)
• PR [Qwen3TTS][Feat] Streaming output #1438 adds REST streaming output (stream=true + response_format=pcm, chunked via SSE)
• PR [Feature][TTS] Streaming Text Input for Qwen3-TTS via WebSocket #1230 has the WebSocket endpoint but calls _generate_audio_bytes() which blocks until full synthesis

Revised scope: Wire the streaming path from #1438/#1189 into #1230's WebSocket handler. Concretely:

• Replace the blocking _generate_audio_bytes() call with the streaming generation path
• Send multiple binary frames between audio.start and audio.done as chunks arrive
• stream_audio: true in session config, PCM format required (same constraint as [Qwen3TTS][Feat] Streaming output #1438)

This is a small follow-up to #1230, not a standalone effort. Depends on #1438 or #1189 merging first.

The protocol design in the original description still applies — no changes needed there.

[Sy0307]

Thanks for the interest! Since we're both assigned now, let's split the work — I'll finalize the protocol design and you can pick up the implementation side (chunked binary frame logic in the WebSocket handler, stream_audio config plumbing, etc.).

Would be good to align on the open questions first, especially chunk size strategy and how to handle WAV headers. Any thoughts?

Thanks! I can pick up the implementation after your desgin.

PCM format will be better than WAV(as other Real-time API does). Chunk size can follow model codecs. I am agree that "This is a small follow-up to #1230"

[lishunyang12]

You can read the updated description and look at #1438's streaming path to prepare for implementation.

[JuanPZuluaga]

Hello, do you think it'd be useful to make TTFA configurable for the current Qwen3TTS implementation? @lishunyang12 @Sy0307 @linyueqian | The idea would be:

• we keep defining the codec_chunk_frames and lc as in: https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/model_executor/stage_configs/qwen3_tts_batch.yaml#L99C1-L101C1

still, we add some logic to reduce the TTFA, where we first flush outputs even with a lower codec_chunk_frames and then after the buffer is filled with the defined codec_chunk_frames + codec_left_context_frames we start flushing the output as it's currently supported.

I can take care of this.

[linyueqian]

Hello, do you think it'd be useful to make TTFA configurable for the current Qwen3TTS implementation? @lishunyang12 @Sy0307 @linyueqian | The idea would be:

• we keep defining the codec_chunk_frames and lc as in: https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/model_executor/stage_configs/qwen3_tts_batch.yaml#L99C1-L101C1

still, we add some logic to reduce the TTFA, where we first flush outputs even with a lower codec_chunk_frames and then after the buffer is filled with the defined codec_chunk_frames + codec_left_context_frames we start flushing the output as it's currently supported.

I can take care of this.

yes this makes sense to me. can yo work on that? i tested yesterday that the time to first audio packet is 1.9s for the 1.7b variant. we definitely need to optimize that.

[JuanPZuluaga]

Hello, do you think it'd be useful to make TTFA configurable for the current Qwen3TTS implementation? @lishunyang12 @Sy0307 @linyueqian | The idea would be:

• we keep defining the codec_chunk_frames and lc as in: https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/model_executor/stage_configs/qwen3_tts_batch.yaml#L99C1-L101C1

still, we add some logic to reduce the TTFA, where we first flush outputs even with a lower codec_chunk_frames and then after the buffer is filled with the defined codec_chunk_frames + codec_left_context_frames we start flushing the output as it's currently supported.
I can take care of this.

yes this makes sense to me. can yo work on that? i tested yesterday that the time to first audio packet is 1.9s for the 1.7b variant. we definitely need to optimize that.

Yea, for the 0.6b variant is around 1.5 seconds. Sounds good @linyueqian. I’ll make a PR soon. Would something like this make sense?

• define a tentative TTFA in initial_codec_chunk_frames
• once reached, we send the first audio chunk
• the next audio chunk, should ideally come either after initial_codec_chunk_frames x2 has been collected, or if we already reach the codec_chunk_frames + left context, we start streaming as how currently is.

ALSO, I will add support to have -1 as left context, so we always use all left context. I expect the audio quality might improve.

[linyueqian]

Hello, do you think it'd be useful to make TTFA configurable for the current Qwen3TTS implementation? @lishunyang12 @Sy0307 @linyueqian | The idea would be:

• we keep defining the codec_chunk_frames and lc as in: https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/model_executor/stage_configs/qwen3_tts_batch.yaml#L99C1-L101C1

still, we add some logic to reduce the TTFA, where we first flush outputs even with a lower codec_chunk_frames and then after the buffer is filled with the defined codec_chunk_frames + codec_left_context_frames we start flushing the output as it's currently supported.
I can take care of this.

yes this makes sense to me. can yo work on that? i tested yesterday that the time to first audio packet is 1.9s for the 1.7b variant. we definitely need to optimize that.

Yea, for the 0.6b variant is around 1.5 seconds. Sounds good @linyueqian. I’ll make a PR soon. Would something like this make sense?

• define a tentative TTFA in initial_codec_chunk_frames
• once reached, we send the first audio chunk
• the next audio chunk, should ideally come either after initial_codec_chunk_frames x2 has been collected, or if we already reach the codec_chunk_frames + left context, we start streaming as how currently is.

ALSO, I will add support to have -1 as left context, so we always use all left context. I expect the audio quality might improve.

The approach makes sense. The first chunk has no left context anyway (ctx_frames = 0), so there's no quality reason to wait for the full 25 frames. The Code2Wav trimming logic already handles variable context sizes, so the main change would be in talker2code2wav_async_chunk to emit earlier on the first chunk.

One thing to watch: very small initial chunks (5 to 10 frames) produce very short audio (~0.04 to 0.08s). Worth testing a few values (10, 15, 20) to find the sweet spot between TTFP and decoder quality.

The left_context = -1 idea is interesting too. More context should help boundary quality at the cost of more decoder compute per chunk.

Would be great if you could share some sample wav files in the PR with different initial chunk sizes so people in the community can hear the quality tradeoff.