[RFC] Multi-Stage Abort / Barge-in for Omni Models

[lishunyang12]

Motivation

In voice agent scenarios, users frequently interrupt ("barge in") while the agent is still speaking. The current streaming TTS WebSocket (PR #1230) has no mechanism for the client to cancel in-progress audio generation mid-sentence.

Without interruption support, the server continues generating audio for the remaining sentences even after the user has started speaking again, wasting GPU resources and adding latency to the next response.

Use Cases

• Voice assistants: User says "stop" or starts a new question while the agent is mid-sentence — the agent should immediately stop speaking and listen
• Contact center bots: Caller interrupts with "actually, never mind" — bot should stop current TTS and process the new input
• Interactive gaming: Player takes an action mid-dialogue — NPC should stop current line and react

Proposed Approach

Add an input.cancel message type to the WebSocket protocol:

// Client sends to cancel current generation:
{"type": "input.cancel"}

// Server responds:
{"type": "audio.cancelled", "sentence_index": 2, "reason": "client_cancel"}
// Then resumes listening for new input.text or input.done

Behavior

1. input.cancel immediately aborts generation for the current sentence
2. Any buffered sentences that haven't started generating are discarded
3. Server sends audio.cancelled and returns to the text-receiving state
4. Client can then send new text or input.done

Session Lifecycle with Cancel

config → text → text → cancel → text → done → session.done
                  ↓        ↓        ↓
             audio.start  audio.cancelled  audio.start → audio → audio.done

Implementation Considerations

• Need to propagate cancellation to the engine (abort_request)
• Race condition: cancel arrives while audio is already being sent — server should stop sending remaining chunks
• Should input.cancel cancel only the current sentence or all pending sentences?
• Consider adding a cancel_scope field: "current" (default) vs "all"

Related

• PR [Feature][TTS] Streaming Text Input for Qwen3-TTS via WebSocket #1230 — Streaming text input (foundation)
• OpenAI Realtime API supports response.cancel for similar purpose

[lishunyang12]

Reframing this RFC — barge-in is primarily an Omni model concern, not standalone TTS.

For standalone TTS (simple):
Adding input.cancel → abort_request() is a small addition to #1230's WebSocket handler. Doesn't need a separate RFC — can be a follow-up PR to #1230.

For Omni models (this RFC's real scope):
Qwen-Omni runs a multi-stage pipeline: encoder → thinker → talker → code2wav, orchestrated by AsyncOmni with inter-stage communication via OmniChunkTransferAdapter / SharedMemoryConnector. Barge-in needs to:

1. Abort across all stages — propagate cancellation through the chunk transfer adapters to kill thinker (stop reasoning), talker (stop token generation), and code2wav (stop audio decoding)
2. Flush pipeline state — clear any buffered chunks in the transfer adapters so stale audio from the cancelled response doesn't leak into the next one
3. Resume listening — return the encoder stage to ready state for new audio input

Open questions:

• How does abort_request() propagate through OmniChunkTransferAdapter today? Does it reach all stages or just stage 0?
• What happens to in-flight shared memory chunks when a request is aborted mid-transfer?
• Should barge-in be server-detected (VAD on incoming audio) or client-triggered (input.cancel), or both?

This depends on the async chunk infrastructure (#1423, #1415) being stable first.

[Sy0307]

I think the idea of this RFC is good. However, based on my practical production experience, in real-time audio/video conversations, the client side (i.e., the audio capture side) typically has barge-in detection enabled. Under normal circumstances, audio is fed in incrementally in chunks of no more than 100ms at a time. These mechanisms ensure that the audio being transmitted is mostly valid/meaningful. We may need to evaluate the actual real-world benefits of this 'Barge-in' feature in practical scenarios. For your reference.

[lishunyang12]

That's a fair point. In that case the server just needs a simple input.cancel → abort_request() path — client handles the detection, server handles the cleanup. Probably simple enough to fold into #1230 as a follow-up rather than a separate RFC.

Curious about your production setup — how does your client handle the transition after barge-in? e.g., does it wait for the server to acknowledge the cancel before sending new audio, or does it just start sending immediately and let the server sort it out?

[Sy0307]

Curious about your production setup — how does your client handle the transition after barge-in? e.g., does it wait for the server to acknowledge the cancel before sending new audio, or does it just start sending immediately and let the server sort it out?

Upon detecting a voice interruption, the client immediately captures and sends the audio without waiting for any commands from the server. At the application layer, the server is typically responsible for discarding all audio generated in the previous round.