Latest output_audio transcripts item in session.history is removed when a user interrupts

[TrevorJamesH]

Please read this first

• Have you read the docs? Yes
• Have you searched for related issues? Yes

Describe the bug

A clear and concise description of what the bug is.

In a realtime twilio session any user speech triggers interrupt, even when the agent isn’t speaking, which forces truncate → retrieve and drops assistant output_audio item transcripts.

Debug information

• Agents SDK version: v0.1.4
• Runtime environment Node.js v20.10.0

Repro steps

Log on audio_interrupted and history_updated during a twilio call. Watch audio_interrupted always fire whenever the user speaks, and watch the previously present assistant transcript get dropped due to truncate.

Expected behavior

The audio_interrupted event should only fire when the assistant is actually being interrupted. Not any time the user speaks.

We should have the option not to truncate even during an interrupt in case we still want to store the transcript for tracing.

Ideally partial transcripts are implemented on interrupt or truncate events.

[seratch]

Thanks for sharing this. I think when a user starts speaking, acknowledging that usually means interrupting the real-time agent, so pausing both audio and transcription makes sense for most cases. That said, there could be a use case like the one you mentioned here. Having a customization option might be worth considering in the long run.

At the same time, I feel a chained approach (speech-to-text → text-to-text → text-to-speech pipeline) might be more suitable for your scenario. While this TS SDK doesn’t support it out of the box just yet, you can try it with the Python SDK here: https://openai.github.io/openai-agents-python/voice/quickstart/

We may add something similar to the TS SDK in the future, but for now, if you want to implement the chained approach in TS, you’ll need to connect the gpt-4o-transcribe model, a text model (like gpt-4.1), and a TTS model in your code.

[TrevorJamesH]

The core issue is the current behavior results in all of the assistant output_audio transcripts being removed from the session.history. It is not unreasonable to expect the history to have a the transcripts of the conversation, especially because they were all once there. The most recent output_audio transcript remains, because it is not truncated until the user speaks, then its removed.

I'd expect the history to be that, a history. I expect it not to be destroyed.

Easy: We can mark that output_audio as interrupted rather than needlessly deleting its transcript.

Medium: We can tell if the user speaking is an actual interruption of the agent speech and skip the interrupt path if its not.

Hard: We can reconstruct the partial transcript on interrupt. Even then, I'd expect to still have the intended full text transcript for the sake of tracing.

[TrevorJamesH]

Can you tell me if this is honestly the intended effect? That output_audio transcripts should always be removed whenever the user speaks? Or is this a bug / unintended side effect?

[seratch]

Thanks for sharing more details. Removing the assistant's output_audio & transcript message data after having it in session.history indeed sounds like a bug. I haven't identified the cause and the viable solution for it, but it looks something this SDK should improve.

[TrevorJamesH]

Right, it seems like a bug to me too, which is why I tagged this issue as a bug when I wrote it.

I tried to break down the code path in my description.

User speaking, input_audio_buffer.speech_started, always fires the interrupt event.

Interrupt always fires the truncate event, conversation.item.truncate

Truncate always fires retrieve, which appears to get the copy of the event from the realtime server, which is missing the transcript.

That fires item_update

Which fires history_update which replaces the event with the one now missing the transcript.