openai-realtime: OpenAIRealtimeWebRTC conversation.item.retrieve has no audio for model responses

[leecjohnny]

Describe the bug

When using openai-realtime webrtc transport, conversation.item.retrieve does not retrieve audio for model responses

Debug information

"@openai/agents": "^0.1.6",
"@openai/agents-openai": "^0.1.6",
"@openai/agents-realtime": "^0.1.6",

Repro steps

1. start RealtimeSession with webrtc transport
2. have at least 1 audio model response (assistant) in history
3. get the item id for the model response
4. send conversation.item.retrieve for the id
5. inspect the conversation.item.retrieved event for the item for base64 audio content

Expected behavior

If the model provided a response, the server representation returned by conversation.item.retrieved should have the audio in base64. The model is responding in audio, so the server representation should be in audio.

This is according to your docs:

retrieve event call: https://platform.openai.com/docs/api-reference/realtime-client-events/conversation/item/retrieve

retrieved event response: https://platform.openai.com/docs/api-reference/realtime-server-events/conversation/item/retrieved

[seratch]

Thanks for sharing this. I confirmed the event data includes only transcript but there is no audio data. Will share this with the relevant team members.

Code snippet:

Modified https://github.com/openai/openai-agents-js/tree/main/examples/realtime-next to have the following additional code

    session.current.on('transport_event', (event) => {
      if (event.type === 'response.output_audio.done') {
        console.log('response.output_audio.done', event);
        session.current?.transport.sendEvent({
          type: 'conversation.item.retrieve',
          event_id: event.event_id,
          item_id: event.item_id,
        });
      }
    });

Responses:

{
  "type": "response.output_audio.done",
  "event_id": "event_CKD5wmYgNHDUhB0nbOsFr",
  "item_id": "item_CKD5vRcdZI28pzwKcs2Ze",
  "content_index": 0,
  "output_index": 0,
  "response_id": "resp_CKD5v0JS7hsqEGu8elch7"
}

the returned client event by the above code snippet:

{
  "type": "conversation.item.retrieved",
  "event_id": "event_CKD5waOFRRPHuNmJJc7JU",
  "item": {
    "id": "item_CKD5vRcdZI28pzwKcs2Ze",
    "content": [
      {
        "transcript": "Good morning! How can I help you today?",
        "type": "output_audio"
      }
    ],
    "role": "assistant",
    "status": "completed",
    "type": "message"
  }
}

[juberti-oai]

this appears to be a regression on the GA interface, it works properly when using the beta interface