Realtime API bugs tracking

[thekid]

• (Azure AI only): No response for session.update request
• First response contains JSON when using text-only - happens frequently
• Assistant answers with random data, ignoring what the user says (also for 1st request in text-only mode)
• Nonsensical image placeholders (<|is_landscape_image|><|clip|><|vq_clip_2420|>...) on rare occasions

I found that if you input only text, it is very unreliable, I also found bugs where it would return seemingly random image information or unrelated text.
Source

See also

• https://community.openai.com/t/realtime-api-potential-content-bleed-and-costs-generating-bugs/981561
• https://community.openai.com/tag/realtime and https://community.openai.com/tag/api-realtime

[thekid]

First response contains JSON when using text-only - happens frequently

It seeming helps to set modalities to text only in the session instead of just supplying it along with the response.create message.

$api->transmit([
  'type' => 'session.update',
  'session' => [
    'instructions'   => 'You are LernGPT.',
    'turn_detection' => null,
    'modalities'     => ['text'],
  ],
]);