WIP: Route playback through VPIO for echo cancellation on iOS#264
WIP: Route playback through VPIO for echo cancellation on iOS#264kavimathur wants to merge 6 commits intolzell:mainfrom
Conversation
Put AVAudioEngine in manual rendering mode on iOS (no headphones) so that the VPIO output render callback can pull rendered playback audio and feed it to the speaker through Bus 0. This gives the VPIO full visibility into both input and output for acoustic echo cancellation (AEC), preventing the AI from hearing its own voice through the speaker and self-interrupting. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
| kAudioOutputUnitProperty_EnableIO, | ||
| kAudioUnitScope_Output, | ||
| 0, | ||
| &zero, // <-- This is not a mistake! If you leave this on, iOS spams the logs with: "from AU (address): auou/vpio/appl, render err: -1" |
There was a problem hiding this comment.
Do you see the console getting spammed with from AU (address): ... render err: -1 with this change?
There was a problem hiding this comment.
Yeah this fixes that console spam - if I understand correctly, the error was occurring because the VPIO was enabled but it wasn't receiving the speaker audio data, i.e. no render source. But now we're routing both input (bus 1) and speaker (bus 0) data through VPIO so it no longer has this unfulfilled callback.
|
Have added support for OpenAI's Noise Reduction feature, which works very nicely combined with the VPIO routing. So far have tried testing in moderately noisy environments and saw no self-interrupts or background noise pickup. Need to continue testing in different environments & I/O sources, but so far this is very promising. |
2 participants
Work in progress branch to fix the problem of OpenAI Realtime hearing itself on iOS - #240
Puts AVAudioEngine in manual rendering mode on iOS (no headphones) so that the VPIO output render callback can pull rendered playback audio and feed it to the speaker through Bus 0. This gives the VPIO full visibility into both input and output for acoustic echo cancellation (AEC), aiming to prevent the AI from hearing its own voice through the speaker and self-interrupting.
Still getting some self interruption, but this improves the frequency.