.Net: New Feature: Add support for GPT-4o Real Time endpoint

[rboen]

Low latency conversational interactions using speech is an impressive enhancement and game changer for audio chat bots. With the emergence of the gpt-40-realtime-preview in Azure OpenAI I'd love to see an integration with the Sematik Kernel in order to facilitate agents, skills / plugins in call agents scenarios.

Please have a look at https://github.com/azure-samples/aoai-realtime-audio-sdk

[rogerbarreto]

@rboen Thanks for the ask.

We will keep track on this feature and investigate how to bring as a Speech-to-Speech streaming abstraction to SK, for now our suggestion is while we don't have this abstraction in place to use our current APIs with the breaking glass option (Providing either the OpenAIClient or AzureOpenAIClient, directly) and consuming the RealtimeConversationClient directly.

[joslat]

@rogerbarreto as this can be easily implemented, even the .NET c# code is a bit obfuscated (tried to improve this here: Azure-Samples/aoai-realtime-audio-sdk#28) it would be great to use this in the context of an Agent / Assistant where I can provide tools and a nice metaprompt.

Using it as the "UserProxy Agent" basically ;)
Otherwise we can just provide the initial prompt and that's all, also bind some tool here for function calling but the current way to do this is hyper-counter-intuitive :( - see here https://github.com/joslat/aoai-realtime-audio-sdk/blob/main/dotnet/samples/console-from-file/Program.cs (lines 28, 238 and 109...)

[jerry2007]

When will be this implemented? This could be gamechanger...

[karpikpl]

Any updates since October on this?

[joslat]

Hi Pietro, This was indeed a challenge, been playing with different implementations until this december... where I joined a project, I am finishing a demo which will end up here, joslat/JSB: Jose's Speaking Buddy <https://github.com/joslat/JSB> And you can guess what I want to achieve ;) Hint: it's a bit in shambles and not nice but it works... (code is not yet checked in though) Best, Jose

…

On Wed, Oct 23, 2024 at 3:51 PM khouloud-yengui ***@***.***> wrote: @joslat <https://github.com/joslat> I have seen how you added "get_weather_for_location" function as a tool in the "ConversationSessionOptions" , but what i want to know is from where the GPT-4o Real Time model can get the "get_weather_for_location" function to execute it , because there is no implementation of this function in your project . Also if you have an idea please , in my project I have already a plugin (a c# file ) linked with a kernel , this plugin manages the generation of response using RAG , is there a possibility to declare the functions that were implemented in my plugin as you did for "get_weather_for_location" function and add them as tool ? — Reply to this email directly, view it on GitHub <#9075 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACLAEYZAHSKARGIQINHTCULZ46SXFAVCNFSM6AAAAABPJLKWJGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZSGI3TMNBTGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Jose Luis Latorre This e-mail is environment friendly, please do not print it.