FunctionChoiceBehaviour support across models

[felixcantet]

Hello,

I'm starting to design an AI Agent abstraction solution that should support different models and providers.
According to the documentation, Planners are deprecated and I should use FunctionChoiceBehavior.Auto() instead.

My issue is that FunctionChoiceBehavior.Auto() is only supported for a subset of models, I would like to be able to target Gemini as exemple.

So my question is : what is the recommended way to support different models and providers function calling ability in a uniform and abstracted way ? It looks like StepwisePlanner is the best solution for now, but I want to make sure I'm not missing something.

Thanks for your help !

[moonbox3]

Hello @felixcantet are you targeting SK Python or .NET? We'd recommend not using any planner. Vanilla function performs much better - it has better latency and uses less tokens.

We already provide the FunctionChoiceBehavior abstraction across various models. Can you help us know if we're missing something or not implementing something properly? The .Auto(), Required() or None() options should match appropriately for various models in the actual implementation of the AI Service.

[felixcantet]

Hello ! Thanks you for your answer !

I'm targeting .Net, sorry for the missing precision.

From all the tests I've made, FunctionChoiceBehavior is not working at all with Google Gemini models.
GeminiToolCallBehavior.AutoInvokeKernelFunctions looks like working, but the documentation is unclear about the ToolCallBehavior. Is it deprecated ? What is the difference with FunctionChoiceBehavior.Auto() ?

Sorry, I'm new to SK and maybe I missed something important,

Thanks !

[moonbox3]

Thanks for your response, @felixcantet. I'm adding @SergeyMenshykh who has more experience with FunctionChoiceBehavior in SK .NET.

[rogerbarreto]

Hi @felixcantet, thanks for your feedback, we do have support for function calling with Gemini. Please have a look on this sample:

https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/FunctionCalling/Gemini_FunctionCalling.cs

Please let me know if this works for you.

[felixcantet]

Hi !
I already tried to replicate this sample (from 1.48 to 1.51 SK version) without success

Function calling is working with Gemini only through the ToolCallBehavior = GeminiToolCallBehavior.AutoInvokeKernelFunctions
but as soon as I set FunctionChoiceBehavior = FunctionChoiceBehavior.Auto() in the GeminiPromptExecutionSettings it stopped working.

Here is a working code snippet :

var kernelBuilder = Kernel.CreateBuilder();
kernelBuilder.AddGoogleAIGeminiChatCompletion(model.Id, key);
var kernel = kernelBuilder.Build();
kernel.Plugins.AddFromType<MyPlugin>();
var chatHistory = new ChatHistory();
chatHistory.AddUserMessage(prompt);

var promptSettings = new GeminiPromptExecutionSettings()
            {
                ToolCallBehavior = GeminiToolCallBehavior.AutoInvokeKernelFunctions,
            };

var chat = kernel.GetGetRequiredService<IChatCompletionService>();
var result = await chat.GetChatMessageContentAsync(chatHistory, promptSettings, kernel);

Now if we replace the promptSettings for

var promptSettings = new GeminiPromptExecutionSettings() { FunctionChoiceBehavior = FunctionChoiceBehavior.Auto() };

the model don't call any function

Is it the desired behavior or should I report an issue ?
I don't know if both solutions are supposed to behave the same way, if it is I can go with ToolCallBehavior that looks like working reliably

Thanks for you help

[cNoNim]

FunctionChoiceBehavior is not supported in Connectors.Google.
I stumbled upon this discussion when searching for a similar issue.
Perhaps it's worth creating one?

[cNoNim]

Here's the closest issue: #10223