How does plugin work for Azure AI Agent?

[ningweinw]

Can plugins added to kernel be used by Azure AI Agent? What will the flow look like with both kernel plugin and Azure AI Agent's own function calling tool? In this situation, what's kernel's role in reasoning for which functions to call?

[moonbox3]

Hi @ningweinw. Yes, plugins can be used with an Azure AI Agent. You can add plugins (functions) and they will have their JSON schema sent to the agent, which will be used for tool calls. The can work alongside other Azure AI Agent tools - if the models deems that it should make a tool call for a broadcasted function, it will leverage it.

Are you working with SK Python or .NET? We have examples in each language showing how plugins are used:

Python: https://github.com/microsoft/semantic-kernel/blob/main/python/samples/getting_started_with_agents/azure_ai_agent/step2_azure_ai_agent_plugin.py
.NET: https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/GettingStartedWithAgents/AzureAIAgent/Step02_AzureAIAgent_Plugins.cs

There are other concept samples either in https://github.com/microsoft/semantic-kernel/tree/main/python/samples/concepts/agents/azure_ai_agent or https://github.com/microsoft/semantic-kernel/tree/main/dotnet/samples/Concepts/Agents

[ningweinw]

Thanks @moonbox3 If I register function calling tools in AgentsClient.CreateAgentAsync and also add kernel plugins, what will be the system's logic to decide which ones to use? Will both be used based on best match?

[moonbox3]

The functions that are registered as part of the AzureAIAgent definition must also exist in the kernel, otherwise there is an error during function calling because the function will not be found.