Loading a Foundry agent in code and adding a plugin to it before executing it

[TheTrainingBoss]

Hello
I can create an Azure AI Foundry agent in code using a definition and then use the AzureAIAgent to create a Semantic Kernel agent, starting a conversation with no issues.
I can also do the above while creating a plugin and adding it to the Semantic Kernel agent, and all works well.

My problem occurs when I create an agent in the Azure AI Foundry Project (in the portal) and write code to retrieve the endpoint and retrieve the agent (all works so far). However, I am unable to add a plugin to an existing agent from Foundry in code.
Does anyone have sample code in any language that demonstrates how to extract the definition from an existing agent and add a plugin to the AzureAIAgent call?
Thank you so much
-Lino

[moonbox3]

Hi Lino,

The are two steps to perform when creating an AzureAIAgent. You first create the "agent definition" which is the server-side record of the agent (or in your case, you created that in the Portal, and you pull it down via the agent id). Secondly, you create the SK AzureAIAgent instance, passing in the Azure client, and the agent's definition (that you either created or pulled down from the server). When you create the AzureAIAgent instance, you can provide it a list of plugins. Here's an example in Python, where I create a first agent instance, and then use the agent.id to get the agent definition and then use that definition to create the second SK agent instance, while adding plugins.

# Copyright (c) Microsoft. All rights reserved.

import asyncio
from typing import Annotated

from azure.identity.aio import AzureCliCredential

from semantic_kernel.agents import AzureAIAgent, AzureAIAgentThread
from semantic_kernel.functions import kernel_function

# Simulate a conversation with the agent
USER_INPUTS = [
    "Hello, I am John Doe.",
]


# A plugin we'll use for the agent we create after pulling down the server's agent definition
class MenuPlugin:
    """A sample Menu Plugin used for the concept sample."""

    @kernel_function(description="Provides a list of specials from the menu.")
    def get_specials(self) -> Annotated[str, "Returns the specials from the menu."]:
        return """
        Special Soup: Clam Chowder
        Special Salad: Cobb Salad
        Special Drink: Chai Tea
        """

    @kernel_function(description="Provides the price of the requested menu item.")
    def get_item_price(
        self, menu_item: Annotated[str, "The name of the menu item."]
    ) -> Annotated[str, "Returns the price of the menu item."]:
        return "$9.99"


async def main() -> None:
    async with (
        AzureCliCredential() as creds,
        AzureAIAgent.create_client(credential=creds) as client,
    ):
        # Create an agent on the Azure AI agent service
        agent_definition = await client.agents.create_agent(
            model="gpt-4o",
            name="Assistant",
            instructions="Answer the user's questions.",
        )

        # Create a Semantic Kernel agent for the Azure AI agent
        agent = AzureAIAgent(
            client=client,
            definition=agent_definition,
        )

        # Create a thread for the agent
        # If no thread is provided, a new thread will be
        # created and returned with the initial response
        thread: AzureAIAgentThread = None

        try:
            for user_input in USER_INPUTS:
                print(f"# User: {user_input}")
                # Invoke the agent with the specified message for response
                response = await agent.get_response(messages=user_input, thread=thread)
                print(f"# {response.name}: {response}")
                thread = response.thread

            # Create the second agent def by getting the def from the server for that agent id
            second_agent_definition = await client.agents.get_agent(agent.id)

            # Now create the SK Agent instance using the def, client, and plugins
            second_azure_ai_agent = AzureAIAgent(
                client=client, definition=second_agent_definition, plugins=[MenuPlugin()]  # now we add plugins to this agent instance
            )

            response = await second_azure_ai_agent.get_response("What is the special food item?")
            print(f"# {response.name}: {response}")
        finally:
            # Cleanup: Delete the thread and agent
            await thread.delete() if thread else None
            await client.agents.delete_agent(agent.id)


if __name__ == "__main__":
    asyncio.run(main())

The output looks like:

# User: Hello, I am John Doe.
# Assistant: Hello, John! How can I assist you today? 😊
# Assistant: The special food items are:

- **Special Soup:** Clam Chowder  
- **Special Salad:** Cobb Salad  
- **Special Drink:** Chai Tea  

Note that the plugins need to exist on the SK AzureAIAgent instance - even if your server-side agent definition contains function tools as part of the definition, they also need to be added to the agent - the server only has a JSON schema representation of the tools, not the actual native code tools required to be invoked during operation/function calling.

[TheTrainingBoss]

Thank you so much; yes, indeed, I got it to work last week using the approach you described above. What messed me up before is that I tried to use the AIProjectClient object and that worked to get the agent and execute but it did not work when Plugins were added to the SK agent as the return type of the agent is NOT the agent definition that the AzureAIAgent is expecting.
However, if you create the client using AzureAIAgent.create_client(...) and then set the AzureAIAgent with Plugins, everything works fine.
So the lesson is, don't get the client from AIProjectClient if you are adding plugins on the SK side.
Cheers
Thanks again for your help, much appreciated
-Lino