Python: Bug: Agent initialised from YAML file does not register the response_format and always responds in free text #13349
Description
Describe the bug
I am trying to implement the initialisation of agents from their specifications in YAML files to avoid writing instructions of the agents within the code. To do so I am using the AgentRegistry.create_from_file method. Using this method, when I initialise the ChatCompletionAgent with my defined AzureChatPromptExecutionSettings, the agent does not follow the response_format provided in the settings. Instead it returns the response as free text. However, if I initialise the agent within the code itself, it uses the response_format to structure its response.
To Reproduce
- The structure of the workspace
The goal is to initialise an agent that will use a native python function to crawl the datasets folder and return only the paths of the datasets that are relevant to the request. Example of a request is "Give me the location of all the datasets which contain customer details". Currently the scope is limited to just retrieving the paths of all files in the datasets folder.
- Defining the native python function
The following python function is used
# function to traverse a directory and get the path relevant data files
from typing import List
import os
def get_data_filenames(datasets_directory: str, safe_file_exts: tuple[str] = ('.csv', '.xml', '.txt', '.json')) -> List[str]:
data_files_paths: List[str] = [] # store the paths of all the files
# start the traversal
for dirpath, dirs, files in os.walk(datasets_directory):
for filename in files:
filepath = os.path.join(dirpath,filename)
if filepath.endswith(safe_file_exts):
data_files_paths.append(filepath)
return data_files_paths
- Registering it in a plugin as a kernel function
A plugin is created with this single function
# create the plugin for the file discovery agent
from semantic_kernel.functions import kernel_function, KernelArguments
class FileDiscoveryPlugin:
@kernel_function(description="Find the files in the given dictionary")
async def find_files(self, datasets_directory: Annotated[str, 'the path of the directory where the datasets are present']) -> Annotated[List[str], 'list of paths of the datasets in that directory']:
"Find the files in the given dictionary"
return get_data_filenames(datasets_directory)
- Defining the class that will be used a response_format
from semantic_kernel.kernel_pydantic import KernelBaseModel
## define the class for the output structure of the file discovery agent
class FilesList(KernelBaseModel):
files_list: Annotated[List[str], "a list of paths of datasets"]
- Function Invocation Filter to check the execution
## define the function invocation filter - to let us see the outputs
from semantic_kernel.filters import FilterTypes, FunctionInvocationContext
from semantic_kernel.connectors.ai.function_choice_behavior import FunctionChoiceBehavior
from typing import Awaitable, Callable
async def function_invocation_filter(
context: FunctionInvocationContext,
next: Callable[[FunctionInvocationContext], Awaitable[None]],
) -> None:
# this runs before the function is called
print(f" ---> Calling Plugin {context.function.plugin_name}.{context.function.name} with arguments `{context.arguments}`")
# let's await the function call
await next(context)
# this runs after our functions has been called
print(f" ---> Plugin response from [{context.function.plugin_name}.{context.function.name} is `{context.result}`")
- Adding the plugin and filter to the kernel
from semantic_kernel import Kernel
kernel = Kernel()
## add the plugin to the kernel
kernel.add_plugin(
plugin = FileDiscoveryPlugin(),
plugin_name = "FileDiscoveryPlugin"
)
## add the filter to observe the outputs
kernel.add_filter(FilterTypes.FUNCTION_INVOCATION, function_invocation_filter)
print(f'plugins listed to the kernel: {kernel.plugins}\n')
print(f'Invocation filters added: {kernel.function_invocation_filters}\n')
- Initialising the agents and passing a request
a. Using a YAML specs file
The YAML File
type: chat_completion_agent
name: FileDiscoveryAgent
template: |
<message role="system">
You are an AI agent that searches for files. As the agent, you give the user all the files that are present in the specified folder.
# File search context
Folder: {{datasets_directory}}
Return the paths of all files found to the user.
template_format: semantic-kernel
description: An assistant that gathers the list of files in a given directory.
instructions: You are an AI agent that searches for files. As the agent, you give the user all the files that are present in the specified folder. Return the response in the format that has been given to you.
tools:
- id: FileDiscoveryPlugin.find_files
type: function
input_variables:
- name: datasets_directory
description: The folder with the datasets.
is_required: true
iniltialising the settings
from semantic_kernel.connectors.ai.open_ai import AzureChatPromptExecutionSettings
model_settings = AzureChatPromptExecutionSettings()
model_settings.function_choice_behavior = FunctionChoiceBehavior.Required()
model_settings.temperature = 0.0
model_settings.top_p = 0.0
model_settings.response_format = FilesList # provide the defined class as the format to structure response
creating the agent from the YAML file and providing the settings, asking for a response
agent: ChatCompletionAgent = await AgentRegistry.create_from_file(
yaml_path, service = AzureChatCompletion(), kernel=kernel, settings = model_settings, arguments=KernelArguments(datasets_directory="./datasets/")
)
response = await agent.get_response(messages=f"find all the files in ./datasets/")
print(f'response content:{response.message.content}, type: {type(response.message.content)}\n')
I get the following
I have also tried a few other ways:
Adding the output format to the YAML and not initialising the setting from AzureChatPromptExecutionSettings. I am not sure if I have defined the output structure in the YAML correctly since I could not find documentation of setting a class as the output variable in a YAML specification.
type: chat_completion_agent
name: FileDiscoveryAgent
template: |
<message role="system">
You are an AI agent that searches for files. As the agent, you give the user all the files that are present in the specified folder.
# File search context
Folder: {{datasets_directory}}
Return the paths of all files found to the user.
template_format: semantic-kernel
description: An assistant that gathers the list of files in a given directory.
instructions: You are an AI agent that searches for files. As the agent, you give the user all the files that are present in the specified folder. Return the response in the format that has been given to you.
tools:
- id: FileDiscoveryPlugin.find_files
type: function
input_variables:
- name: datasets_directory
description: The folder with the datasets.
is_required: true
output_variable:
- description: The list of files in the folder
json_schema: {"files_list": ["path_1", "path_2", "path_3", "path_n"]}
execution_settings:
default:
temperature: 0.0
top_p: 0.0
agent: ChatCompletionAgent = await AgentRegistry.create_from_file(
yaml_path, service = AzureChatCompletion(), kernel=kernel, settings = model_settings, arguments=KernelArguments(datasets_directory="./datasets/"
)
response = await agent.get_response(messages=f"find all the files in ./datasets/")
print(f'response content:{response.message.content}, type: {type(response.message.content)}\n')
Adding the output format to the YAML and also initialising the setting using AzureChatPromptExecutionSettings. However, if my understanding is correct, the settings defined in the code are going to overwrite the settings imported from the YAML?
type: chat_completion_agent
name: FileDiscoveryAgent
template: |
<message role="system">
You are an AI agent that searches for files. As the agent, you give the user all the files that are present in the specified folder.
# File search context
Folder: {{datasets_directory}}
Return the paths of all files found to the user.
template_format: semantic-kernel
description: An assistant that gathers the list of files in a given directory.
instructions: You are an AI agent that searches for files. As the agent, you give the user all the files that are present in the specified folder. Return the response in the format that has been given to you.
tools:
- id: FileDiscoveryPlugin.find_files
type: function
input_variables:
- name: datasets_directory
description: The folder with the datasets.
is_required: true
output_variable:
- description: The list of files in the folder
json_schema: {"files_list": ["path_1", "path_2", "path_3", "path_n"]}
execution_settings:
default:
temperature: 0.0
top_p: 0.0
model_settings = AzureChatPromptExecutionSettings()
model_settings.function_choice_behavior = FunctionChoiceBehavior.Required()
model_settings.temperature = 0.0
model_settings.top_p = 0.0
model_settings.response_format = FilesList # provide the defined class as the format to structure response
agent: ChatCompletionAgent = await AgentRegistry.create_from_file(
yaml_path, service = AzureChatCompletion(), kernel=kernel, settings = model_settings, arguments=KernelArguments(datasets_directory="./datasets/"
)
response = await agent.get_response(messages=f"find all the files in ./datasets/")
print(f'response content:{response.message.content}, type: {type(response.message.content)}\n')
However, I get the same response in all cases. The result is the same whether I do
output_variable:
- description: The list of files in the folder
json_schema: '{"files_list": ["path_1", "path_2", "path_3", "path_n"]}'
or
output_variable:
- description: The list of files in the folder
json_schema: {"files_list": []}
or
output_variable:
- description: The list of files in the folder
json_schema: '{"files_list": []}'
b. Using ChatCompletionAgent constructor, no YAML
# define the settings for the agent
model_settings = AzureChatPromptExecutionSettings()
model_settings.function_choice_behavior = FunctionChoiceBehavior.Required()
model_settings.temperature = 0.0
model_settings.top_p = 0.0
model_settings.response_format = FilesList
kernel = Kernel()
## add the plugin to the kernel
kernel.add_plugin(
plugin = FileDiscoveryPlugin(),
plugin_name = "FileDiscoveryPlugin"
)
## define the function invocation filter - to let us see the outputs
async def function_invocation_filter(
context: FunctionInvocationContext,
next: Callable[[FunctionInvocationContext], Awaitable[None]],
) -> None:
# this runs before the function is called
print(f" ---> Calling Plugin {context.function.plugin_name}.{context.function.name} with arguments `{context.arguments}`")
# let's await the function call
await next(context)
# this runs after our functions has been called
print(f" ---> Plugin response from [{context.function.plugin_name}.{context.function.name} is `{context.result}`")
kernel.add_filter(FilterTypes.FUNCTION_INVOCATION, function_invocation_filter) # add the filter to observe the outputs
# initiate the agent that will look for data files
files_discovery = ChatCompletionAgent(
service = AzureChatCompletion(),
name = "FileDiscoveryAgent",
instructions = """You are an AI agent that searches for files. As the agent, you give the user all the files that are present in the specified folder. Return the response in the format that has been given to you""",
kernel = kernel,
arguments = KernelArguments(settings = model_settings),
# settings = model_settings
)
query = 'find all the files in ./datasets/'
response = await files_discovery.get_response(messages = query)
print(f'response content:{response.message.content}, type: {type(response.message.content)}\n')
response_json = json.loads(response.message.content)
print(f'response_json:{response_json}, type: {type(response_json)}\n')
I get the following, which is what I want the result to be when I use the YAML agent initialisation and give it the FilesList class as the response format. This is to make extracting of the results easy.
Expected behavior
Both these approaches should have resulted in the same output of a dictionary like response following the response_format supplied in the settings, as shown in case b.
Platform
- Language: [Python]
- Source: [Semantic Ker]
- AI model: [OpenAI:GPT-4o(2024-11-20)]
- IDE: [Azure ML Studio]
- OS: [Linux 25.06.10]