How to control timeouts for KernelFunction tool calls?

[jbnidev]

We are using Semantic Kernel (C# NuGet 1.55.0) to integrate Azure OpenAI with our application. We have given our AI a tool to launch modal dialogs in our application. This tool works except that Semantic Kernel times out after about 100 seconds. We need Semantic Kernel to wait until the dialog closes and the tool that launched the dialog returns. This could be forever.

We would like to do this while continuing to leverage the very convenient plugin orchestration provided by Semantic Kernel. That is, we add our method using CreatePluginFromFunctions(). When invoking GetChatMessageContentAsync(), we pass options that include ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions. Aside from the timeout, this seems to work pretty well. I believe that in principle we could write our own orchestration layer, but that would negate a lot of the value Semantic Kernel gives us.

I have poked around, but I cannot find any way to control this. Some observations:

1. The CancellationToken we pass to GetChatMessageContentAsync() does not seem to control the timeouts for tool calls. (We were using the default None, but I also tried passing a custom token with a very long, but finite timeout.)
2. I found ToolCallBehavior.DefaultMaximumAutoInvokeAttempts. This is not what we want, but it makes me think this is where a timeout setting would be if one existed.
3. GetChatMessageContentAsync returns with content: "The AI I attempted to open the Station Options dialog, but the operation timed out. This could be due to system load or a temporary issue. Would you like me to try again, or are you looking for a specific Station Option setting? Let me know how you'd like to proceed!" It in fact had correctly launched the dialog. It just gave up after 100 seconds of waiting for me to dismiss it.

Is there a way to control this that I am missing? Would this be an appropriate feature request?

[runceel]

I tried reproducing the issue using Semantic Kernel 1.61.0, but I wasn’t able to observe the same timeout behavior. The plugin function continued running beyond 100 seconds without interruption.

To help investigate further, could you share:

1. Which KernelFunction you're using to launch the modal dialog? It’s possible the issue lies in the implementation of the function being invoked.
2. A minimal reproducible sample of your code. That would allow us to dig deeper and understand the orchestration flow and where the timeout might be triggered.

Here’s the code I used for testing:

using Azure.Identity;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.ChatCompletion;
using Microsoft.SemanticKernel.Connectors.AzureOpenAI;
using Microsoft.SemanticKernel.Connectors.OpenAI;
using System.ComponentModel;

var builder = Kernel.CreateBuilder();

builder.AddAzureOpenAIChatCompletion(
    "gpt-4.1",
    "https://xxxxxx.azure.com/",
    new AzureCliCredential());
builder.Plugins.AddFromType<SomePlugin>();

var kernel = builder.Build();

var client = kernel.Services.GetRequiredService<IChatCompletionService>();

var result = await client.GetChatMessageContentAsync("Add 'go for a walk with the dog' to the task database.",
    new AzureOpenAIPromptExecutionSettings
    {
        ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions,
    },
    kernel);
Console.WriteLine(result.Content);


class SomePlugin
{
    [KernelFunction, Description("Add task to database.")]
    public async Task AddTaskAsync(string task)
    {
        for (int i = 0; i < 110; i++)
        {
            Console.WriteLine($"Adding task: {task} - Attempt {i + 1}");
            await Task.Delay(1000); // Simulate some work
        }

        Console.WriteLine("Task added successfully.");
    }
}

Here's the execution result I observed.

Adding task: go for a walk with the dog - Attempt 1
Adding task: go for a walk with the dog - Attempt 2
...
Adding task: go for a walk with the dog - Attempt 109
Adding task: go for a walk with the dog - Attempt 110
Task added successfully.
The task "go for a walk with the dog" has been added to the task database. If you need to add more tasks or manage your list, let me know!

[jbnidev]

Here is a brief postmortem for anybody mining this conversation for understanding of a similar issue in the future. The timeout was a gRPC timeout, not a Semantic Kernel timeout. Our plugin tool invokes the dialog via gRPC. This is just inter-process communication on the same machine, but it involves gRPC nevertheless. I am relatively new to using LLMs, so the elegant response we got from the LLM had me thinking I was looking for a tool orchestration setting. However, the orchestrator and LLM are so general that they happily parse out the messages from whatever exceptions come through. One key takeaway for me is that nice looking error responses can reflect any failure at any layer in the stack.

[jbnidev]

I tried updating to 1.61.0. The timeout still occurs on cue.

Although I have not wended my way through the orchestrator code yet, I had assumed the 100-second timeout was standard in Semantic Kernel because I found things like these:

1. .Net: The operation was cancelled because it exceeded the configured timeout of 0:01:40 #3412
2. .Net: The operation was cancelled because it exceeded the configured timeout of 0:01:40 #3223

I realize that these are not identical to my situation, but it left me thinking the 100-second timeout made sense, so I started spelunking here. The big thing I take from your answer is that you do not think Semantic Kernel should be timing out on tool calls. I will take a step out and review exactly how we are invoking our dialog. I may be barking up the wrong tree.

[jbnidev]

OK. I tried a simple case like yours: I just sleep for 200 seconds, and Semantic Kernel is not timing out at 100 seconds. My simple methods waits until the 200 seconds expire. We apparently have a timeout I was not aware of lower down in our stack.

Thank you very much for your help.