.NET SDK: Slow performance on Chat Completions API with Azure Open AI

[miguelisidoro]

Hello,

We are using Semantic Kernel C# SDK to interact with Azure Open AI.

In one of the steps of our flow, we are calling chat completions api in the following manner.

Adding the Kernel.

var builder = Kernel.CreateBuilder();
builder.AddAzureOpenAIChatCompletion(
    _config.OpenAICompletionsDeployment, //gpt-4-32k
    _config.OpenAIEndpoint,
    _config.OpenAIKey
);

Call to Chat Completions API:

var executionSettings = new OpenAIPromptExecutionSettings()
{
    MaxTokens = 6000,
    FrequencyPenalty = 0,
    PresencePenalty = 0,
    Temperature = 0,
    ResultsPerPrompt = 1,
};

ChatMessageContent messageContent = await chatCompletion.GetChatMessageContentAsync(
    chatHistory,
    executionSettings
);

The call to ChatMessageContent messageContent = await chatCompletion.GetChatMessageContentAsync( is taking between 10 to 15s.

In another implementation, that is prior to this one, we are using Chat Completions REST API directly and a similar call takes between 2 and 3s. Using this approach we have full control on the endpoint and the version we are using of that endpoint.

Using Semantic Kernel SDK, we are facing this performance problem.

The version of Semantic Kernel SDK is 1.16.1.
We are using gpt-4-32k as deployment model (in the other implementation we are using gpt-35-turbo).

I have tried testing the API both in localhost in my DEV box and deploying to Azure using an App Service using the following App Service Plan (1 instance only):

and the problem happens both in DEV and Azure deployment.

Any help is appreciated!

Thanks

[dmytrostruk]

@miguelisidoro Thanks for reporting this problem! Would it be possible for you to run Debug > Performance Profiler with Instrumentation enabled in Visual Studio?
https://learn.microsoft.com/en-us/visualstudio/profiling/profiling-feature-tour?view=vs-2022#instrumentation

This should output the breakdown for each C# function, execution time and number of calls:

With this view, you should be able to see where exactly there is a delay in execution.

We are using gpt-4-32k as deployment model (in the other implementation we are using gpt-35-turbo).

Would be interesting to compare both approaches using the same deployment model, just for results to be more accurate. Things like different deployment and AI model used can impact the performance as well.

[miguelisidoro]

Hello @dmytrostruk ,

About instrumentation, all I could gather is:

That basically doesn't help. Any other suggestion?

About your other suggestion regarding deployment and AI models, I will make some tests.

Thanks

[miguelisidoro]

Hello @dmytrostruk,

Problem solved.
I changed the AI model to GPT 3.5 and performance is 5x faster.
I will however try out GPT 4o Mini AI model that is supposed to be cheaper, faster and more powerful than GPT 3.5.

Thanks

[dmytrostruk]

Great to hear, thanks for letting us know!