Implementing RetryFilter for Streamed IChatCompletionService

[KossiH]

Hello everyone,

I'm currently working with the RetryFilter (an implementation of IFunctionInvocationFilter) which works perfectly when using the kernel.InvokeAsync() method. However, I am facing challenges getting the RetryFilter to work with the streaming method of IChatCompletionService, specifically GetStreamingChatMessageContentsAsync.

My goal is to implement a fallback mechanism to a secondary service (similar to examples found in .NET) if the streaming operation fails.

Has anyone managed to achieve this? If so, could you provide an example or guide on how to accomplish this?

Below is a short code snippet demonstrating the service/method I am using:
var chatCompletion = this._kernel.GetRequiredService();
return chatCompletion.GetStreamingChatMessageContentsAsync(
prompt.MetaPromptTemplate,
this.CreateChatRequestSettings(),
this._kernel,
cancellationToken
);

I appreciate any help or insights you can offer.

Thank you!

[moonbox3]

We're now tracking this as an issue in #7856.

[madsbolaris]

@KossiH, are you able to share what you're retrying? Is it the AI service? Or a service for one of your functions?

[KossiH]

@matthewbolanos My goal is to get GetStreamingChatMessageContentsAsync call by using default AI service and if the request fails > RetryFilter would automaticly retry the request by using secondary AI service.. Similar as in Dotnet examples [https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Filtering/RetryWithFilters.cs]

I would also assume that if the GetStreamingChatMessageContentsAsync is using any plugins (ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions), retryFilter would also apply to those if there is any errors e.g. max request / token limits etc...