[BUG] GetStreamingResponseAsync cancellationToken is not working

[bharathm03]

Describe the bug

I'm using AzureOpenAIClient, I've set timeout for streaming response using CancellationTokenSource.CancelAfter. But sometimes the timeout is not happening. The program struck at await foreach (ChatResponseUpdate update in stream.WithCancellation(cts.Token))

Steps to reproduce

Complete code

using var overallCts = new CancellationTokenSource(TimeSpan.FromMinutes(2));
using var cts = CancellationTokenSource.CreateLinkedTokenSource(overallCts.Token, cancellationToken);
cts.CancelAfter(TimeSpan.FromSeconds(45)); // initial timeout of 45 seconds
var stream = client.GetStreamingResponseAsync(messages, options, cts.Token);
await foreach (ChatResponseUpdate update in stream.WithCancellation(cts.Token))
{
  cts.CancelAfter(TimeSpan.FromSeconds(15));
  // my code
}

Sometimes code struck at await foreach forever. None of the other logs were triggered within the loop or outside the loop. What could be reason?

Code snippets

OS

Azure App Service (Linux)

.NET version

9

Library version

2.3.0-beta.2

[jsquire]

Hi @bharathm03. Thanks for reaching out and we regret that you're experiencing difficulties. Where cancellation is concerned, it's important to remember that cancellation is "best effort" and not a guarantee. The client does its best to honor cancellation when it is safe to do so, however there are points where an operation has progressed too far to safely cancel or control has passed to the network and the client no longer has control.

Your snippet is incomplete and does not allow us to test your full end-to-end scenario. I was unable to reproduce the behavior that you're seeing with a simple Responses-based sample that mimics the cancellation token pattern you're using. I'm seeing cancellation trigger as expected with a short duration and the operation complete if I allow sufficient time before cancellation.

var client = new OpenAIResponseClient("gpt-4.1", "<< YOUR API KEY >>");

using var cts = new CancellationTokenSource();
cts.CancelAfter(TimeSpan.FromSeconds(45));
    
AsyncCollectionResult<StreamingResponseUpdate> responseUpdates = client.CreateResponseStreamingAsync("Say 'this is a test.'", null, cts.Token);

await foreach (var update in responseUpdates.WithCancellation(cts.Token))
{
    Console.WriteLine(update.GetType().Name.ToString());
    cts.CancelAfter(TimeSpan.FromSeconds(1));
}

If you're still experiencing issues and require further assistance, please provide a small, stand-alone example that shows how to reproduce the behavior that you're seeing.

[bharathm03]

Locally it can not be reproduced. Now I understand that cancellation token is best effort based. I'll use different approach for cancelling the task. Thanks for the input. I'll close this now

[webwarrior06]

Hi,
I am using cts to extend default timeout span for the httpclient used in the library.

var cts = new CancellationTokenSource();
cts.CancelAfter(TimeSpan.FromMinutes(5));
                
var streamingResponse = ResponseClient.CreateResponseStreamAsync(messages, options, cts.Token);

await foreach (var update in streamingResponse)
{}

But it fails after 1 minute and 40 seconds like:
The operation was cancelled because it exceeded the configured timeout of 0:01:40

Am I using it wrong? How can I increase the timeout limit?
It is very typical for AI requests to finish in couple of minutes especially using the reasoning models like GPT-5.

PS: The client I use above is a simple wrapper. Just assume it as original.

[jsquire]

@webwarrior06: That's not how cancellation works. Using a cancellation token allows you an additional signal to request cancellation for that specific call. It does not supersede or replace the configured network timeout associated with the shared HttpClient. To do that, you'll need to set the NetworkTimeout property on the options that you pass when creating your client.

[webwarrior06]

Thanks for the reply! Yeah, that's right. I managed to sort it out by setting the timeout during client initialization.