BodyTimeout causes local models with long prompt load times to be unusable

[pwilkin]

App Version

3.25.5

API Provider

OpenAI Compatible

Model Used

Qwen3-Coder-30B-A3B

Roo Code Task Links (Optional)

No response

🔁 Steps to Reproduce

1. Open RooCode on the Roo repository
2. Pass it two big files (like Task.ts and ClineProvider.ts)
3. Ask it to analyze the files in any way

💥 Outcome Summary

On a slow model, after 5 minutes the connection will get terminated due to
(a) OpenAI using fetch under the hood
(b) fetch using undici under the hood
(c) undici having a separate bodyTimeout which defaults to 5 minutes (300s)

This becomes especially annoying when coupled with context condensing (which makes i.e. llama.cpp unable to use prompt caching).

Preferrably, if a timeout parameter could be added to the provider, then something like this could possibly work:

import { fetch as undiciFetch } from 'undici';

function customFetch(input: RequestInfo | URL, init?: RequestInit) {
	// Convert the standard fetch parameters to Undici format
	const undiciOptions = {
		...init,
		bodyTimeout: 180000
	}
	
	// Type assertion to handle the compatibility issue
	return undiciFetch(input as any, undiciOptions as any) as any
}

and then:

			this.client = new OpenAI({
				baseURL,
				apiKey,
				defaultHeaders: headers,
				timeout: 180000,
				// Use customFetch to inject Undici's extended options like bodyTimeout
				fetch: customFetch
			})

(with the configured timeout instead of 30 minutes)

📄 Relevant Logs or Errors (Optional)

terminated

Retry attempt 1
Retrying now...

[roomote]

Hi @pwilkin! I'm here to help out the maintainers and am going to see if I can fix this issue. I understand the problem - the 5-minute bodyTimeout from undici is causing issues with slow local models. Your suggested solution using a custom fetch function looks promising. I'll explore the codebase and implement a fix that allows configurable timeouts for OpenAI-compatible providers. Thanks for the detailed report and suggested approach!

[roomote]

I've created a pull request to fix this issue: #6573

The solution adds a new openAiRequestTimeout configuration option that allows users to configure the request timeout for OpenAI-compatible providers. This addresses the problem where local models with long prompt load times would timeout after 5 minutes due to undici's default bodyTimeout.

The implementation:

• Adds the timeout configuration to the ProviderSettings schema
• Creates a custom fetch function that properly configures undici's bodyTimeout
• Updates both OpenAiHandler and BaseOpenAiCompatibleProvider to use the custom fetch
• Includes comprehensive tests

Once merged, users will be able to configure the timeout like this:

{
  apiProvider: "openai",
  openAiRequestTimeout: 1800000 // 30 minutes in milliseconds
}

The CI checks are currently running on the PR. Thank you for reporting this issue!

[pwilkin]

@roomote-agent actually from my tests it seems like neither passing fetch nor fetchOptions to the OpenAI client works correctly, feel free to search for other options.

If you want to test for it, you need to create an actually long-running test and fake, say, a 6 minute provider first chunk.

[roomote]

Thank you for the feedback @pwilkin! I understand that passing fetch or fetchOptions to the OpenAI client isn't working as expected. I'll investigate alternative approaches and create a proper long-running test that simulates a 6-minute delay for the first chunk from the provider. Let me explore the codebase further to find a better solution.

[roomote]

I've created a new pull request with an alternative approach: #6574

Since passing fetch or fetchOptions to the OpenAI client doesn't work as expected, I implemented a wrapper approach that monitors the stream response:

1. Stream Wrapper: Created a withTimeout function that wraps async iterables/streams
2. Configurable Timeout: Added openAiRequestTimeout option to provider settings
3. Smart Timeout: The timeout resets on each chunk received, so only idle time counts
4. Comprehensive Tests: Added tests covering various scenarios including the 6-minute delay case

This approach works by wrapping the OpenAI SDK stream response after it's created, rather than trying to modify the underlying fetch behavior. Users can configure the timeout in milliseconds:

{
  "apiProvider": "openai",
  "openAiRequestTimeout": 1800000  // 30 minutes
}

The CI checks are currently running on the PR. This should resolve the issue with local models that have long prompt load times.

[pwilkin]

@roomote-agent The problem is, this happens during prompt processing, so before any chunks are sent.