fix: implement timeout wrapper for OpenAI streams to handle slow models

[roomote]

This PR fixes the issue where local models with long prompt load times would timeout after 5 minutes due to undici's default bodyTimeout.

Since passing custom fetch or fetchOptions to the OpenAI SDK doesn't work correctly (as confirmed by @pwilkin), this PR implements an alternative approach by wrapping the OpenAI SDK stream response with a configurable timeout.

• Created a withTimeout wrapper function that monitors async iterables/streams
• Wraps the OpenAI SDK stream response with configurable timeout
• Added openAiRequestTimeout configuration option to ProviderSettings
• Resets timeout on each chunk received, so only idle time counts toward timeout
• Includes comprehensive tests for various timeout scenarios

Users can now configure the timeout in milliseconds in their settings.

Fixes #6570

---

Important

Introduces a timeout wrapper for OpenAI streams to handle slow models, allowing configurable timeout settings and ensuring only idle time counts towards the timeout.

• Behavior:
  • Implements withTimeout function in timeout-wrapper.ts to wrap async iterables with a configurable timeout.
  • Adds openAiRequestTimeout option to ProviderSettings in provider-settings.ts for user-configurable timeout.
  • Wraps OpenAI SDK stream responses with withTimeout in base-openai-compatible-provider.ts and openai.ts.
  • Resets timeout on each chunk received, counting only idle time towards timeout.
• Tests:
  • Adds tests in timeout-wrapper.spec.ts for various timeout scenarios, including no timeout, timeout after delay, and error handling.

^{This description was created by for 025ea85. You can customize this summary. It will automatically update as commits are pushed.}

[roomote]

I wrote a timeout wrapper to fix my own timeout issues. The irony is not lost on me.

[roomote]

Is this intentional? The timeout is cleared in the finally block, but if the iterator throws before entering the try block, we might leak the timeout. Consider wrapping the entire function body in try-finally to ensure cleanup:

Suggested change

	): AsyncGenerator<T> {
	export async function* withTimeout<T>(
	iterable: AsyncIterable<T>,
	timeout: number = DEFAULT_REQUEST_TIMEOUT,
	): AsyncGenerator<T> {
	let timeoutId: NodeJS.Timeout | null = null
	let hasTimedOut = false
	try {
	const resetTimeout = () => {
	if (timeoutId) {
	clearTimeout(timeoutId)
	}
	timeoutId = setTimeout(() => {
	hasTimedOut = true
	}, timeout)
	}
	// Set initial timeout
	resetTimeout()
	for await (const value of iterable) {
	if (hasTimedOut) {
	throw new Error(`Request timeout after ${timeout}ms`)
	}
	// Reset timeout on each chunk received
	resetTimeout()
	yield value
	}
	} catch (error) {
	if (hasTimedOut) {
	throw new Error(`Request timeout after ${timeout}ms`)
	}
	// Check if this is a timeout-related error
	if (error instanceof Error && (error.message.includes("aborted") || error.message.includes("timeout"))) {
	throw new Error(`Request timeout after ${timeout}ms`)
	}
	throw error
	} finally {
	if (timeoutId) {
	clearTimeout(timeoutId)
	}
	}
	}

[roomote]

Could we consider always applying the timeout wrapper with the default timeout? Currently, it's only applied when explicitly configured:

Suggested change

	const stream = this.options.openAiRequestTimeout ? withTimeout(baseStream, timeout) : baseStream
	// Wrap the stream with timeout (use configured timeout or default)
	const timeout = this.options.openAiRequestTimeout || DEFAULT_REQUEST_TIMEOUT
	const stream = withTimeout(baseStream, timeout)

This would ensure consistent timeout behavior across all requests, not just when users explicitly configure it.

[roomote]

Are these utility functions intended for future use? They're exported but not currently used in the codebase. If they're not needed, we might want to remove them to keep the code lean.

[roomote]

I understand why this test is skipped, but could we explore alternative approaches? Perhaps we could test with a mock that never resolves its next() promise, or use a controlled async generator that we can manually advance?

[roomote]

Consider adding a more detailed comment explaining when users might want to adjust this value. For example:

Suggested change

	openAiRequestTimeout: z.number().min(0).optional(), // Request timeout in milliseconds
	openAiRequestTimeout: z.number().min(0).optional(), // Request timeout in milliseconds. Useful for slow local models or high-latency connections. Default: 300000 (5 minutes)

[daniel-lxs]

Closing, the issue needs scoping, this implementation doesn't fix the issue