Config setting for request timeout and retries in custom endpoints

[jameslamine]

Summary

Add two new settings to custom endpoints:

• timeout - timeout in milliseconds. Default 10 minutes (10 * 60 * 1000)
• max_retries - Default 2

Under the hood these will set maxRetries and timeout in the Open AI SDK.

const client = new OpenAI({
  timeout: 20 * 1000, // 20 seconds (default is 10 minutes),
  maxRetries: 2
});

Example

endpoints:
  custom:
    - name: "Example"
      timeout: 5000
      maxRetries: 3

Details

Sometimes the downstream LLM API hangs and we have to wait for the API call to time-out before seeing an error. Users can manually cancel the request, but generally when a request is hanging it's never going to respond. Better to cancel the request and re-try.

In general, it's best practice to set relatively low timeouts, for example 5 seconds. When using the streaming API you can use relatively low timeouts, but in-general this should be user configurable.

The openai nodejs SDK supports a configurable timeout parameter: https://github.com/openai/openai-node?tab=readme-ov-file#timeouts

Let's add support for that as a yaml config option in endpoints

[jameslamine]

Implemented here: #5568