[AI gateway] Request timeouts for fallback providers

src/content/changelogs/ai-gateway.yaml

@@ -5,11 +5,15 @@ productLink: "/ai-gateway/"
 productArea: Developer platform
 productAreaLink: /workers/platform/changelog/platform/
 entries:
+  - publish_date: "2025-01-23"
+    title: Request timeouts for Universal gateways and fallback providers
+    description: |-
+      * Added [request timeouts](/ai-gateway/configuration/fallbacks/#request-timeouts) as a configuration option for fallback providers. This property triggers a fallback proivder based on a predetermined response time (measured in milliseconds).
   - publish_date: "2025-01-02"
     title: DeepSeek
     description: |-
       * **Configuration**: Added [DeepSeek](/ai-gateway/providers/deepseek/) as a new provider.
-    
+
   - publish_date: "2024-12-17"
     title: AI Gateway Dashboard
     description: |-

src/content/docs/ai-gateway/configuration/fallbacks.mdx

@@ -9,11 +9,15 @@ import { Render } from "~/components";
 
 Specify model or provider fallbacks with your [Universal endpoint](/ai-gateway/providers/universal/) to handle request failures and ensure reliability.
 
-Fallbacks are currently triggered only when a request encounters an error. We are working to expand fallback functionality to include time-based triggers, which will allow requests that exceed a predefined response time to timeout and fallback. 
+Cloudflare can trigger your fallback provider in response to [request errors](#request-failures) or [predetermined request timeouts](#request-timeouts).
 
-## Example
+## Request failures
 
-In the following example, a request first goes to the [Workers AI](/workers-ai/) Inference API. If the request fails, it falls back to OpenAI. The response header `cf-aig-step` indicates which provider successfully processed the request. 
+By default, Cloudflare triggers your fallback if a model request returns an error.
+
+### Request failure example
+
+In the following example, a request first goes to the [Workers AI](/workers-ai/) Inference API. If the request fails, it falls back to OpenAI. The response header `cf-aig-step` indicates which provider successfully processed the request.
 
 1. Sends a request to Workers AI Inference API.
 2. If that request fails, proceeds to OpenAI.
@@ -32,6 +36,84 @@ You can add as many fallbacks as you need, just by adding another object in the
 
 <Render file="universal-gateway-example" />
 
+---
+
+## Request timeouts
+
+If set, a request timeout triggers a fallback provider based on a predetermined response time (measured in milliseconds). This feature is helpful for latency-sensitive applications because your gateway does not have to wait for a [request error](#request-failure) before moving to a fallback provider.
+
+You can configure request timeouts by using one or more of the following properties, which are listed in order of priority:
+
+| Priority | Property                                                                                                               |
+| -------- | ---------------------------------------------------------------------------------------------------------------------- |
+| 1        | `requestTimeout` (added as a universal attribute)                                                                      |
+| 2        | `cf-aig-request-timeout` (header included at the [provider level](/ai-gateway/providers/universal/#payload-reference)) |
+| 3        | `cf-aig-request-timeout` (header included at the request level)                                                        |
+
+Your gateway follows this hierarchy to determine the timeout duration before implementing a fallback.
+
+### Request timeout example
+
+These request timeout values can interact to customize the behavior of your universal gateway.
+
+In this example, the request will try to answer `What is Cloudflare?` within 1000 milliseconds using the normal `@cf/meta/llama-3.1-8b-instruct` model. The `requestTimeout` property takes precedence over the `cf-aig-request-timeout` for `@cf/meta/llama-3.1-8b-insruct`.
+
+If that fails, then the gateway will timeout and move to the fallback `@cf/meta/llama-3.1-8b-instruct-fast` model. This model has 3000 milliseconds - determined by the request-level `cf-aig-request-timeout` value - to complete the request and provide an answer.
+
+```bash title="Request" collapse={36-50} {2,11,13-15}
+curl 'https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}' \
+	--header 'cf-aig-request-timeout: 3000' \
+	--header 'Content-Type: application/json' \
+	--data '[
+    {
+        "provider": "workers-ai",
+        "endpoint": "@cf/meta/llama-3.1-8b-instruct",
+        "headers": {
+            "Authorization": "Bearer {cloudflare_token}",
+            "Content-Type": "application/json",
+            "cf-aig-request-timeout": "2000"
+        },
+        "config": {
+            "requestTimeout": 1000
+        },
+        "query": {
+            "messages": [
+                {
+                    "role": "system",
+                    "content": "You are a friendly assistant"
+                },
+                {
+                    "role": "user",
+                    "content": "What is Cloduflare?"
+                }
+            ]
+        }
+    },
+    {
+        "provider": "workers-ai",
+        "endpoint": "@cf/meta/llama-3.1-8b-instruct-fast",
+        "headers": {
+            "Authorization": "Bearer XXXXXXXXXXXXXXXXXXXX",
+            "Content-Type": "application/json"
+        },
+        "query": {
+            "messages": [
+                {
+                    "role": "system",
+                    "content": "You are a friendly assistant"
+                },
+                {
+                    "role": "user",
+                    "content": "What is Cloudflare?"
+                }
+            ]
+        }
+    }
+]'
+```
+
+---
+
 ## Response header(cf-aig-step)
 
 When using the [Universal endpoint](/ai-gateway/providers/universal/) with fallbacks, the response header `cf-aig-step` indicates which model successfully processed the request by returning the step number. This header provides visibility into whether a fallback was triggered and which model ultimately processed the response.

src/content/docs/ai-gateway/providers/universal.mdx

@@ -16,18 +16,24 @@ https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}
 
 AI Gateway offers multiple endpoints for each Gateway you create - one endpoint per provider, and one Universal Endpoint. The Universal Endpoint requires some adjusting to your schema, but supports additional features. Some of these features are, for example, retrying a request if it fails the first time, or configuring a [fallback model/provider](/ai-gateway/configuration/fallbacks/).
 
+## Payload reference
+
 You can use the Universal endpoint to contact every provider. The payload is expecting an array of message, and each message is an object with the following parameters:
 
 - `provider` : the name of the provider you would like to direct this message to. Can be OpenAI, workers-ai, or any of our supported providers.
 - `endpoint`: the pathname of the provider API you’re trying to reach. For example, on OpenAI it can be `chat/completions`, and for Workers AI this might be [`@cf/meta/llama-3.1-8b-instruct`](/workers-ai/models/llama-3.1-8b-instruct/). See more in the sections that are specific to [each provider](/ai-gateway/providers/).
-- `authorization`: the content of the Authorization HTTP Header that should be used when contacting this provider. This usually starts with “Token” or “Bearer”.
+- `headers`:
+  - `Authorization`: the content of the Authorization HTTP Header that should be used when contacting this provider. This usually starts with “Token” or “Bearer”.
+  - Any other custom header in a model's [configuration](/ai-gateway/configuration/), such as [Caching](/ai-gateway/configuration/caching/) or [Authentication](/ai-gateway/configuration/authentication/).
 - `query`: the payload as the provider expects it in their official API.
 
 ## cURL example
 
+The following example shows a simple setup with a primary model and a [fallback](/ai-gateway/configuration/fallbacks/) option.
+
 <Render file="universal-gateway-example" />
 
-The above will send a request to Workers AI Inference API, if it fails it will proceed to OpenAI. You can add as many fallbacks as you need, just by adding another JSON in the array.
+The above will send a request to Workers AI Inference API, if it fails it will proceed to OpenAI. You can add as many [fallbacks](/ai-gateway/configuration/fallbacks/) as you need, just by adding another JSON in the array.
 
 ## WebSockets API <Badge text="beta" variant="tip" size="small" />
 

src/content/glossary/ai-gateway.yaml

@@ -41,6 +41,10 @@ entries:
     general_definition: |-
       Header to [bypass caching for a specific request](/ai-gateway/configuration/caching/#skip-cache-cf-aig-skip-cache).
 
+  - term: cf-aig-request-timeout
+    general_definition: |-
+      Header to trigger a fallback provider based on a [predetermined response time](/ai-gateway/configuration/fallbacks/#request-timeouts) (measured in milliseconds).
+
   # Deprecated headers
   - term: cf-cache-ttl
     general_definition: |-