Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions src/content/docs/ai-gateway/providers/universal.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,75 @@ ws.on("message", function incoming(message) {
console.log(message.toString());
});
```

## Header configuration hierarchy

The Universal Endpoint allows you to set fallback models or providers and customize headers for each provider or request. You can configure headers at three levels:

1. **Provider level**: Headers specific to a particular provider.
2. **Request level**: Headers included in individual requests.
3. **Gateway settings**: Default headers configured in your gateway dashboard.

Since the same settings can be configured in multiple locations, AI Gateway applies a hierarchy to determine which configuration takes precedence:

- **Provider-level headers** override all other configurations.
- **Request-level headers** are used if no provider-level headers are set.
- **Gateway-level settings** are used only if no headers are configured at the provider or request levels.

This hierarchy ensures consistent behavior, prioritizing the most specific configurations. Use provider-level and request-level headers for fine-tuned control, and gateway settings for general defaults.

## Example

This example demonstrates how headers set at different levels impact caching behavior:

- **Request-level header**: The `cf-aig-cache-ttl` is set to `3600` seconds, applying this caching duration to the request by default.
- **Provider-level header**: For the fallback provider (OpenAI), `cf-aig-cache-ttl` is explicitly set to `0` seconds, overriding the request-level header and disabling caching for responses when OpenAI is used as the provider.

This shows how provider-level headers take precedence over request-level headers, allowing for granular control of caching behavior.

```bash
curl https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id} \
--header 'Content-Type: application/json' \
--header 'cf-aig-cache-ttl: 3600' \
--data '[
{
"provider": "workers-ai",
"endpoint": "@cf/meta/llama-3.1-8b-instruct",
"headers": {
"Authorization": "Bearer {cloudflare_token}",
"Content-Type": "application/json"
},
"query": {
"messages": [
{
"role": "system",
"content": "You are a friendly assistant"
},
{
"role": "user",
"content": "What is Cloudflare?"
}
]
}
},
{
"provider": "openai",
"endpoint": "chat/completions",
"headers": {
"Authorization": "Bearer {open_ai_token}",
"Content-Type": "application/json",
"cf-aig-cache-ttl": "0"
},
"query": {
"model": "gpt-4o-mini",
"stream": true,
"messages": [
{
"role": "user",
"content": "What is Cloudflare?"
}
]
}
}
]'
```
51 changes: 51 additions & 0 deletions src/content/glossary/ai-gateway.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
---
productName: AI Gateway
entries:
- term: cf-aig-event-id
general_definition: |-
[cf-aig-event-id](/ai-gateway/evaluations/add-human-feedback-api/#3-retrieve-the-cf-aig-log-id)Unique identifier for an event, used to trace specific events through the system.

- term: cf-aig-step
general_definition: |-
Identifies the processing step in the AI Gateway flow for better tracking and debugging.

- term: cf-aig-log-id
general_definition: |-
Identifier for logging, enabling detailed tracking of requests.

- term: cf-aig-collect-log
general_definition: |-
Specifies if logging data should be collected for a particular request.

- term: cf-aig-custom-cost
general_definition: |-
Allows the customization of request cost to reflect user-defined parameters.

- term: cf-aig-cache-key
general_definition: |-
Custom key to determine cache storage and retrieval.

- term: cf-aig-cache-status
general_definition: |-
Status indicator for caching, showing if a request was served from cache.

- term: cf-aig-metadata
general_definition: |-
Additional metadata associated with a request.

- term: cf-aig-cache-ttl
general_definition: |-
Specifies the cache time-to-live for responses.

- term: cf-aig-skip-cache
general_definition: |-
Header to bypass caching for a specific request.

# Deprecated headers
- term: cf-cache-ttl
general_definition: |-
Deprecated: This header is replaced by `cf-aig-cache-ttl`. It specifies cache time-to-live.

- term: cf-skip-cache
general_definition: |-
Deprecated: This header is replaced by `cf-aig-skip-cache`. It bypasses caching for a specific request.
Loading