Skip to content

Commit 4d27ab9

Browse files
daisyfaithaumadaisyfaithauma
committed
new docs for Websockets
1 parent 548271e commit 4d27ab9

File tree

3 files changed

+286
-0
lines changed

3 files changed

+286
-0
lines changed

‎src/content/docs/ai-gateway/configuration/Websockets API/index.mdx‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
---
2+
title: WebSockets API
3+
pcx_content_type: configuration
4+
sidebar:
5+
group:
6+
badge: Beta
7+
---
8+
9+
The AI Gateway WebSockets API provides a persistent connection for AI interactions, eliminating repeated handshakes and reducing latency. This API is divided into two categories:
10+
11+
1. **Non-Realtime APIs** - Supports standard WebSocket communication for AI providers, including those that do not natively support WebSockets.
12+
2. **Realtime APIs** - Designed for AI providers that offer low-latency, multimodal interactions over WebSockets.
13+
14+
## **Key differences**
15+
16+
| Feature | Non-Realtime APIs | Realtime APIs |
17+
| :---------------------- | :----------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------- |
18+
| **Purpose** | Supports WebSocket-based AI interactions with providers that do not natively support WebSockets. | Enables real-time, multimodal AI interactions for providers that offer dedicated WebSocket endpoints. |
19+
| **Use Case** | Text-based queries and responses, such as LLM requests. | Streaming responses for voice, video, and live interactions. |
20+
| **AI Provider Support** | All AI providers in AI Gateway. | Limited to providers offering real-time WebSocket APIs. |
21+
| **Streaming Support** | AI Gateway handles streaming via WebSockets. | Providers natively support real-time data streaming. |
22+
23+
For details on implementation, see the next section:
24+
25+
- [Non-Realtime WebSockets API](/ai-gateway/configuration/websockets/non-realtime-api.mdx)
26+
- [Realtime WebSockets API](ai-gateway/configuration/websockets/realtime-api.mdx)

‎src/content/docs/ai-gateway/configuration/Websockets API/non-realtime-api.mdx‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
---
2+
pcx_content_type: configuration
3+
title: Non-realtime WebSockets API
4+
sidebar:
5+
order: 2
6+
---
7+
8+
The Non-realtime WebSockets API allows you to establish persistent connections for AI requests without requiring repeated handshakes. This approach is ideal for applications that do not require real-time interactions but still benefit from reduced latency and continuous communication.
9+
10+
## Set up WebSockets API
11+
12+
1. Generate an AI Gateway token with appropriate AI Gateway Run and opt in to using an authenticated gateway.
13+
2. Modify your Universal Endpoint URL by replacing `https://` with `wss://` to initiate a WebSocket connection:
14+
```
15+
wss://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}
16+
```
17+
3. Open a WebSocket connection authenticated with a Cloudflare token with the AI Gateway Run permission.
18+
19+
:::note
20+
Alternatively, we also support authentication via the `sec-websocket-protocol` header if you are using a browser WebSocket.
21+
:::
22+
23+
## Example request
24+
25+
```javascript
26+
import WebSocket from "ws";
27+
28+
const ws = new WebSocket(
29+
"wss://gateway.ai.cloudflare.com/v1/my-account-id/my-gateway/",
30+
{
31+
headers: {
32+
"cf-aig-authorization": "Bearer AI_GATEWAY_TOKEN",
33+
},
34+
},
35+
);
36+
37+
ws.send(
38+
JSON.stringify({
39+
type: "universal.create",
40+
request: {
41+
eventId: "my-request",
42+
provider: "workers-ai",
43+
endpoint: "@cf/meta/llama-3.1-8b-instruct",
44+
headers: {
45+
Authorization: "Bearer WORKERS_AI_TOKEN",
46+
"Content-Type": "application/json",
47+
},
48+
query: {
49+
prompt: "tell me a joke",
50+
},
51+
},
52+
}),
53+
);
54+
55+
ws.on("message", function incoming(message) {
56+
console.log(message.toString());
57+
});
58+
```
59+
60+
## Example response
61+
62+
```json
63+
{
64+
"type": "universal.created",
65+
"metadata": {
66+
"cacheStatus": "MISS",
67+
"eventId": "my-request",
68+
"logId": "01JC3R94FRD97JBCBX3S0ZAXKW",
69+
"step": "0",
70+
"contentType": "application/json"
71+
},
72+
"response": {
73+
"result": {
74+
"response": "Why was the math book sad? Because it had too many problems. Would you like to hear another one?"
75+
},
76+
"success": true,
77+
"errors": [],
78+
"messages": []
79+
}
80+
}
81+
```
82+
83+
## Example streaming request
84+
85+
For streaming requests, AI Gateway sends an initial message with request metadata indicating the stream is starting:
86+
87+
```json
88+
{
89+
"type": "universal.created",
90+
"metadata": {
91+
"cacheStatus": "MISS",
92+
"eventId": "my-request",
93+
"logId": "01JC40RB3NGBE5XFRZGBN07572",
94+
"step": "0",
95+
"contentType": "text/event-stream"
96+
}
97+
}
98+
```
99+
100+
After this initial message, all streaming chunks are relayed in real-time to the WebSocket connection as they arrive from the inference provider. Only the `eventId` field is included in the metadata for these streaming chunks. The `eventId` allows AI Gateway to include a client-defined ID with each message, even in a streaming WebSocket environment.
101+
102+
```json
103+
{
104+
"type": "universal.stream",
105+
"metadata": {
106+
"eventId": "my-request"
107+
},
108+
"response": {
109+
"response": "would"
110+
}
111+
}
112+
```
113+
114+
Once all chunks for a request have been streamed, AI Gateway sends a final message to signal the completion of the request. For added flexibility, this message includes all the metadata again, even though it was initially provided at the start of the streaming process.
115+
116+
```json
117+
{
118+
"type": "universal.done",
119+
"metadata": {
120+
"cacheStatus": "MISS",
121+
"eventId": "my-request",
122+
"logId": "01JC40RB3NGBE5XFRZGBN07572",
123+
"step": "0",
124+
"contentType": "text/event-stream"
125+
}
126+
}
127+
```

‎src/content/docs/ai-gateway/configuration/Websockets API/realtime-api.mdx‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
---
2+
pcx_content_type: configuration
3+
title: Realtime WebSockets API
4+
sidebar:
5+
order: 3
6+
---
7+
8+
Some AI providers support real-time, low-latency interactions over WebSockets. AI Gateway allows seamless integration with these APIs, supporting multimodal interactions such as text, audio, and video.
9+
10+
## Supported Providers
11+
12+
- [OpenAI](https://platform.openai.com/docs/guides/realtime-websocket)
13+
- [Google AI Studio](https://ai.google.dev/gemini-api/docs/multimodal-live)
14+
- [Cartesia](https://docs.cartesia.ai/api-reference/tts/tts)
15+
- [ElevenLabs](https://elevenlabs.io/docs/conversational-ai/api-reference/conversational-ai/websocket)
16+
17+
## Authentication
18+
19+
For real-time WebSockets, authentication can be done using:
20+
21+
- Headers (for non-browser environments)
22+
- `sec-websocket-protocol` (for browsers)
23+
24+
## Examples
25+
26+
### OpenAI
27+
28+
```javascript
29+
import WebSocket from "ws";
30+
31+
const url =
32+
"wss://gateway.ai.cloudflare.com/v1/<account_id>/<gateway>/openai?model=gpt-4o-realtime-preview-2024-12-17";
33+
const ws = new WebSocket(url, {
34+
headers: {
35+
"cf-aig-authorization": process.env.CLOUDFLARE_API_KEY,
36+
Authorization: "Bearer " + process.env.OPENAI_API_KEY,
37+
"OpenAI-Beta": "realtime=v1",
38+
},
39+
});
40+
41+
ws.on("open", () => console.log("Connected to server."));
42+
ws.on("message", (message) => console.log(JSON.parse(message.toString())));
43+
44+
ws.send(
45+
JSON.stringify({
46+
type: "response.create",
47+
response: { modalities: ["text"], instructions: "Tell me a joke" },
48+
}),
49+
);
50+
```
51+
52+
### Google AI Studio
53+
54+
```javascript
55+
const ws = new WebSocket(
56+
"wss://gateway.ai.cloudflare.com/v1/<account_id>/<gateway>/google?api_key=<google_api_key>",
57+
["cf-aig-authorization.<cloudflare_token>"],
58+
);
59+
60+
ws.on("open", () => console.log("Connected to server."));
61+
ws.on("message", (message) => console.log(message.data));
62+
63+
ws.send(
64+
JSON.stringify({
65+
setup: {
66+
model: "models/gemini-2.0-flash-exp",
67+
generationConfig: { responseModalities: ["TEXT"] },
68+
},
69+
}),
70+
);
71+
```
72+
73+
### Cartesia
74+
75+
```javascript
76+
const ws = new WebSocket(
77+
"wss://gateway.ai.cloudflare.com/v1/<account_id>/<gateway>/cartesia?cartesia_version=2024-06-10&api_key=<cartesia_api_key>",
78+
["cf-aig-authorization.<cloudflare_token>"],
79+
);
80+
81+
ws.on("open", function open() {
82+
console.log("Connected to server.");
83+
});
84+
85+
ws.on("message", function incoming(message) {
86+
console.log(message.data);
87+
});
88+
89+
ws.send(
90+
JSON.stringify({
91+
model_id: "sonic",
92+
transcript: "Hello, world! I'm generating audio on ",
93+
voice: { mode: "id", id: "a0e99841-438c-4a64-b679-ae501e7d6091" },
94+
language: "en",
95+
context_id: "happy-monkeys-fly",
96+
output_format: {
97+
container: "raw",
98+
encoding: "pcm_s16le",
99+
sample_rate: 8000,
100+
},
101+
add_timestamps: true,
102+
continue: true,
103+
}),
104+
);
105+
```
106+
107+
### ElevenLabs
108+
109+
```javascript
110+
const ws = new WebSocket(
111+
"wss://gateway.ai.cloudflare.com/v1/<account_id>/<gateway>/elevenlabs?agent_id=<elevenlabs_agent_id>",
112+
[
113+
"xi-api-key.<elevenlabs_api_key>",
114+
"cf-aig-authorization.<cloudflare_token>",
115+
],
116+
);
117+
118+
ws.on("open", function open() {
119+
console.log("Connected to server.");
120+
});
121+
122+
ws.on("message", function incoming(message) {
123+
console.log(message.data);
124+
});
125+
126+
ws.send(
127+
JSON.stringify({
128+
text: "This is a sample text ",
129+
voice_settings: { stability: 0.8, similarity_boost: 0.8 },
130+
generation_config: { chunk_length_schedule: [120, 160, 250, 290] },
131+
}),
132+
);
133+
```

0 commit comments

Comments
 (0)