From 359d0d016d45f90850071803dc44711c488c2fa3 Mon Sep 17 00:00:00 2001 From: daisyfaithauma Date: Mon, 18 Nov 2024 16:44:11 +0000 Subject: [PATCH 01/10] Initial websocket documentation --- .../configuration/websockets-api.mdx | 137 ++++++++++++++++++ src/content/docs/ai-gateway/get-started.mdx | 2 + .../docs/ai-gateway/providers/universal.mdx | 40 +++++ 3 files changed, 179 insertions(+) create mode 100644 src/content/docs/ai-gateway/configuration/websockets-api.mdx diff --git a/src/content/docs/ai-gateway/configuration/websockets-api.mdx b/src/content/docs/ai-gateway/configuration/websockets-api.mdx new file mode 100644 index 000000000000000..24a0df891f59d35 --- /dev/null +++ b/src/content/docs/ai-gateway/configuration/websockets-api.mdx @@ -0,0 +1,137 @@ +--- +title: Websockets API +pcx_content_type: configuration +sidebar: + badge: + text: Beta +--- + +The AI Gateway WebSockets API provides a single persistent connection, enabling continuous communication. By using WebSockets, you can establish a single connection for multiple AI requests, eliminating the need for repeated handshakes and TLS negotiations, which enhances performance and reduces latency. This API supports all AI providers connected to AI Gateway, including those that don’t natively support WebSockets. + +## When to use webSockets? + +WebSockets are long-lived TCP connections that enable bi-directional, real-time communication between client and server. Unlike HTTP connections, which require repeated handshakes for each request, WebSocket maintains the connection, supporting continuous data exchange with reduced overhead. WebSockets are ideal for applications needing low-latency, real-time data, such as voice assistants. + +## Key benefits + +- **Reduced Overhead**: Avoid overhead of repeated handshakes and TLS negotiations by maintaining a single, persistent connection. +- **Provider Compatibility**: Works with all AI providers in AI Gateway. Even if your chosen provider doesn’t support WebSockets, we handle it for you, managing the requests to your preferred AI provider. + +## Setting up + +1. Generate an AI Gateway token with appropriate AI Gateway Run and opt in to using an authenticated gateway. +2. Modify your Universal Endpoint URL by replacing `https://` with `wss://` to initiate a WebSocket connection: + ``` + wss://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id} + ``` +3. Open a WebSocket connection authenticated with a Cloudflare token with the AI Gateway Run permission. + +:::note +Alternatively, we also support authentication via the `sec-websocket-protocol` header if you are using a browser WebSocket. +::: + +## Example request + +```javascript +import WebSocket from "ws"; + +const ws = new WebSocket( + "wss://gateway.ai.cloudflare.com/v1/my-account-id/my-gateway/", + { + headers: { + "cf-aig-authorization": "Bearer AI_GATEWAY_TOKEN", + }, + }, +); + +ws.send( + JSON.stringify({ + type: "universal.create", + request: { + eventId: "my-request", + provider: "workers-ai", + endpoint: "@cf/meta/llama-3.1-8b-instruct", + headers: { + Authorization: "Bearer WORKERS_AI_TOKEN", + "Content-Type": "application/json", + }, + query: { + prompt: "tell me a joke", + }, + }, + }), +); + +ws.on("message", function incoming(message) { + console.log(message.toString()); +}); +``` + +## Example response + +```json +{ + "type": "universal.created", + "metadata": { + "cacheStatus": "MISS", + "eventId": "my-request", + "logId": "01JC3R94FRD97JBCBX3S0ZAXKW", + "step": "0", + "contentType": "application/json" + }, + "response": { + "result": { + "response": "Why was the math book sad? Because it had too many problems. Would you like to hear another one?" + }, + "success": true, + "errors": [], + "messages": [] + } +} +``` + +## Example for streaming requests + +For streaming requests, AI Gateway sends an initial message with request metadata indicating the stream is starting: + +```json +{ + "type": "universal.created", + "metadata": { + "cacheStatus": "MISS", + "eventId": "my-request", + "logId": "01JC40RB3NGBE5XFRZGBN07572", + "step": "0", + "contentType": "text/event-stream" + } +} +``` + +After this initial message, all streaming chunks are relayed in real-time to the WebSocket connection as they arrive from the inference provider. Only the `eventId` field is included in the metadata for these streaming chunks. The `eventId` allows AI Gateway to include a client-defined ID with each message, even in a streaming WebSocket environment. + +```json +{ + "type": "universal.stream", + "metadata": { + "eventId": "my-request" + }, + "response": { + "response": "would" + } +} +``` + +Once all chunks for a request have been streamed, AI Gateway sends a final message to signal the request’s completion. For added flexibility, this message includes all the metadata again, even though it was initially provided at the start of the streaming process. + +```json +{ + "type": "universal.done", + "metadata": { + "cacheStatus": "MISS", + "eventId": "my-request", + "logId": "01JC40RB3NGBE5XFRZGBN07572", + "step": "0", + "contentType": "text/event-stream" + } +} +``` diff --git a/src/content/docs/ai-gateway/get-started.mdx b/src/content/docs/ai-gateway/get-started.mdx index 8173bc564600c50..e7aa1e6fa8c3b8a 100644 --- a/src/content/docs/ai-gateway/get-started.mdx +++ b/src/content/docs/ai-gateway/get-started.mdx @@ -30,6 +30,8 @@ Next, connect your AI provider to your gateway. AI Gateway offers multiple endpoints for each Gateway you create - one endpoint per provider, and one Universal Endpoint. +Additionally, AI Gateway has a [WebSockets API](/ai-gateway/configuration/websockets-api/) which provides a single persistent connection, enabling continuous communication. This API supports all AI providers connected to AI Gateway, including those that don’t natively support WebSockets. + Below is a list of our supported model providers: diff --git a/src/content/docs/ai-gateway/providers/universal.mdx b/src/content/docs/ai-gateway/providers/universal.mdx index 452a8a8658e49de..f841902e9550f00 100644 --- a/src/content/docs/ai-gateway/providers/universal.mdx +++ b/src/content/docs/ai-gateway/providers/universal.mdx @@ -30,3 +30,43 @@ You can use the Universal endpoint to contact every provider. The payload is exp The above will send a request to Workers AI Inference API, if it fails it will proceed to OpenAI. You can add as many fallbacks as you need, just by adding another JSON in the array. + +## Websockets API (beta) + +The Universal Endpoint can also be accessed via a [WebSockets API](/ai-gateway/configuration/websockets-api/) which provides a single persistent connection, enabling continuous communication. This API supports all AI providers connected to AI Gateway, including those that don’t natively support WebSockets. + +## Example request + +```javascript +import WebSocket from "ws"; +const ws = new WebSocket( + "wss://gateway.ai.cloudflare.com/v1/my-account-id/my-gateway/", + { + headers: { + "cf-aig-authorization": "Bearer AI_GATEWAY_TOKEN", + }, + }, +); + +ws.send( + JSON.stringify({ + type: "universal.create", + request: { + eventId: "my-request", + provider: "workers-ai", + endpoint: "@cf/meta/llama-3.1-8b-instruct", + headers: { + Authorization: "Bearer WORKERS_AI_TOKEN", + "Content-Type": "application/json", + }, + query: { + prompt: "tell me a joke", + }, + }, + }), +); + +ws.on("message", function incoming(message) { + console.log(message.toString()); +}); +``` From 5ca48b98ac8b058a27413384e9a94a08f3aeaa8c Mon Sep 17 00:00:00 2001 From: daisyfaithauma Date: Mon, 18 Nov 2024 16:51:38 +0000 Subject: [PATCH 02/10] Update src/content/docs/ai-gateway/get-started.mdx Co-authored-by: hyperlint-ai[bot] <154288675+hyperlint-ai[bot]@users.noreply.github.com> --- src/content/docs/ai-gateway/get-started.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/docs/ai-gateway/get-started.mdx b/src/content/docs/ai-gateway/get-started.mdx index e7aa1e6fa8c3b8a..22bafe46e30e540 100644 --- a/src/content/docs/ai-gateway/get-started.mdx +++ b/src/content/docs/ai-gateway/get-started.mdx @@ -30,7 +30,7 @@ Next, connect your AI provider to your gateway. AI Gateway offers multiple endpoints for each Gateway you create - one endpoint per provider, and one Universal Endpoint. -Additionally, AI Gateway has a [WebSockets API](/ai-gateway/configuration/websockets-api/) which provides a single persistent connection, enabling continuous communication. This API supports all AI providers connected to AI Gateway, including those that don’t natively support WebSockets. +Additionally, AI Gateway has a [WebSockets API](/ai-gateway/configuration/websockets-api/) which provides a single persistent connection, enabling continuous communication. This API supports all AI providers connected to AI Gateway, including those that don't natively support WebSockets. Below is a list of our supported model providers: From 4525cd3179fc6f5b8d177d98120e2a654d63dcc7 Mon Sep 17 00:00:00 2001 From: daisyfaithauma Date: Mon, 18 Nov 2024 16:51:52 +0000 Subject: [PATCH 03/10] Update src/content/docs/ai-gateway/providers/universal.mdx Co-authored-by: hyperlint-ai[bot] <154288675+hyperlint-ai[bot]@users.noreply.github.com> --- src/content/docs/ai-gateway/providers/universal.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/docs/ai-gateway/providers/universal.mdx b/src/content/docs/ai-gateway/providers/universal.mdx index f841902e9550f00..3be08059e02ec76 100644 --- a/src/content/docs/ai-gateway/providers/universal.mdx +++ b/src/content/docs/ai-gateway/providers/universal.mdx @@ -33,7 +33,7 @@ The above will send a request to Workers AI Inference API, if it fails it will p ## Websockets API (beta) -The Universal Endpoint can also be accessed via a [WebSockets API](/ai-gateway/configuration/websockets-api/) which provides a single persistent connection, enabling continuous communication. This API supports all AI providers connected to AI Gateway, including those that don’t natively support WebSockets. +The Universal Endpoint can also be accessed via a [WebSockets API](/ai-gateway/configuration/websockets-api/) which provides a single persistent connection, enabling continuous communication. This API supports all AI providers connected to AI Gateway, including those that don't natively support WebSockets. ## Example request From a4f94b98782d48ad873f0719b436e8eb3dc54afa Mon Sep 17 00:00:00 2001 From: daisyfaithauma Date: Mon, 18 Nov 2024 16:52:02 +0000 Subject: [PATCH 04/10] Update src/content/docs/ai-gateway/configuration/websockets-api.mdx Co-authored-by: hyperlint-ai[bot] <154288675+hyperlint-ai[bot]@users.noreply.github.com> --- src/content/docs/ai-gateway/configuration/websockets-api.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/docs/ai-gateway/configuration/websockets-api.mdx b/src/content/docs/ai-gateway/configuration/websockets-api.mdx index 24a0df891f59d35..e1186390e152387 100644 --- a/src/content/docs/ai-gateway/configuration/websockets-api.mdx +++ b/src/content/docs/ai-gateway/configuration/websockets-api.mdx @@ -6,7 +6,7 @@ sidebar: text: Beta --- -The AI Gateway WebSockets API provides a single persistent connection, enabling continuous communication. By using WebSockets, you can establish a single connection for multiple AI requests, eliminating the need for repeated handshakes and TLS negotiations, which enhances performance and reduces latency. This API supports all AI providers connected to AI Gateway, including those that don’t natively support WebSockets. +The AI Gateway WebSockets API provides a single persistent connection, enabling continuous communication. By using WebSockets, you can establish a single connection for multiple AI requests, eliminating the need for repeated handshakes and TLS negotiations, which enhances performance and reduces latency. This API supports all AI providers connected to AI Gateway, including those that don't natively support WebSockets. ## When to use webSockets? From 2113745ec1eb2f976015b100d4e9fb9f7f60fa6e Mon Sep 17 00:00:00 2001 From: daisyfaithauma Date: Mon, 18 Nov 2024 16:52:11 +0000 Subject: [PATCH 05/10] Update src/content/docs/ai-gateway/configuration/websockets-api.mdx Co-authored-by: hyperlint-ai[bot] <154288675+hyperlint-ai[bot]@users.noreply.github.com> --- src/content/docs/ai-gateway/configuration/websockets-api.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/docs/ai-gateway/configuration/websockets-api.mdx b/src/content/docs/ai-gateway/configuration/websockets-api.mdx index e1186390e152387..7c59a927ad7c383 100644 --- a/src/content/docs/ai-gateway/configuration/websockets-api.mdx +++ b/src/content/docs/ai-gateway/configuration/websockets-api.mdx @@ -15,7 +15,7 @@ WebSockets are long-lived TCP connections that enable bi-directional, real-time ## Key benefits - **Reduced Overhead**: Avoid overhead of repeated handshakes and TLS negotiations by maintaining a single, persistent connection. -- **Provider Compatibility**: Works with all AI providers in AI Gateway. Even if your chosen provider doesn’t support WebSockets, we handle it for you, managing the requests to your preferred AI provider. +- **Provider Compatibility**: Works with all AI providers in AI Gateway. Even if your chosen provider doesn't support WebSockets, we handle it for you, managing the requests to your preferred AI provider. ## Setting up From 50227938131472e1974ab6a1a080661e163466d7 Mon Sep 17 00:00:00 2001 From: daisyfaithauma Date: Mon, 18 Nov 2024 16:52:22 +0000 Subject: [PATCH 06/10] Update src/content/docs/ai-gateway/configuration/websockets-api.mdx Co-authored-by: hyperlint-ai[bot] <154288675+hyperlint-ai[bot]@users.noreply.github.com> --- src/content/docs/ai-gateway/configuration/websockets-api.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/docs/ai-gateway/configuration/websockets-api.mdx b/src/content/docs/ai-gateway/configuration/websockets-api.mdx index 7c59a927ad7c383..d83156479112b48 100644 --- a/src/content/docs/ai-gateway/configuration/websockets-api.mdx +++ b/src/content/docs/ai-gateway/configuration/websockets-api.mdx @@ -121,7 +121,7 @@ After this initial message, all streaming chunks are relayed in real-time to the } ``` -Once all chunks for a request have been streamed, AI Gateway sends a final message to signal the request’s completion. For added flexibility, this message includes all the metadata again, even though it was initially provided at the start of the streaming process. +Once all chunks for a request have been streamed, AI Gateway sends a final message to signal the request's completion. For added flexibility, this message includes all the metadata again, even though it was initially provided at the start of the streaming process. ```json { From ac8eaf1b6f3a0463f7ad6e440546d2f3fa6b44d3 Mon Sep 17 00:00:00 2001 From: daisyfaithauma Date: Mon, 18 Nov 2024 16:57:17 +0000 Subject: [PATCH 07/10] grammar --- .../docs/ai-gateway/configuration/websockets-api.mdx | 6 +++--- src/content/docs/ai-gateway/get-started.mdx | 2 +- src/content/docs/ai-gateway/providers/universal.mdx | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/src/content/docs/ai-gateway/configuration/websockets-api.mdx b/src/content/docs/ai-gateway/configuration/websockets-api.mdx index d83156479112b48..1bd0ccb381b0f7a 100644 --- a/src/content/docs/ai-gateway/configuration/websockets-api.mdx +++ b/src/content/docs/ai-gateway/configuration/websockets-api.mdx @@ -6,7 +6,7 @@ sidebar: text: Beta --- -The AI Gateway WebSockets API provides a single persistent connection, enabling continuous communication. By using WebSockets, you can establish a single connection for multiple AI requests, eliminating the need for repeated handshakes and TLS negotiations, which enhances performance and reduces latency. This API supports all AI providers connected to AI Gateway, including those that don't natively support WebSockets. +The AI Gateway WebSockets API provides a single persistent connection, enabling continuous communication. By using WebSockets, you can establish a single connection for multiple AI requests, eliminating the need for repeated handshakes and TLS negotiations, which enhances performance and reduces latency. This API supports all AI providers connected to AI Gateway, including those that do not natively support WebSockets. ## When to use webSockets? @@ -15,7 +15,7 @@ WebSockets are long-lived TCP connections that enable bi-directional, real-time ## Key benefits - **Reduced Overhead**: Avoid overhead of repeated handshakes and TLS negotiations by maintaining a single, persistent connection. -- **Provider Compatibility**: Works with all AI providers in AI Gateway. Even if your chosen provider doesn't support WebSockets, we handle it for you, managing the requests to your preferred AI provider. +- **Provider Compatibility**: Works with all AI providers in AI Gateway. Even if your chosen provider does not support WebSockets, we handle it for you, managing the requests to your preferred AI provider. ## Setting up @@ -121,7 +121,7 @@ After this initial message, all streaming chunks are relayed in real-time to the } ``` -Once all chunks for a request have been streamed, AI Gateway sends a final message to signal the request's completion. For added flexibility, this message includes all the metadata again, even though it was initially provided at the start of the streaming process. +Once all chunks for a request have been streamed, AI Gateway sends a final message to signal the completion of the request. For added flexibility, this message includes all the metadata again, even though it was initially provided at the start of the streaming process. ```json { diff --git a/src/content/docs/ai-gateway/get-started.mdx b/src/content/docs/ai-gateway/get-started.mdx index 22bafe46e30e540..e146c0588e28445 100644 --- a/src/content/docs/ai-gateway/get-started.mdx +++ b/src/content/docs/ai-gateway/get-started.mdx @@ -30,7 +30,7 @@ Next, connect your AI provider to your gateway. AI Gateway offers multiple endpoints for each Gateway you create - one endpoint per provider, and one Universal Endpoint. -Additionally, AI Gateway has a [WebSockets API](/ai-gateway/configuration/websockets-api/) which provides a single persistent connection, enabling continuous communication. This API supports all AI providers connected to AI Gateway, including those that don't natively support WebSockets. +Additionally, AI Gateway has a [WebSockets API](/ai-gateway/configuration/websockets-api/) which provides a single persistent connection, enabling continuous communication. This API supports all AI providers connected to AI Gateway, including those that do not natively support WebSockets. Below is a list of our supported model providers: diff --git a/src/content/docs/ai-gateway/providers/universal.mdx b/src/content/docs/ai-gateway/providers/universal.mdx index 3be08059e02ec76..2c87f3c7d728172 100644 --- a/src/content/docs/ai-gateway/providers/universal.mdx +++ b/src/content/docs/ai-gateway/providers/universal.mdx @@ -33,7 +33,7 @@ The above will send a request to Workers AI Inference API, if it fails it will p ## Websockets API (beta) -The Universal Endpoint can also be accessed via a [WebSockets API](/ai-gateway/configuration/websockets-api/) which provides a single persistent connection, enabling continuous communication. This API supports all AI providers connected to AI Gateway, including those that don't natively support WebSockets. +The Universal Endpoint can also be accessed via a [WebSockets API](/ai-gateway/configuration/websockets-api/) which provides a single persistent connection, enabling continuous communication. This API supports all AI providers connected to AI Gateway, including those that do not natively support WebSockets. ## Example request From 61f4325b5bf908d7ef7a55cd84b8812ce6a63a74 Mon Sep 17 00:00:00 2001 From: Kathy <153706637+kathayl@users.noreply.github.com> Date: Mon, 18 Nov 2024 10:37:29 -0800 Subject: [PATCH 08/10] Update websockets-api.mdx --- src/content/docs/ai-gateway/configuration/websockets-api.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/content/docs/ai-gateway/configuration/websockets-api.mdx b/src/content/docs/ai-gateway/configuration/websockets-api.mdx index 1bd0ccb381b0f7a..c394215280de2b9 100644 --- a/src/content/docs/ai-gateway/configuration/websockets-api.mdx +++ b/src/content/docs/ai-gateway/configuration/websockets-api.mdx @@ -10,7 +10,7 @@ The AI Gateway WebSockets API provides a single persistent connection, enabling ## When to use webSockets? -WebSockets are long-lived TCP connections that enable bi-directional, real-time communication between client and server. Unlike HTTP connections, which require repeated handshakes for each request, WebSocket maintains the connection, supporting continuous data exchange with reduced overhead. WebSockets are ideal for applications needing low-latency, real-time data, such as voice assistants. +WebSockets are long-lived TCP connections that enable bi-directional, real-time communication between client and server. Unlike HTTP connections, which require repeated handshakes for each request, WebSockets maintain the connection, supporting continuous data exchange with reduced overhead. WebSockets are ideal for applications needing low-latency, real-time data, such as voice assistants. ## Key benefits @@ -90,7 +90,7 @@ ws.on("message", function incoming(message) { } ``` -## Example for streaming requests +## Example streaming request For streaming requests, AI Gateway sends an initial message with request metadata indicating the stream is starting: From f488d87670c2424da973a7986499f346c702fcc6 Mon Sep 17 00:00:00 2001 From: daisyfaithauma Date: Tue, 19 Nov 2024 12:35:20 +0000 Subject: [PATCH 09/10] added a badge --- src/content/docs/ai-gateway/providers/universal.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/content/docs/ai-gateway/providers/universal.mdx b/src/content/docs/ai-gateway/providers/universal.mdx index 2c87f3c7d728172..99e4f83ba625de9 100644 --- a/src/content/docs/ai-gateway/providers/universal.mdx +++ b/src/content/docs/ai-gateway/providers/universal.mdx @@ -6,7 +6,7 @@ sidebar: order: 1 --- -import { Render } from "~/components"; +import { Render, Badge } from "~/components"; You can use the Universal Endpoint to contact every provider. @@ -31,7 +31,7 @@ You can use the Universal endpoint to contact every provider. The payload is exp The above will send a request to Workers AI Inference API, if it fails it will proceed to OpenAI. You can add as many fallbacks as you need, just by adding another JSON in the array. -## Websockets API (beta) +## Websockets API The Universal Endpoint can also be accessed via a [WebSockets API](/ai-gateway/configuration/websockets-api/) which provides a single persistent connection, enabling continuous communication. This API supports all AI providers connected to AI Gateway, including those that do not natively support WebSockets. From 2368b007d5f869c04c79ccdbbb51675623115147 Mon Sep 17 00:00:00 2001 From: daisyfaithauma Date: Tue, 19 Nov 2024 12:48:31 +0000 Subject: [PATCH 10/10] Update src/content/docs/ai-gateway/configuration/websockets-api.mdx Co-authored-by: Maddy <130055405+Maddy-Cloudflare@users.noreply.github.com> --- src/content/docs/ai-gateway/configuration/websockets-api.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/docs/ai-gateway/configuration/websockets-api.mdx b/src/content/docs/ai-gateway/configuration/websockets-api.mdx index c394215280de2b9..70737aa6dd98afe 100644 --- a/src/content/docs/ai-gateway/configuration/websockets-api.mdx +++ b/src/content/docs/ai-gateway/configuration/websockets-api.mdx @@ -17,7 +17,7 @@ WebSockets are long-lived TCP connections that enable bi-directional, real-time - **Reduced Overhead**: Avoid overhead of repeated handshakes and TLS negotiations by maintaining a single, persistent connection. - **Provider Compatibility**: Works with all AI providers in AI Gateway. Even if your chosen provider does not support WebSockets, we handle it for you, managing the requests to your preferred AI provider. -## Setting up +## Set up WebSockets API 1. Generate an AI Gateway token with appropriate AI Gateway Run and opt in to using an authenticated gateway. 2. Modify your Universal Endpoint URL by replacing `https://` with `wss://` to initiate a WebSocket connection: