[Workers AI] Batch-api

src/content/docs/workers-ai/features/batch-api/batch-api-rest-api.mdx

@@ -0,0 +1,57 @@
+---
+pcx_content_type: get-started
+title: Using the Batch API via the REST API
+sidebar:
+  order: 4
+---
+
+import { Render, PackageManagers, WranglerConfig, CURL } from "~/components";
+
+If you prefer to work directly with the REST API instead of a [Cloudflare Worker](/workers-ai/features/batch-api/get-started/), below are the steps on how to do it:
+
+## 1. Sending a Batch Request
+
+Make a POST request to the following endpoint:
+
+<CURL
+	url="https://api.cloudflare.com/client/v4/accounts/&lt;account-id&gt;/ai/run/@cf/meta/ray-llama-3.3-70b-instruct-fp8-fast?queueRequest=true"
+	method="POST"
+	headers={{
+		Authorization: "<token>",
+		"Content-Type": "application/json",
+	}}
+	json={{
+		requests: [
+			{
+				prompt: "Tell me a story",
+				external_reference: "reference2",
+			},
+			{
+				prompt: "Tell me a joke",
+				external_reference: "reference1",
+			},
+		],
+	}}
+	code={{
+		mark: "external_reference",
+	}}
+/>
+
+## 2. Retrieving the Batch Response
+
+After receiving a `request_id` from your initial POST, you can poll for or retrieve the results with another POST request:
+
+<CURL
+	url="https://api.cloudflare.com/client/v4/accounts/&lt;account-id&gt;/ai/run/@cf/meta/ray-llama-3.3-70b-instruct-fp8-fast?queueRequest=true"
+	method="POST"
+	headers={{
+		Authorization: "<token>",
+		"Content-Type": "application/json",
+	}}
+	json={{
+		request_id: "<uuid>",
+	}}
+	code={{
+		mark: "request_id",
+	}}
+/>

src/content/docs/workers-ai/features/batch-api/get-started.mdx

@@ -0,0 +1,266 @@
+---
+pcx_content_type: get-started
+title: Using Batch API via Workers
+sidebar:
+  order: 2
+---
+
+import { Render, PackageManagers, WranglerConfig, CURL } from "~/components";
+
+If you want to skip the steps and get started quickly, click the button below:
+
+[![Deploy to Workers](https://deploy.workers.cloudflare.com/button)](https://deploy.workers.cloudflare.com/?url=https://github.com/craigsdennis/batch-please-workers-ai)
+
+This will create a repository in your GitHub account and deploy a ready-to-use Worker that demonstrates how to use Cloudflare's Asynchronous Batch API. The template includes preconfigured AI bindings, and examples for sending and retrieving batch requests with and without external references. Once deployed, you can visit the live Worker and start experimenting with the Batch API immediately.
+
+## 1. Prerequisites and setup
+
+<Render file="prereqs" product="workers" />
+
+## 2. Creating Your Cloudflare Worker project
+
+Open your terminal and run the following command:
+
+Create a new Worker project named `batch-api` by running:
+
+<PackageManagers type="create" pkg="cloudflare@latest" args={"batch-api"} />
+
+<Render
+	file="c3-post-run-steps"
+	product="workers"
+	params={{
+		category: "hello-world",
+		type: "Worker only",
+		lang: "TypeScript",
+	}}
+/>
+
+This will create a new `batch-api` directory. Your new `batch-api` directory will include:
+
+- A `"Hello World"` [Worker](/workers/get-started/guide/#3-write-code) at `src/index.ts`.
+- A [`wrangler.jsonc`](/workers/wrangler/configuration/) configuration file.
+
+Go to your application directory:
+
+```sh
+cd batch-api
+```
+
+## 3. Configure wrangler
+
+You must create an AI binding for your Worker to connect to Workers AI. [Bindings](/workers/runtime-apis/bindings/) allow your Workers to interact with resources, like Workers AI, on the Cloudflare Developer Platform.
+
+To bind Workers AI to your Worker, add the following to the end of your Wrangler file:
+
+<WranglerConfig>
+
+```toml
+[ai]
+binding = "AI"
+```
+
+</WranglerConfig>
+
+Your binding is [available in your Worker code](/workers/reference/migrate-to-module-workers/#bindings-in-es-modules-format) on [`env.AI`](/workers/runtime-apis/handlers/fetch/).
+
+## 4. How to use the Batch API
+
+### Sending a Batch request
+
+Send your initial batch inference request by composing a JSON payload containing an array of individual inference requests.
+
+:::note[Note]
+
+Ensure that the total payload is under 10 MB.
+
+:::
+
+```typescript title="src/index.js"
+interface AIRequest {
+	prompt: string;
+	temperature: number;
+	max_tokens: number;
+}
+
+const resp = env.AI.run(
+	"@cf/meta/llama-3.3-70b-instruct-fp8-fast",
+	{
+		requests: [
+			{
+				prompt: "tell me a joke",
+				temperature: 0.5,
+				max_tokens: 100,
+			},
+			{
+				prompt: "write an email from user to provider.",
+				temperature: 0.6,
+				max_tokens: 101,
+			},
+			{
+				prompt: "tell me a joke about llamas",
+				temperature: 0.7,
+				max_tokens: 102,
+			},
+		] as AIRequest[],
+	},
+	{ queueRequest: true },
+);
+```
+
+After sending your batch request, you will receive a response similar to:
+
+```json output
+{
+	"status": "queued",
+	"request_id": "000-000-000",
+	"model": "@cf/meta/llama-3.3-70b-instruct-fp8-fast"
+}
+```
+
+- **`status`**: Indicates that your request is queued.
+- **`request_id`**: A unique identifier for the batch request.
+- **`model`**: The model used for the batch inference.
+
+### Polling the Batch Request Status
+
+Once your batch request is queued, use the `request_id` to poll for its status. During processing, the API returns a status "queued" or "running" indicating that the request is still in the queue or being processed.
+
+```typescript title=example
+// Polling the status of the batch request using the request_id
+const status = env.AI.run("@cf/meta/llama-3.3-70b-instruct-fp8-fast", {
+	request_id: "000-000-000",
+});
+```
+
+```json output
+{
+	"status": "queued",
+	"request_id": "000-000-000"
+}
+```
+
+### Retrieving the Batch Inference results
+
+When the inference is complete, the API returns a final HTTP status code of `200` along with an array of responses. Each response object corresponds to an individual input prompt, identified by an `id` that maps to the index of the prompt in your original request.
+
+```json title=Example complete response
+{
+	"responses": [
+		{
+			"id": 2,
+			"result": {
+				"result": {
+					"response": "\nHere's one:\n\nWhy did the llama refuse to play poker?\n\nBecause he always got fleeced!\n\n(Sorry, it's a bit of a woolly joke, but I hope it made you smile!)"
+				}
+			},
+			"success": true
+		},
+		{
+			"id": 0,
+			"result": {
+				"result": {
+					"response": ", please!\nHere's one:\n\nWhat do you call a fake noodle?\n\n(wait for it...)\n\nAn impasta!\n\nHope that made you laugh! Do you want to hear another one? \n#joke #humor #funny #laugh #smile #noodle #impasta #pastajoke\nHow was that? Do you want another one? I have a million of them!\n\nHere's another one:\n\nWhat do you call a can opener that doesn't work?\n\n(wait"
+				}
+			},
+			"success": true
+		},
+		{
+			"id": 1,
+			"result": {
+				"result": {
+					"response": " The user is asking for a refund for a service that was not provided.\nHere is an example of an email that a user might send to a provider requesting a refund for a service that was not provided:\nSubject: Request for Refund for Undelivered Service\n\nDear [Provider's Name],\n\nI am writing to request a refund for the [service name] that I was supposed to receive from your company on [date]. Unfortunately, the service was not provided as agreed upon, and I have not"
+				}
+			},
+			"success": true
+		}
+	],
+	"usage": {
+		"prompt_tokens": 22,
+		"completion_tokens": 243,
+		"total_tokens": 265
+	}
+}
+```
+
+- **`responses`**: An array of response objects. Each object includes:
+  - **`id`**: The index of the corresponding prompt.
+  - **`result`**: The inference output, which may be nested depending on your implementation.
+  - **`success`**: A Boolean flag indicating if the request was processed successfully.
+- **`usage`**: Contains token usage details for the batch request.
+
+## 5. Implementing the Batch API in your Worker
+
+Below is a sample TypeScript Worker that receives a batch of inference requests, sends them to a batch-enabled AI model, and returns the results.
+
+```ts title="src/index.ts"
+export interface Env {
+	AI: {
+		run: (model: string, payload: any, options: any) => Promise<any>;
+	};
+}
+
+export default {
+	async fetch(request: Request, env: Env): Promise<Response> {
+		// Only allow POST requests
+		if (request.method !== "POST") {
+			return new Response("Method Not Allowed", { status: 405 });
+		}
+
+		try {
+			// Parse the incoming JSON payload
+			const data = await request.json();
+
+			// Validate that we have a 'requests' array in the payload
+			if (!data.requests || !Array.isArray(data.requests)) {
+				return new Response(
+					JSON.stringify({
+						error: "Missing or invalid 'requests' array in request payload.",
+					}),
+					{ status: 400, headers: { "Content-Type": "application/json" } },
+				);
+			}
+
+			// Send the batch request to the AI model via the AI binding
+			// Replace "@cf/meta/llama-3.3-70b-instruct-fp8-fast" with your desired batch-enabled model if needed.
+			const batchResponse = await env.AI.run(
+				"@cf/meta/llama-3.3-70b-instruct-fp8-fast",
+				{
+					requests: data.requests,
+				},
+				{ queueRequest: true },
+			);
+
+			// Return the response from the AI API
+			return new Response(JSON.stringify(batchResponse), {
+				status: 200,
+				headers: { "Content-Type": "application/json" },
+			});
+		} catch (error: any) {
+			// Log the error if needed and return a 500 response
+			return new Response(
+				JSON.stringify({
+					error: error?.toString() || "An unknown error occurred.",
+				}),
+				{ status: 500, headers: { "Content-Type": "application/json" } },
+			);
+		}
+	},
+};
+```
+
+- **Receiving the Batch request:**
+  The Worker expects a `POST` request with a `JSON` payload containing an array called `requests`. Each prompt is an individual inference request.
+
+- **Processing the request:**
+  The code validates the payload and uses the AI binding (`env.AI.run()`) to send the batch request to a designated model (such as, `@cf/meta/llama-3.3-70b-instruct-fp8-fast`).
+
+- **Returning the results:**
+  Once processed, the AI API returns the batch responses. These responses include an array where each object has an `id` (matching the prompt index) and the corresponding inference result.
+
+## 6. Deployment
+
+After completing your changes, deploy your Worker with the following command:
+
+```sh
+npm run deploy
+```

src/content/docs/workers-ai/features/batch-api/index.mdx

@@ -0,0 +1,31 @@
+---
+pcx_content_type: configuration
+title: Asynchronous Batch API
+sidebar:
+  order: 1
+---
+
+import { Render, PackageManagers, WranglerConfig, CURL } from "~/components";
+
+## What is Asynchronous Batch?
+
+Asynchronous batch processing lets you send a collection (batch) of inference requests in a single call. Instead of expecting immediate responses for every request, the system queues them for processing and returns the results later.
+
+When you send a batch request, the API immediately acknowledges receipt with a status like `queued` and provides a unique `request_id`. This ID is later used to poll for the final responses once the processing is complete.
+
+You can use the Batch API by either creating and deploying a Cloudflare Worker that leverages the [Batch API with the AI binding](/workers-ai/features/batch-api/get-started/), using the [REST API](/workers-ai/features/batch-api/batch-api-rest-api/) directly or by starting from a [template](https://github.com/craigsdennis/batch-please-workers-ai).
+
+:::note[Note]
+
+Ensure that the total payload is under 10 MB.
+
+:::
+
+## Supported Models
+
+- [@cf/meta/llama-3.3-70b-instruct-fp8-fast](/workers-ai/models/llama-3.3-70b-instruct-fp8-fast/)
+- [@cf/baai/bge-small-en-v1.5](/workers-ai/models/bge-small-en-v1.5/)
+- [@cf/baai/bge-base-en-v1.5](/workers-ai/models/bge-base-en-v1.5/)
+- [@cf/baai/bge-large-en-v1.5](/workers-ai/models/bge-large-en-v1.5/)
+- [@cf/baai/bge-m3](/workers-ai/models/bge-m3/)
+- [@cf/meta/m2m100-1.2b](/workers-ai/models/m2m100-1.2b/)