metadata filtering support

Author: aninibread

src/content/docs/autorag/configuration/metadata-filtering.mdx

@@ -0,0 +1,117 @@
+---
+pcx_content_type: concept
+title: Metadata filtering
+sidebar:
+  order: 6
+---
+
+Metadata filtering narrows down search results based on metadata, so only relevant content is retrieved. The filter narrows down results prior to retrieval, so that you only query the scope of documents that matter.
+
+Here is an example of metadata filtering in [Workers Binding](/autorag/usage/workers-binding/) but it can be easily adapted to use the [REST API](/autorag/usage/rest-api/) instead.
+
+```js
+const answer = await env.AI.autorag("my-autorag").search({
+	query: "How do I train a llama to deliver coffee?",
+	filters: {
+		type: "and",
+		filters: [
+			{
+				type: "eq",
+				key: "folder",
+				value: "llama/logistics/",
+			},
+			{
+				type: "gte",
+				key: "modified_date",
+				value: "1735689600000", // unix timestamp for 2025-01-01
+			},
+		],
+	},
+});
+```
+
+## Metadata attributes
+
+You can currently filter by the `folder` and `modified_date` of an R2 object. Currently, custom metadata attributes are not supported.
+
+### `folder`
+
+The directory to the object. For example, the `folder` of the object at `llama/logistics/llama-logistics.mdx` is `llama/logistics/`. Note that the `folder` does not include a leading `/`.
+
+Note that `folder` filter only includes files exactly on that folder, so files in subdirectories are not included. For example, specifying `folder: "llama/"` will match files in `llama/` but does not match files in `llama/logistics`.
+
+### `modified_date`
+
+The timestamp for when the object was last modified. The comparison is supported in Unix timestamp format. For example, `value: "1735689600000"` can be used to compare the date of 2025-01-01.
+
+## Filter schema
+
+You can create simple comparison filters or an array of comparison filters using a compound filter.
+
+### Comparison filter
+
+You can compare a metadata attribute (e.g. `folder` or `modified_date`) with a target value using a comparison filter.
+
+```js
+filters: {
+  type: "operator",
+  key: "metadata_attribute",
+  value: "target_value"
+}
+```
+
+The available operators for the comparison are:
+
+| Operator | Description               |
+| -------- | ------------------------- |
+| `eq`     | Equals                    |
+| `ne`     | Not equals                |
+| `gt`     | Greater than              |
+| `gte`    | Greater than or equals to |
+| `lt`     | Less than                 |
+| `lte`    | Less than or equals to    |
+
+### Compound filter
+
+You can use a compound filter to combine multiple comparison filters with a logical operator.
+
+```js
+filters: {
+  type: "compound_operator",
+  filters: [...]
+}
+```
+
+The available compound operators are: `and`, `or`.
+
+Note the following limitations with the compound operators:
+
+- No nesting combinations of `and`'s and `or`'s, meaning you can only pick 1 `and` or 1 `or`.
+- When using `or`:
+  - Only the `eq` operator is allowed.
+  - All conditions must filter on the **same key** (e.g. all on `folder`)
+
+## Response
+
+You can see the metadata attributes of your retrieved data in the response under the property `attributes` for each retrieved chunk. For example:
+
+```js
+"data": [
+  {
+    "file_id": "llama001",
+    "filename": "llama/logistics/llama-logistics.md",
+    "score": 0.45,
+    "attributes": {
+      "modified_date": 1735689600000,   // unix timestamp for 2025-01-01
+      "folder": "llama/logistics/",
+    },
+    "content": [
+      {
+        "id": "llama001",
+        "type": "text",
+        "text": "Llamas can carry 3 drinks max."
+      }
+    ]
+  }
+]
+```

src/content/docs/autorag/get-started.mdx

@@ -1,5 +1,5 @@
 ---
-title: Get started
+title: Getting started
 pcx_content_type: get-started
 sidebar:
   order: 2

src/content/docs/autorag/how-to/bring-your-own-generation-model.mdx

@@ -0,0 +1,83 @@
+---
+pcx_content_type: concept
+title: Bring your own generation model
+sidebar:
+  order: 5
+---
+
+import {
+	Badge,
+	Description,
+	Render,
+	TabItem,
+	Tabs,
+	WranglerConfig,
+	MetaInfo,
+	Type,
+} from "~/components";
+
+When using `AI Search`, AutoRAG leverages a Workers AI model to generate the response. If you want to use a model outside of Workers AI, you can use AutoRAG for search while leveraging a model outside of Workers AI to generate responses.
+
+Here is an example of how you can use an OpenAI model to generate your responses. This example is done using [Workers Binding](/autorag/usage/workers-binding/) but can be easily adapted to use the [REST API](/autorag/usage/rest-api/) instead.
+
+```ts
+import { openai } from "@ai-sdk/openai";
+import { generateText } from "ai";
+
+export interface Env {
+	AI: Ai;
+	OPENAI_API_KEY: string;
+}
+
+export default {
+	async fetch(request, env): Promise<Response> {
+		// Parse incoming url
+		const url = new URL(request.url);
+
+		// Get the user query or default to a predefined one
+		const userQuery =
+			url.searchParams.get("query") ??
+			"How do I train a llama to deliver coffee?";
+
+		// Search for documents in AutoRAG
+		const searchResult = await env.AI.autorag("my-rag").search({
+			query: userQuery,
+		});
+
+		if (searchResult.data.length === 0) {
+			// No matching documents
+			return Response.json({ text: `No data found for query "${userQuery}"` });
+		}
+
+		// Join all document chunks into a single string
+		const chunks = searchResult.data
+			.map((item) => {
+				const data = item.content
+					.map((content) => {
+						return content.text;
+					})
+					.join("\n\n");
+
+				return `<file name="${item.filename}">${data}</file>`;
+			})
+			.join("\n\n");
+
+		// Send the user query + matched documents to openai for answer
+		const generateResult = await generateText({
+			model: openai("gpt-4o-mini"),
+			messages: [
+				{
+					role: "system",
+					content:
+						"You are a helpful assistant and your task is to answer the user question using the provided files.",
+				},
+				{ role: "user", content: chunks },
+				{ role: "user", content: userQuery },
+			],
+		});
+
+		// Return the generated answer
+		return Response.json({ text: generateResult.text });
+	},
+} satisfies ExportedHandler<Env>;
+```

src/content/docs/autorag/how-to/create-simple-search-engine.mdx

@@ -0,0 +1,36 @@
+---
+pcx_content_type: concept
+title: Create a simple search engine
+sidebar:
+  order: 5
+---
+
+Using the `search` method you can implement a simple but fast search engine. This example is done using [Workers Binding](/autorag/usage/workers-binding/) but can be easily adapted to use the [REST API](/autorag/usage/rest-api/) instead.
+
+To replicate this example remember to:
+
+- Disable `rewrite_query` as you want to match the original user query
+- Configure your AutoRAG to have small chunk sizes, usually 256 tokens is enough
+
+```ts
+export interface Env {
+	AI: Ai;
+}
+
+export default {
+	async fetch(request, env): Promise<Response> {
+		const url = new URL(request.url);
+		const userQuery =
+			url.searchParams.get("query") ??
+			"How do I train a llama to deliver coffee?";
+		const searchResult = await env.AI.autorag("my-rag").search({
+			query: userQuery,
+			rewrite_query: false,
+		});
+
+		return Response.json({
+			files: searchResult.data.map((obj) => obj.filename),
+		});
+	},
+} satisfies ExportedHandler<Env>;
+```

src/content/docs/autorag/how-to/index.mdx

@@ -0,0 +1,12 @@
+---
+pcx_content_type: navigation
+title: How to
+sidebar:
+  order: 4
+  group:
+    hideIndex: true
+---
+
+import { DirectoryListing } from "~/components";
+
+<DirectoryListing />

src/content/docs/autorag/how-to/multitenancy.mdx

@@ -0,0 +1,41 @@
+---
+pcx_content_type: concept
+title: Create multitenancy
+sidebar:
+  order: 5
+---
+
+AutoRAG supports multitenancy by letting you segment content by tenant, so each user, customer, or workspace can only access their own data. This is typically done by organizing documents into per-tenant folders and applying [metadata filters](/autorag/configuration/metadata-filtering/) at query time.
+
+## 1. Organize Content by Tenant
+
+When uploading files to R2, structure your content by tenant using unique folder paths.
+
+Example folder structure:
+
+```bash
+customer-a/logs/
+customer-a/contracts/
+customer-b/contracts/
+```
+
+When indexing, AutoRAG will automatically store the folder path as metadata under the `folder` attribute. It’s recommended to enforce folder separation during upload or indexing to prevent accidental data access across tenants.
+
+## 2. Search Using Folder Filters
+
+To ensure a tenant only retrieves their own documents, apply a `folder` filter when performing a search.
+
+Example using [Workers Binding](/autorag/usage/workers-binding/):
+
+```js
+const response = await env.AI.autorag("my-autorag").search({
+	query: "When did I sign my agreement contract?",
+	filters: {
+		type: "eq",
+		key: "folder",
+		value: `customer-a/contracts/`,
+	},
+});
+```
+
+To filter across multiple folders, or to add date-based filtering, you can use a compound filter with an array of [comparison filters](/autorag/configuration/metadata-filtering/#compound-filter).

src/content/docs/autorag/index.mdx

@@ -55,6 +55,12 @@ Automatically and continuously index your data source, keeping your content fres
 
 </Feature>
 
+<Feature header="Multitenancy support" href="/autorag/how-to/multitenancy/" cta="Add filters">
+
+Create multitenancy by scoping search to each tenant’s data using folder-based metadata filters.
+
+</Feature>
+
 <Feature header="Workers Binding" href="/autorag/usage/workers-binding/" cta="Add to Worker">
 
 Call your AutoRAG instance for search or AI Search directly from a Cloudflare Worker using the native binding integration.