metadata filtering support (#21870)

* metadata filtering support

* changelog

* url update

* redirect

* add link to autorag

* Update public/__redirects

Co-authored-by: Jun Lee <[email protected]>

* PCX Review

* update changelog date and content

* update cta

* ms timestamp

---------

Co-authored-by: Jun Lee <[email protected]>

Author: aninibread

public/__redirects

@@ -183,6 +183,9 @@
 # api-shield
 /api-shield/security/sequential-abuse-detection/ /api-shield/security/sequence-analytics/ 301
 
+#autorag
+/autorag/usage/recipes/ /autorag/how-to/ 301
+
 # bots
 /bots/about/plans/ /bots/plans/ 301
 /bots/about/plans/biz-and-ent/ /bots/plans/biz-and-ent/ 301

src/content/changelog/autorag/2025-04-23-autorag-metadata-filtering.mdx

@@ -0,0 +1,36 @@
+---
+title: Metadata filtering and multitenancy support in AutoRAG
+description: Add metadata filters to AutoRAG queries to enable multitenancy and control the scope of retrieved results.
+products:
+  - autorag
+date: 2025-04-23T6:00:00Z
+---
+
+You can now filter [AutoRAG](/autorag) search results by `folder` and `modified_date` using [metadata filtering](/autorag/configuration/metadata-filtering/) to narrow down the scope of your query.
+
+This makes it easy to build [multitenant experiences](/autorag/how-to/multitenancy/) where each user can only access their own data. By organizing your content into per-tenant folders and applying a `folder` filter at query time, you ensure that each tenant retrieves only their own documents.
+
+**Example folder structure:**
+
+```bash
+customer-a/logs/
+customer-a/contracts/
+customer-b/contracts/
+```
+
+**Example query:**
+
+```js
+const response = await env.AI.autorag("my-autorag").search({
+	query: "When did I sign my agreement contract?",
+	filters: {
+		type: "eq",
+		key: "folder",
+		value: "customer-a/contracts/",
+	},
+});
+```
+
+You can use metadata filtering by creating a new AutoRAG or reindexing existing data. To reindex all content in an existing AutoRAG, update any chunking setting and select **Sync index**. Metadata filtering is available for all data indexed on or after **April 21, 2025**.
+
+If you are new to AutoRAG, get started with the [Get started AutoRAG guide](/autorag/get-started/).

src/content/docs/autorag/configuration/metadata-filtering.mdx

@@ -0,0 +1,117 @@
+---
+pcx_content_type: concept
+title: Metadata filtering
+sidebar:
+  order: 6
+---
+
+Metadata filtering narrows down search results based on metadata, so only relevant content is retrieved. The filter narrows down results prior to retrieval, so that you only query the scope of documents that matter.
+
+Here is an example of metadata filtering using [Workers Binding](/autorag/usage/workers-binding/) but it can be easily adapted to use the [REST API](/autorag/usage/rest-api/) instead.
+
+```js
+const answer = await env.AI.autorag("my-autorag").search({
+	query: "How do I train a llama to deliver coffee?",
+	filters: {
+		type: "and",
+		filters: [
+			{
+				type: "eq",
+				key: "folder",
+				value: "llama/logistics/",
+			},
+			{
+				type: "gte",
+				key: "modified_date",
+				value: "1735689600000", // unix timestamp for 2025-01-01
+			},
+		],
+	},
+});
+```
+
+## Metadata attributes
+
+You can currently filter by the `folder` and `modified_date` of an R2 object. Currently, custom metadata attributes are not supported.
+
+### `folder`
+
+The directory to the object. For example, the `folder` of the object at `llama/logistics/llama-logistics.mdx` is `llama/logistics/`. Note that the `folder` does not include a leading `/`.
+
+Note that `folder` filter only includes files exactly in that folder, so files in subdirectories are not included. For example, specifying `folder: "llama/"` will match files in `llama/` but does not match files in `llama/logistics`.
+
+### `modified_date`
+
+The timestamp indicating when the object was last modified. Comparisons are supported using a 13-digit Unix timestamp (milliseconds), but values will be rounded to 10 digits (seconds). For example, `1735689600999` or `2025-01-01 00:00:00.999 UTC` will be rounded down to `1735689600000`, corresponding to `2025-01-01 00:00:00 UTC`.
+
+## Filter schema
+
+You can create simple comparison filters or an array of comparison filters using a compound filter.
+
+### Comparison filter
+
+You can compare a metadata attribute (for example, `folder` or `modified_date`) with a target value using a comparison filter.
+
+```js
+filters: {
+  type: "operator",
+  key: "metadata_attribute",
+  value: "target_value"
+}
+```
+
+The available operators for the comparison are:
+
+| Operator | Description               |
+| -------- | ------------------------- |
+| `eq`     | Equals                    |
+| `ne`     | Not equals                |
+| `gt`     | Greater than              |
+| `gte`    | Greater than or equals to |
+| `lt`     | Less than                 |
+| `lte`    | Less than or equals to    |
+
+### Compound filter
+
+You can use a compound filter to combine multiple comparison filters with a logical operator.
+
+```js
+filters: {
+  type: "compound_operator",
+  filters: [...]
+}
+```
+
+The available compound operators are: `and`, `or`.
+
+Note the following limitations with the compound operators:
+
+- No nesting combinations of `and`'s and `or`'s, meaning you can only pick 1 `and` or 1 `or`.
+- When using `or`:
+  - Only the `eq` operator is allowed.
+  - All conditions must filter on the **same key** (for example, all on `folder`)
+
+## Response
+
+You can see the metadata attributes of your retrieved data in the response under the property `attributes` for each retrieved chunk. For example:
+
+```js
+"data": [
+  {
+    "file_id": "llama001",
+    "filename": "llama/logistics/llama-logistics.md",
+    "score": 0.45,
+    "attributes": {
+      "modified_date": 1735689600000,   // unix timestamp for 2025-01-01
+      "folder": "llama/logistics/",
+    },
+    "content": [
+      {
+        "id": "llama001",
+        "type": "text",
+        "text": "Llamas can carry 3 drinks max."
+      }
+    ]
+  }
+]
+```

src/content/docs/autorag/get-started.mdx

@@ -1,5 +1,5 @@
 ---
-title: Get started
+title: Getting started
 pcx_content_type: get-started
 sidebar:
   order: 2

src/content/docs/autorag/how-to/bring-your-own-generation-model.mdx

@@ -0,0 +1,83 @@
+---
+pcx_content_type: concept
+title: Bring your own generation model
+sidebar:
+  order: 5
+---
+
+import {
+	Badge,
+	Description,
+	Render,
+	TabItem,
+	Tabs,
+	WranglerConfig,
+	MetaInfo,
+	Type,
+} from "~/components";
+
+When using `AI Search`, AutoRAG leverages a Workers AI model to generate the response. If you want to use a model outside of Workers AI, you can use AutoRAG for search while leveraging a model outside of Workers AI to generate responses.
+
+Here is an example of how you can use an OpenAI model to generate your responses. This example uses [Workers Binding](/autorag/usage/workers-binding/), but can be easily adapted to use the [REST API](/autorag/usage/rest-api/) instead.
+
+```ts
+import { openai } from "@ai-sdk/openai";
+import { generateText } from "ai";
+
+export interface Env {
+	AI: Ai;
+	OPENAI_API_KEY: string;
+}
+
+export default {
+	async fetch(request, env): Promise<Response> {
+		// Parse incoming url
+		const url = new URL(request.url);
+
+		// Get the user query or default to a predefined one
+		const userQuery =
+			url.searchParams.get("query") ??
+			"How do I train a llama to deliver coffee?";
+
+		// Search for documents in AutoRAG
+		const searchResult = await env.AI.autorag("my-rag").search({
+			query: userQuery,
+		});
+
+		if (searchResult.data.length === 0) {
+			// No matching documents
+			return Response.json({ text: `No data found for query "${userQuery}"` });
+		}
+
+		// Join all document chunks into a single string
+		const chunks = searchResult.data
+			.map((item) => {
+				const data = item.content
+					.map((content) => {
+						return content.text;
+					})
+					.join("\n\n");
+
+				return `<file name="${item.filename}">${data}</file>`;
+			})
+			.join("\n\n");
+
+		// Send the user query + matched documents to openai for answer
+		const generateResult = await generateText({
+			model: openai("gpt-4o-mini"),
+			messages: [
+				{
+					role: "system",
+					content:
+						"You are a helpful assistant and your task is to answer the user question using the provided files.",
+				},
+				{ role: "user", content: chunks },
+				{ role: "user", content: userQuery },
+			],
+		});
+
+		// Return the generated answer
+		return Response.json({ text: generateResult.text });
+	},
+} satisfies ExportedHandler<Env>;
+```

src/content/docs/autorag/how-to/index.mdx

@@ -0,0 +1,12 @@
+---
+pcx_content_type: navigation
+title: How to
+sidebar:
+  order: 4
+  group:
+    hideIndex: true
+---
+
+import { DirectoryListing } from "~/components";
+
+<DirectoryListing />

src/content/docs/autorag/how-to/multitenancy.mdx

@@ -0,0 +1,41 @@
+---
+pcx_content_type: concept
+title: Create multitenancy
+sidebar:
+  order: 5
+---
+
+AutoRAG supports multitenancy by letting you segment content by tenant, so each user, customer, or workspace can only access their own data. This is typically done by organizing documents into per-tenant folders and applying [metadata filters](/autorag/configuration/metadata-filtering/) at query time.
+
+## 1. Organize Content by Tenant
+
+When uploading files to R2, structure your content by tenant using unique folder paths.
+
+Example folder structure:
+
+```bash
+customer-a/logs/
+customer-a/contracts/
+customer-b/contracts/
+```
+
+When indexing, AutoRAG will automatically store the folder path as metadata under the `folder` attribute. It is recommended to enforce folder separation during upload or indexing to prevent accidental data access across tenants.
+
+## 2. Search Using Folder Filters
+
+To ensure a tenant only retrieves their own documents, apply a `folder` filter when performing a search.
+
+Example using [Workers Binding](/autorag/usage/workers-binding/):
+
+```js
+const response = await env.AI.autorag("my-autorag").search({
+	query: "When did I sign my agreement contract?",
+	filters: {
+		type: "eq",
+		key: "folder",
+		value: `customer-a/contracts/`,
+	},
+});
+```
+
+To filter across multiple folders, or to add date-based filtering, you can use a compound filter with an array of [comparison filters](/autorag/configuration/metadata-filtering/#compound-filter).

src/content/docs/autorag/how-to/simple-search-engine.mdx

@@ -0,0 +1,36 @@
+---
+pcx_content_type: concept
+title: Create a simple search engine
+sidebar:
+  order: 5
+---
+
+By using the `search` method, you can implement a simple but fast search engine. This example uses [Workers Binding](/autorag/usage/workers-binding/), but can be easily adapted to use the [REST API](/autorag/usage/rest-api/) instead.
+
+To replicate this example remember to:
+
+- Disable `rewrite_query`, as you want to match the original user query
+- Configure your AutoRAG to have small chunk sizes, usually 256 tokens is enough
+
+```ts
+export interface Env {
+	AI: Ai;
+}
+
+export default {
+	async fetch(request, env): Promise<Response> {
+		const url = new URL(request.url);
+		const userQuery =
+			url.searchParams.get("query") ??
+			"How do I train a llama to deliver coffee?";
+		const searchResult = await env.AI.autorag("my-rag").search({
+			query: userQuery,
+			rewrite_query: false,
+		});
+
+		return Response.json({
+			files: searchResult.data.map((obj) => obj.filename),
+		});
+	},
+} satisfies ExportedHandler<Env>;
+```

src/content/docs/autorag/index.mdx

@@ -55,6 +55,12 @@ Automatically and continuously index your data source, keeping your content fres
 
 </Feature>
 
+<Feature header="Multitenancy support" href="/autorag/how-to/multitenancy/" cta="Add filters">
+
+Create multitenancy by scoping search to each tenant’s data using folder-based metadata filters.
+
+</Feature>
+
 <Feature header="Workers Binding" href="/autorag/usage/workers-binding/" cta="Add to Worker">
 
 Call your AutoRAG instance for search or AI Search directly from a Cloudflare Worker using the native binding integration.