Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 48 additions & 2 deletions src/content/docs/autorag/configuration/metadata-filtering.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ sidebar:
order: 6
---

import { FileTree } from "~/components"

Metadata filtering narrows down search results based on metadata, so only relevant content is retrieved. The filter narrows down results prior to retrieval, so that you only query the scope of documents that matter.

Here is an example of metadata filtering using [Workers Binding](/autorag/usage/workers-binding/) but it can be easily adapted to use the [REST API](/autorag/usage/rest-api/) instead.
Expand Down Expand Up @@ -34,13 +36,13 @@ const answer = await env.AI.autorag("my-autorag").search({

You can currently filter by the `folder` and `timestamp` of an R2 object. Currently, custom metadata attributes are not supported.

### `folder`
### Folder

The directory to the object. For example, the `folder` of the object at `llama/logistics/llama-logistics.mdx` is `llama/logistics/`. Note that the `folder` does not include a leading `/`.

Note that `folder` filter only includes files exactly in that folder, so files in subdirectories are not included. For example, specifying `folder: "llama/"` will match files in `llama/` but does not match files in `llama/logistics`.

### `timestamp`
### Timestamp

The timestamp indicating when the object was last modified. Comparisons are supported using a 13-digit Unix timestamp (milliseconds), but values will be rounded to 10 digits (seconds). For example, `1735689600999` or `2025-01-01 00:00:00.999 UTC` will be rounded down to `1735689600000`, corresponding to `2025-01-01 00:00:00 UTC`.

Expand Down Expand Up @@ -91,6 +93,50 @@ Note the following limitations with the compound operators:
- Only the `eq` operator is allowed.
- All conditions must filter on the **same key** (for example, all on `folder`)

### "Starts with" filter for folders

You can use "starts with" filtering on the `folder` metadata attribute to search for all files and subfolders within a specific path.

For example, consider this file structure:

<FileTree>
- customer-a
- profile.md
- contracts
- property
- contract-1.pdf
</FileTree>

If you were to filter using an `eq` (equals) operator with `value: "customer-a/"`, it would only match files directly within that folder, like `profile.md`. It would not include files in subfolders like `customer-a/contracts/`.

To recursively filter for all items starting with the path `customer-a/`, you can use the following compound filter:

```js
filters: {
type: "and",
filters: [
{
type: "gt",
key: "folder",
value: "customer-a//",
},
{
type: "lte",
key: "folder",
value: "customer-a/z",
},
],
},
```

This filter identifies paths starting with `customer-a/` by using:

- The `and` condition to combine the effects of the `gt` and `lte` conditions.
- The `gt` condition to include paths greater than the `/` ASCII character.
- The `lte` condition to include paths less than and including the lower case `z` ASCII character.

Together, these conditions effectively select paths that begin with the provided path value.

## Response

You can see the metadata attributes of your retrieved data in the response under the property `attributes` for each retrieved chunk. For example:
Expand Down
54 changes: 49 additions & 5 deletions src/content/docs/autorag/how-to/multitenancy.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ sidebar:
order: 5
---

import { FileTree } from "~/components"

AutoRAG supports multitenancy by letting you segment content by tenant, so each user, customer, or workspace can only access their own data. This is typically done by organizing documents into per-tenant folders and applying [metadata filters](/autorag/configuration/metadata-filtering/) at query time.

## 1. Organize Content by Tenant
Expand All @@ -13,11 +15,13 @@ When uploading files to R2, structure your content by tenant using unique folder

Example folder structure:

```bash
customer-a/logs/
customer-a/contracts/
customer-b/contracts/
```
<FileTree>
- customer-a
- logs/
- contracts/
- customer-b
- contracts/
</FileTree>

When indexing, AutoRAG will automatically store the folder path as metadata under the `folder` attribute. It is recommended to enforce folder separation during upload or indexing to prevent accidental data access across tenants.

Expand All @@ -39,3 +43,43 @@ const response = await env.AI.autorag("my-autorag").search({
```

To filter across multiple folders, or to add date-based filtering, you can use a compound filter with an array of [comparison filters](/autorag/configuration/metadata-filtering/#compound-filter).

## Tip: Use "Starts with" filter

While an `eq` filter targets files at the specific folder, you'll often want to retrieve all documents belonging to a tenant regardless if there are files in its subfolders. For example, all files in `customer-a/` with a structure like:

<FileTree>
- customer-a
- profile.md
- contracts
- property
- contract-1.pdf
</FileTree>

To achieve this [starts with](/autorag/configuration/metadata-filtering/#starts-with-filter-for-folders) behavior, use a compound filter like:

```js
filters: {
type: "and",
filters: [
{
type: "gt",
key: "folder",
value: "customer-a//",
},
{
type: "lte",
key: "folder",
value: "customer-a/z",
},
],
},
```

This filter identifies paths starting with `customer-a/` by using:

- The `and` condition to combine the effects of the `gt` and `lte` conditions.
- The `gt` condition to include paths greater than the `/` ASCII character.
- The `lte` condition to include paths less than and including the lower case `z` ASCII character.

This filter captures both files `profile.md` and `contract-1.pdf`.
Loading