Skip to content

Commit 822f292

Browse files
committed
starts with filter instructions added
1 parent 5ac8f5e commit 822f292

File tree

2 files changed

+73
-2
lines changed

2 files changed

+73
-2
lines changed

src/content/docs/autorag/configuration/metadata-filtering.mdx

Lines changed: 42 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,13 +34,13 @@ const answer = await env.AI.autorag("my-autorag").search({
3434

3535
You can currently filter by the `folder` and `timestamp` of an R2 object. Currently, custom metadata attributes are not supported.
3636

37-
### `folder`
37+
### Folder
3838

3939
The directory to the object. For example, the `folder` of the object at `llama/logistics/llama-logistics.mdx` is `llama/logistics/`. Note that the `folder` does not include a leading `/`.
4040

4141
Note that `folder` filter only includes files exactly in that folder, so files in subdirectories are not included. For example, specifying `folder: "llama/"` will match files in `llama/` but does not match files in `llama/logistics`.
4242

43-
### `timestamp`
43+
### Timestamp
4444

4545
The timestamp indicating when the object was last modified. Comparisons are supported using a 13-digit Unix timestamp (milliseconds), but values will be rounded to 10 digits (seconds). For example, `1735689600999` or `2025-01-01 00:00:00.999 UTC` will be rounded down to `1735689600000`, corresponding to `2025-01-01 00:00:00 UTC`.
4646

@@ -91,6 +91,46 @@ Note the following limitations with the compound operators:
9191
- Only the `eq` operator is allowed.
9292
- All conditions must filter on the **same key** (for example, all on `folder`)
9393

94+
### "Starts with" filter for folders
95+
96+
You can use "starts with" filtering on the `folder` metadata attribute to search for all files and subfolders within a specific path.
97+
98+
For example, consider this file structure:
99+
```
100+
customer-a/profile.md
101+
customer-a/contracts/property/contract-1.pdf
102+
```
103+
104+
If you were to filter using an `eq` (equals) operator with `value: "customer-a/"`, it would only match files directly within that folder, like `profile.md`. It wouldn't include files in subfolders like `customer-a/contracts/`.
105+
106+
To recursively filter for all items starting with the path `customer-a/`, you can use the following compound filter:
107+
108+
```js
109+
filters: {
110+
type: "and",
111+
filters: [
112+
{
113+
type: "gt",
114+
key: "folder",
115+
value: "customer-a//",
116+
},
117+
{
118+
type: "lte",
119+
key: "folder",
120+
value: "customer-a/z",
121+
},
122+
],
123+
},
124+
```
125+
126+
This filter identifies paths starting with `customer-a/` by using:
127+
128+
- The `and` condition to combine the effects of the `gt` and `lte` conditions.
129+
- The `gt` condition to include pathes greater than the `/` ASCII character.
130+
- The `lte` condition to include pathes less than and including the lower case `z` ASCII character.
131+
132+
Together, these conditions effectively select paths that begin with the provided path value.
133+
94134
## Response
95135

96136
You can see the metadata attributes of your retrieved data in the response under the property `attributes` for each retrieved chunk. For example:

src/content/docs/autorag/how-to/multitenancy.mdx

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,3 +39,34 @@ const response = await env.AI.autorag("my-autorag").search({
3939
```
4040

4141
To filter across multiple folders, or to add date-based filtering, you can use a compound filter with an array of [comparison filters](/autorag/configuration/metadata-filtering/#compound-filter).
42+
43+
## Tip: Use "Starts with" filter
44+
45+
While an `eq` filter targets files at the specific folder, you'll often want to retrieve all documents belonging to a tenant regardless if there are files in its subfolders. For example, all files in `customer-a/` with a structure like:
46+
47+
```
48+
customer-a/profile.md
49+
customer-a/contracts/property/contract-1.pdf
50+
```
51+
52+
To achieve this [starts with](/autorag/configuration/metadata-filtering/#starts-with-filter-for-folders) behavior, use a compound filter like:
53+
54+
```js
55+
filters: {
56+
type: "and",
57+
filters: [
58+
{
59+
type: "gt",
60+
key: "folder",
61+
value: "customer-a//",
62+
},
63+
{
64+
type: "lte",
65+
key: "folder",
66+
value: "customer-a/z",
67+
},
68+
],
69+
},
70+
```
71+
72+
With this filter you would capture both files `profile.md` and `contract-1.pdf`.

0 commit comments

Comments
 (0)