You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
title: View custom metadata in responses and guide AI-search with context in AutoRAG
3
+
description: You can now view custom metadata in AutoRAG search responses and use a context field to provide additional guidance to AI-generated answers.
4
+
products:
5
+
- autorag
6
+
date: 2025-06-16T6:10:00Z
7
+
---
8
+
9
+
In [AutoRAG](/autorag/), you can now view your object's custom metadata in the response from [`/search`](/autorag/usage/workers-binding/) and [`/ai-search`](/autorag/usage/workers-binding/), and optionally add a `context` field in the custom metadata of an object to provide additional guidance for AI-generated answers.
10
+
11
+
You can add [custom metadata](/r2/api/workers/workers-api-reference/#r2putoptions) to an object when uploading it to your R2 bucket.
12
+
13
+
# Object's custom metadata in search responses
14
+
15
+
When you run a search, AutoRAG now returns any custom metadata associated with the object. This metadata appears in the response under `file`, inside `attributes`, and can be used for downstream processing.
16
+
17
+
For example, the `attributes` section of your search response may look like:
"context": "A checklist for internal launch readiness, including legal, engineering, and marketing steps."
28
+
}
29
+
}
30
+
}
31
+
```
32
+
33
+
# Add a `context` field to guide LLM answers
34
+
35
+
When you include a custom metadata field named `context`, AutoRAG attaches that value to each chunk of the file. When you run an `/ai-search` query, this `context` is passed to the LLM and can be used as additional input when generating an answer.
36
+
37
+
We recommend using the `context` field to describe supplemental information you want the LLM to consider, such as a summary of the document or a source URL.
38
+
39
+
For example:
40
+
41
+
```json
42
+
context: "summary: 'Checklist for internal product launch readiness, including legal, engineering, and marketing steps.'; url: 'https://wiki.company.com/docs/product-launch-checklist'"
43
+
```
44
+
45
+
This gives you more control over how your content is interpreted, without requiring you to modify the original contents of the file.
46
+
47
+
Learn more in AutoRAG's [metadata filtering documentation](/autorag/configuration/metadata-filtering/).
description: You can now filter AutoRAG search queries by file name, allowing you to control which files can be retrieved for a given query.
4
+
products:
5
+
- autorag
6
+
date: 2025-06-16T6:00:00Z
7
+
---
8
+
9
+
In [AutoRAG](/autorag/), you can now [filter](/autorag/configuration/metadata-filtering/) by an object's file name using the `filename` attribute, giving you more control over which files are searched for a given query.
10
+
11
+
This is useful when your application has already determined which files should be searched. For example, you might query a PostgreSQL database to get a list of files a user has access to based on their permissions, and then use that list to limit what AutoRAG retrieves.
This allows you to connect your application logic with AutoRAG's retrieval process, making it easy to control what gets searched without needing to reindex or modify your data.
27
+
28
+
Learn more in AutoRAG's [metadata filtering documentation](/autorag/configuration/metadata-filtering/).
Copy file name to clipboardExpand all lines: src/content/docs/autorag/configuration/metadata.mdx
+35-17Lines changed: 35 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,12 +1,13 @@
1
1
---
2
2
pcx_content_type: concept
3
-
title: Metadata filtering
3
+
title: Metadata
4
4
sidebar:
5
5
order: 6
6
6
---
7
7
8
8
import { FileTree } from"~/components"
9
9
10
+
## Metadata filtering
10
11
Metadata filtering narrows down search results based on metadata, so only relevant content is retrieved. The filter narrows down results prior to retrieval, so that you only query the scope of documents that matter.
11
12
12
13
Here is an example of metadata filtering using [Workers Binding](/autorag/usage/workers-binding/) but it can be easily adapted to use the [REST API](/autorag/usage/rest-api/) instead.
You can currently filter by the `folder` and `timestamp` of an R2 object. Currently, custom metadata attributes are not supported.
38
+
| Attribute | Description | Example |
39
+
| --- | --- | --- |
40
+
|`filename`| The name of the file. |`dog.png` or `animals/mammals/cat.png`|
41
+
|`folder`| The prefix or directory to the object. | For the object `animals/mammals/cat.png`, the folder is `animals/mammals/`|
42
+
|`timestamp`| The timestamp for when the object was last modified. Comparisons are supported using a 13-digit Unix timestamp (milliseconds), but values will be rounded to 10 digits (seconds). | The timestamp `2025-01-01 00:00:00.999 UTC` is `1735689600999` and it will be rounded to `1735689600000`, corresponding to `2025-01-01 00:00:00 UTC`|
38
43
39
-
### Folder
40
-
41
-
The directory to the object. For example, the `folder` of the object at `llama/logistics/llama-logistics.mdx` is `llama/logistics/`. Note that the `folder` does not include a leading `/`.
42
-
43
-
Note that `folder` filter only includes files exactly in that folder, so files in subdirectories are not included. For example, specifying `folder: "llama/"` will match files in `llama/` but does not match files in `llama/logistics`.
44
-
45
-
### Timestamp
46
-
47
-
The timestamp indicating when the object was last modified. Comparisons are supported using a 13-digit Unix timestamp (milliseconds), but values will be rounded to 10 digits (seconds). For example, `1735689600999` or `2025-01-01 00:00:00.999 UTC` will be rounded down to `1735689600000`, corresponding to `2025-01-01 00:00:00 UTC`.
48
-
49
-
## Filter schema
44
+
### Filter schema
50
45
51
46
You can create simple comparison filters or an array of comparison filters using a compound filter.
52
47
53
-
### Comparison filter
48
+
####Comparison filter
54
49
55
50
You can compare a metadata attribute (for example, `folder` or `timestamp`) with a target value using a comparison filter.
56
51
@@ -73,7 +68,7 @@ The available operators for the comparison are:
73
68
|`lt`| Less than |
74
69
|`lte`| Less than or equals to |
75
70
76
-
### Compound filter
71
+
####Compound filter
77
72
78
73
You can use a compound filter to combine multiple comparison filters with a logical operator.
79
74
@@ -93,7 +88,7 @@ Note the following limitations with the compound operators:
93
88
- Only the `eq` operator is allowed.
94
89
- All conditions must filter on the **same key** (for example, all on `folder`)
95
90
96
-
### "Starts with" filter for folders
91
+
####"Starts with" filter for folders
97
92
98
93
You can use "starts with" filtering on the `folder` metadata attribute to search for all files and subfolders within a specific path.
99
94
@@ -137,6 +132,25 @@ This filter identifies paths starting with `customer-a/` by using:
137
132
138
133
Together, these conditions effectively select paths that begin with the provided path value.
139
134
135
+
## Add a `context` field to guide AI Search
136
+
You can optionally include a custom metadata field named `context` when uploading an object to your R2 bucket.
137
+
138
+
The `context` field is attached to each chunk and passed to the LLM during an `/ai-search` query. It does not affect retrieval but helps the LLM interpret and frame the answer.
139
+
140
+
The field can be used for providing document summaries, source links, or custom instructions without modifying the file content.
141
+
142
+
You can add [custom metadata](/r2/api/workers/workers-api-reference/#r2putoptions) to an object in the `/PUT` operation when uploading the object to your R2 bucket. For example if you are using the [Workers binding with R2](/r2/api/workers/workers-api-usage/):
143
+
144
+
```javascript
145
+
awaitenv.MY_BUCKET.put("cat.png", file, {
146
+
customMetadata: {
147
+
context:"This is a picture of Joe's cat. His name is Max."
148
+
}
149
+
});
150
+
```
151
+
152
+
During `/ai-search`, this context appears in the response under `attributes.file.context`, and is included in the data passed to the LLM for generating a response.
153
+
140
154
## Response
141
155
142
156
You can see the metadata attributes of your retrieved data in the response under the property `attributes` for each retrieved chunk. For example:
@@ -150,6 +164,10 @@ You can see the metadata attributes of your retrieved data in the response under
150
164
"attributes": {
151
165
"timestamp":1735689600000, // unix timestamp for 2025-01-01
152
166
"folder":"llama/logistics/",
167
+
"file": {
168
+
"url":"www.llamasarethebest.com/logistics"
169
+
"context":"This file contains information about how llamas can logistically deliver coffee."
title: Rich format file size limit increased to 4 MB
10
+
description: |-
11
+
You can now index rich format files (e.g., PDF) up to 4 MB in size, up from the previous 1 MB limit.
12
+
- publish_date: "2025-06-12"
13
+
title: Index processing status
14
+
description: |-
15
+
The dashboard now includes a new “Processing” step for the indexing pipeline that displays the files currently being processed.
16
+
- publish_date: "2025-06-12"
17
+
title: Sync AutoRAG REST API published
18
+
description: |-
19
+
You can now trigger a sync job for an AutoRAG using the [Sync REST API](/api/resources/autorag/subresources/rags/methods/sync/). This scans your data source for changes and queues updated or previously errored files for indexing.
20
+
- publish_date: "2025-06-10"
21
+
title: Files modified in the data source will now be updated
22
+
description: |-
23
+
Files modified in your source R2 bucket will now be updated in the AutoRAG index during the next sync. For example, if you upload a new version of an existing file, the changes will be reflected in the index after the subsequent sync job.
24
+
Please note that deleted files are not yet removed from the index. We are actively working on this functionality.
8
25
- publish_date: "2025-05-31"
9
26
title: Errored files will now be retried in next sync
10
27
description: |-
11
-
Files that fail to index will now be automatically retried in the next indexing job. For instance, if a file initially failed because it was oversized but was then corrected (e.g. replaced with a file of the same name/key within the size limit), it will be re-attempted during the next scheduled sync.
28
+
Files that failed to index will now be automatically retried in the next indexing job. For instance, if a file initially failed because it was oversized but was then corrected (e.g. replaced with a file of the same name/key within the size limit), it will be re-attempted during the next scheduled sync.
12
29
- publish_date: "2025-05-31"
13
30
title: Fixed character cutoff in recursive chunking
0 commit comments