You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/search/cognitive-search-incremental-indexing-conceptual.md
+22-16Lines changed: 22 additions & 16 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,13 +8,13 @@ ms.service: cognitive-search
8
8
ms.custom:
9
9
- ignite-2023
10
10
ms.topic: conceptual
11
-
ms.date: 06/18/2024
11
+
ms.date: 06/25/2024
12
12
---
13
13
14
14
# Incremental enrichment and caching in Azure AI Search
15
15
16
16
> [!IMPORTANT]
17
-
> This feature is in public preview under [supplemental terms of use](https://azure.microsoft.com/support/legal/preview-supplemental-terms/). The [preview REST API](/rest/api/searchservice/index-preview) supports this feature.
17
+
> This feature is in public preview under [supplemental terms of use](https://azure.microsoft.com/support/legal/preview-supplemental-terms/). The [preview REST API](/rest/api/searchservice/search-service-api-versions#preview-versions) supports this feature.
18
18
19
19
*Incremental enrichment* refers to the use of cached enrichments during [skillset execution](cognitive-search-working-with-skillsets.md) so that only new and changed skills and documents incur pay-as-you-go processing charges for API calls to Azure AI services. The cache contains the output from [document cracking](search-indexer-overview.md#document-cracking), plus the outputs of each skill for every document. Although caching is billable (it uses Azure Storage), the overall cost of enrichment is reduced because the costs of storage are less than image extraction and AI processing.
20
20
@@ -35,10 +35,12 @@ Physically, the cache is stored in a blob container in your Azure Storage accoun
35
35
36
36
The cache is created when you specify the "cache" property and run the indexer. Only enriched content can be cached. If your indexer doesn't have an attached skillset, then caching doesn't apply.
37
37
38
-
The following example illustrates an indexer with caching enabled. See [Enable enrichment caching](search-howto-incremental-index.md) for full instructions. Notice that when adding the cache property, use a [preview API version](/rest/api/searchservice/search-service-api-versions#preview-versions), 2020-06-30-Preview or later, on the request.
38
+
The following example illustrates an indexer with caching enabled. See [Enable enrichment caching](search-howto-incremental-index.md) for full instructions.
39
+
40
+
To use the cache property, you can use 2020-06-30-preview or later when you [create or update an indexer](/rest/api/searchservice/indexers/create-or-update?view=rest-searchservice-2024-05-01-preview&preserve-view=true). We recommend the latest preview API.
39
41
40
42
```json
41
-
POST https://[search service name].search.windows.net/indexers?api-version=2020-06-30-Preview
43
+
POST https://[search service name].search.windows.net/indexers?api-version=2024-05-01-rreview
42
44
{
43
45
"name": "myIndexerName",
44
46
"targetIndexName": "myIndex",
@@ -85,21 +87,21 @@ If you know that a change to the skill is indeed superficial, you should overrid
85
87
1. Append the "disableCacheReprocessingChangeDetection=true" parameter on the request.
86
88
1. Submit the change.
87
89
88
-
Setting this parameter ensures that only updates to the skillset definition are committed and the change isn't evaluated for effects on the existing cache. Use a preview API version, 2020-06-30-Preview or later.
90
+
Setting this parameter ensures that only updates to the skillset definition are committed and the change isn't evaluated for effects on the existing cache. Use a preview API version, 2020-06-30-Preview or later. We recommend the latest preview API.
89
91
90
92
```http
91
-
PUT https://[servicename].search.windows.net/skillsets/[skillset name]?api-version=2020-06-30-Preview&disableCacheReprocessingChangeDetection
93
+
PUT https://[servicename].search.windows.net/skillsets/[skillset name]?api-version=2024-05-01-preview&disableCacheReprocessingChangeDetection
92
94
93
95
```
94
96
95
97
<aname="Bypass-data-source-check"></a>
96
98
97
99
### Bypass data source validation checks
98
100
99
-
Most changes to a data source definition will invalidate the cache. However, for scenarios where you know that a change shouldn't invalidate the cache - such as changing a connection string or rotating the key on the storage account - append the `ignoreResetRequirement` parameter on the [data source update](/rest/api/searchservice/update-data-source). Setting this parameter to true allows the commit to go through, without triggering a reset condition that would result in all objects being rebuilt and populated from scratch.
101
+
Most changes to a data source definition will invalidate the cache. However, for scenarios where you know that a change shouldn't invalidate the cache - such as changing a connection string or rotating the key on the storage account - append the `ignoreResetRequirement` parameter on the [data source update](/rest/api/searchservice/data-sources/create-or-update). Setting this parameter to true allows the commit to go through, without triggering a reset condition that would result in all objects being rebuilt and populated from scratch.
100
102
101
103
```http
102
-
PUT https://[search service].search.windows.net/datasources/[data source name]?api-version=2020-06-30-Preview&ignoreResetRequirement
104
+
PUT https://[search service].search.windows.net/datasources/[data source name]?api-version=2024-05-01-preview&ignoreResetRequirement
103
105
104
106
```
105
107
@@ -109,11 +111,13 @@ PUT https://[search service].search.windows.net/datasources/[data source name]?a
109
111
110
112
The purpose of the cache is to avoid unnecessary processing, but suppose you make a change to a skill that the indexer doesn't detect (for example, changing something in external code, such as a custom skill).
111
113
112
-
In this case, you can use the [Reset Skills](/rest/api/searchservice/preview-api/reset-skills) to force reprocessing of a particular skill, including any downstream skills that have a dependency on that skill's output. This API accepts a POST request with a list of skills that should be invalidated and marked for reprocessing. After Reset Skills, follow with a [Run Indexer](/rest/api/searchservice/run-indexer) request to invoke the pipeline processing.
114
+
In this case, you can use the [Reset Skills](/rest/api/searchservice/preview-api/reset-skills) to force reprocessing of a particular skill, including any downstream skills that have a dependency on that skill's output. This API accepts a POST request with a list of skills that should be invalidated and marked for reprocessing. After Reset Skills, follow with a [Run Indexer](/rest/api/searchservice/indexers/run) request to invoke the pipeline processing.
113
115
114
116
## Re-cache specific documents
115
117
116
-
[Resetting an indexer](/rest/api/searchservice/reset-indexer) will result in all documents in the search corpus being reprocessed. In scenarios where only a few documents need to be reprocessed, use [Reset Documents (preview)](/rest/api/searchservice/preview-api/reset-documents) to force reprocessing of specific documents. When a document is reset, the indexer invalidates the cache for that document, which is then reprocessed by reading it from the data source. For more information, see [Run or reset indexers, skills, and documents](search-howto-run-reset-indexers.md).
118
+
[Resetting an indexer](/rest/api/searchservice/indexers/reset) will result in all documents in the search corpus being reprocessed.
119
+
120
+
In scenarios where only a few documents need to be reprocessed, use [Reset Documents (preview)](/rest/api/searchservice/indexers/reset-docs?view=rest-searchservice-2024-05-01-preview&preserve-view=true) to force reprocessing of specific documents. When a document is reset, the indexer invalidates the cache for that document, which is then reprocessed by reading it from the data source. For more information, see [Run or reset indexers, skills, and documents](search-howto-run-reset-indexers.md).
117
121
118
122
To reset specific documents, the request provides a list of document keys as read from the search index. If the key is mapped to a field in the external data source, the value that you provide should be the one used in the search index.
119
123
@@ -128,7 +132,7 @@ Depending on how you call the API, the request will either append, overwrite, or
128
132
The following example illustrates a reset document request:
129
133
130
134
```http
131
-
POST https://[search service name].search.windows.net/indexers/[indexer name]/resetdocs?api-version=2020-06-30-Preview
135
+
POST https://[search service name].search.windows.net/indexers/[indexer name]/resetdocs?api-version=2024-05-01-preview
132
136
{
133
137
"documentKeys" : [
134
138
"key1",
@@ -175,15 +179,17 @@ Incremental processing evaluates your skillset definition and determines which s
175
179
176
180
## APIs used for caching
177
181
178
-
REST API version `2020-06-30-Preview` or later provides incremental enrichment through additional properties on indexers. Skillsets and data sources can use the generally available version. In addition to the reference documentation, see [Configure caching for incremental enrichment](search-howto-incremental-index.md) for details about order of operations.
182
+
REST API version `2020-06-30-Preview` or later provides incremental enrichment through extra properties on indexers. We recommend the latest preview API.
183
+
184
+
Skillsets and data sources can use the generally available version. In addition to the reference documentation, see [Configure caching for incremental enrichment](search-howto-incremental-index.md) for details about order of operations.
179
185
180
-
+[Create or Update Indexer (api-version=2020-06-30-Preview)](/rest/api/searchservice/preview-api/create-or-update-indexer)
186
+
+[Create or Update Indexer (api-version=2024-05-01-preview)](/rest/api/searchservice/indexers/create-or-update?view=rest-searchservice-2024-05-01-preview&preserve-view=true)
181
187
182
-
+[Update Skillset (api-version=2020-06-30)](/rest/api/searchservice/update-skillset) (New URI parameter on the request)
+[Create or Update Skillset (api-version=2023-11-01)](/rest/api/searchservice/skillsets/create-or-update) (New URI parameter on the request)
185
191
186
-
+[Update Data Source](/rest/api/searchservice/update-data-source), when called with a preview API version, provides a new parameter named "ignoreResetRequirement", which should be set to true when your update action shouldn't invalidate the cache. Use "ignoreResetRequirement" sparingly as it could lead to unintended inconsistency in your data that won't be detected easily.
192
+
+[Create or Update Data Source (api-version=2023-11-01)](/rest/api/searchservice/data-sources/create-or-update), when called with a preview API version, provides a new parameter named "ignoreResetRequirement", which should be set to true when your update action shouldn't invalidate the cache. Use "ignoreResetRequirement" sparingly as it could lead to unintended inconsistency in your data that won't be detected easily.
Copy file name to clipboardExpand all lines: articles/search/knowledge-store-concept-intro.md
+4-6Lines changed: 4 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,7 +9,7 @@ ms.service: cognitive-search
9
9
ms.custom:
10
10
- ignite-2023
11
11
ms.topic: conceptual
12
-
ms.date: 01/10/2024
12
+
ms.date: 06/25/2024
13
13
---
14
14
15
15
# Knowledge store in Azure AI Search
@@ -75,7 +75,7 @@ The type of projection you specify in this structure determines the type of stor
75
75
76
76
To create knowledge store, use the portal or an API.
77
77
78
-
You'll need [Azure Storage](../storage/index.yml), a [skillset](cognitive-search-working-with-skillsets.md), and an [indexer](search-indexer-overview.md). Because indexers require a search index, you'll also need to provide an index definition.
78
+
You need [Azure Storage](../storage/index.yml), a [skillset](cognitive-search-working-with-skillsets.md), and an [indexer](search-indexer-overview.md). Because indexers require a search index, you also need to provide an index definition.
79
79
80
80
Go with the portal approach for the fastest route to a finished knowledge store. Or, choose the REST API for a deeper understanding of how objects are defined and related.
81
81
@@ -97,10 +97,8 @@ The wizard automates several tasks. Specifically, both shaping and projections (
97
97
98
98
[**Create a knowledge store using REST**](knowledge-store-create-rest.md) is a tutorial that walks you through the objects and requests belonging to this [knowledge store collection](https://github.com/Azure-Samples/azure-search-rest-samples/tree/main/knowledge-store).
99
99
100
-
REST API version `2020-06-30` and higher can be used to create a knowledge store through additions to a skillset.
Copy file name to clipboardExpand all lines: articles/search/search-file-storage-integration.md
+13-12Lines changed: 13 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,13 +9,13 @@ ms.service: cognitive-search
9
9
ms.custom:
10
10
- ignite-2023
11
11
ms.topic: how-to
12
-
ms.date: 12/12/2023
12
+
ms.date: 06/25/2024
13
13
---
14
14
15
15
# Index data from Azure Files
16
16
17
17
> [!IMPORTANT]
18
-
> Azure Files indexer is currently in public preview under [Supplemental Terms of Use](https://azure.microsoft.com/support/legal/preview-supplemental-terms/). Use a [preview REST API (2020-06-30-preview or later)](search-api-preview.md) to create the indexer data source.
18
+
> Azure Files indexer is currently in public preview under [Supplemental Terms of Use](https://azure.microsoft.com/support/legal/preview-supplemental-terms/). Use a [preview REST API](/rest/api/searchservice/search-service-api-versions#preview-versions) to create the indexer data source.
19
19
20
20
In this article, learn how to configure an [**indexer**](search-indexer-overview.md) that imports content from Azure Files and makes it searchable in Azure AI Search. Inputs to the indexer are your files in a single share. Output is a search index with searchable content and metadata stored in individual fields.
21
21
@@ -39,7 +39,6 @@ The Azure Files indexer can extract text from the following document formats:
By default, most files are indexed as a single search document in the index, including files with structured content, such as JSON or CSV, which are indexed as a single chunk of text.
@@ -48,14 +47,16 @@ A compound or embedded document (such as a ZIP archive, a Word document with emb
48
47
49
48
Textual content of a document is extracted into a string field named "content". You can also extract standard and user-defined metadata.
50
49
51
-
52
50
## Define the data source
53
51
54
52
The data source definition specifies the data to index, credentials, and policies for identifying changes in the data. A data source is defined as an independent resource so that it can be used by multiple indexers.
55
53
56
-
1.[Create or update a data source](/rest/api/searchservice/preview-api/create-or-update-data-source) to set its definition, using a preview API version 2020-06-30-Preview or later for "type": `"azurefile"`.
54
+
You can use 2020-06-30-preview or later for "type": `"azurefile"`. We recommend the latest preview API.
57
55
58
-
```json
56
+
1.[Create a data source](/rest/api/searchservice/indexers/create?view=rest-searchservice-2024-05-01-preview&preserve-view=true) to set its definition, using a preview API for "type": `"azurefile"`.
57
+
58
+
```http
59
+
POST /datasources?api-version=2024-05-01-preview
59
60
{
60
61
"name" : "my-file-datasource",
61
62
"type" : "azurefile",
@@ -87,10 +88,10 @@ Indexers can connect to a file share using the following connections.
87
88
88
89
In the [search index](search-what-is-an-index.md), add fields to accept the content and metadata of your Azure files.
89
90
90
-
1. [Create or update an index](/rest/api/searchservice/create-index) to define search fields that will store file content and metadata:
91
+
1. [Create or update an index](/rest/api/searchservice/indexes/create-or-update) to define search fields that will store file content and metadata.
91
92
92
93
```http
93
-
POST /indexes?api-version=2020-06-30
94
+
POST /indexes?api-version=2023-11-01
94
95
{
95
96
"name" : "my-search-index",
96
97
"fields": [
@@ -128,10 +129,10 @@ In the [search index](search-what-is-an-index.md), add fields to accept the cont
128
129
129
130
Once the index and data source have been created, you're ready to create the indexer. Indexer configuration specifies the inputs, parameters, and properties controlling run time behaviors.
130
131
131
-
1. [Create or update an indexer](/rest/api/searchservice/create-indexer) by giving it a name and referencing the data source and target index:
132
+
1. [Create or update an indexer](/rest/api/searchservice/indexers/create-or-update) by giving it a name and referencing the data source and target index:
132
133
133
134
```http
134
-
POST https://[service name].search.windows.net/indexers?api-version=2020-06-30
135
+
POST /indexers?api-version=2023-11-01
135
136
{
136
137
"name" : "my-file-indexer",
137
138
"dataSourceName" : "my-file-datasource",
@@ -165,10 +166,10 @@ An indexer runs automatically when it's created. You can prevent this by setting
165
166
166
167
## Check indexer status
167
168
168
-
To monitor the indexer status and execution history, send a [Get Indexer Status](/rest/api/searchservice/get-indexer-status) request:
169
+
To monitor the indexer status and execution history, send a [Get Indexer Status](/rest/api/searchservice/indexers/get-status) request:
169
170
170
171
```http
171
-
GET https://myservice.search.windows.net/indexers/myindexer/status?api-version=2020-06-30
172
+
GET https://myservice.search.windows.net/indexers/myindexer/status?api-version=2023-11-01
Copy file name to clipboardExpand all lines: articles/search/search-howto-index-changed-deleted-blobs.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -117,7 +117,7 @@ There are steps to follow in both Azure Storage and Azure AI Search, but there a
117
117
1. In Azure AI Search, edit the data source definition to include a "dataDeletionDetectionPolicy" property. For example, the following policy considers a file to be deleted if it has a metadata property `IsDeleted` with the value `true`:
118
118
119
119
```http
120
-
PUT https://[service name].search.windows.net/datasources/file-datasource?api-version=2020-06-30
120
+
PUT https://[service name].search.windows.net/datasources/file-datasource?api-version=2023-11-01
Copy file name to clipboardExpand all lines: articles/search/search-howto-index-cosmosdb-mongodb.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -50,7 +50,7 @@ As an alternative to this connector, if your scenario has any of those requireme
50
50
51
51
The data source definition specifies the data to index, credentials, and policies for identifying changes in the data. A data source is defined as an independent resource so that it can be used by multiple indexers.
52
52
53
-
For this call, specify a [preview REST API version](search-api-preview.md). You can use 2020-06-30-preview or later to create a data source that connects via the MongoDB API. We recommend using the latest preview REST API.
53
+
For this call, specify a [preview REST API version](search-api-preview.md). You can use 2020-06-30-preview or later to create a data source that connects via the MongoDB API. We recommend the latest preview REST API.
54
54
55
55
1.[Create or update a data source](/rest/api/searchservice/data-sources/create-or-update?view=rest-searchservice-2024-05-01-preview&preserve-view=true) to set its definition:
0 commit comments