You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/search/cognitive-search-incremental-indexing-conceptual.md
+11-23Lines changed: 11 additions & 23 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -94,7 +94,7 @@ PUT https://customerdemos.search.windows.net/datasources/callcenter-ds?api-versi
94
94
95
95
The purpose of the cache is to avoid unnecessary processing, but suppose you make a change to a skill that the indexer doesn't detect (for example, changing something in external code, such as a custom skill).
96
96
97
-
In this case, you can use the [Reset Skills](preview-api-resetskills.md) to force reprocessing of a particular skill, including any downstream skills that have a dependency on that skill's output. This API accepts a POST request with a list of skills that should be invalidated and marked for reprocessing. After Reset Skills, run the indexer to invoke the pipeline.
97
+
In this case, you can use the [Reset Skills](https://docs.microsoft.com/rest/api/searchservice/2019-05-06-preview/reset-skills) to force reprocessing of a particular skill, including any downstream skills that have a dependency on that skill's output. This API accepts a POST request with a list of skills that should be invalidated and marked for reprocessing. After Reset Skills, run the indexer to invoke the pipeline.
98
98
99
99
## Change detection
100
100
@@ -133,39 +133,27 @@ Incremental processing evaluates your skillset definition and determines which s
133
133
* Changes to the knowledge store projections, results in reprojecting documents
134
134
* Output field mappings changed on an indexer results in reprojecting documents to the index
135
135
136
-
## API reference content for incremental enrichment
136
+
## API reference
137
137
138
-
REST `api-version=2019-05-06-Preview` provides the APIs for incremental enrichment, with additions to indexers, skillsets, and data sources. [Official reference documentation](https://docs.microsoft.com/rest/api/searchservice/) is for generally available APIs and does not cover preview features. The following section provides the reference content for impacted APIs.
138
+
REST API version`2019-05-06-Preview` provides incremental enrichment through additional properties on indexers, skillsets, and data sources. In addition to the reference documentation, see [Configure caching for incremental enrichment](search-howto-incremental-index.md) for details on how to call the APIs.
139
139
140
-
Usage information and examples can be found in [Configure caching for incremental enrichment](search-howto-incremental-index.md).
[Create Indexer](https://docs.microsoft.com/rest/api/searchservice/create-indexer) and [Update Indexer](https://docs.microsoft.com/rest/api/searchservice/update-indexer) will now expose new properties relating to the cache:
144
+
+[Update Skillset (api-version=2019-05-06-Preview)](https://docs.microsoft.com/rest/api/searchservice/2019-05-06-preview/update-skillset) (New URI parameter on the request)
145
145
146
-
+`StorageAccountConnectionString`: The connection string to the storage account that will be used to cache the intermediate results.
+`EnableReprocessing`: Set to `true` by default, when set to `false`, documents will continue to be written to the cache, but no existing documents will be reprocessed based on the cache data.
148
+
+Database indexers (Azure SQL, Cosmos DB). Some indexers retrieve data through queries. For queries that retrieve data, [Update Data Source](https://docs.microsoft.com/rest/api/searchservice/update-data-source) supports a new parameter on a request **ignoreResetRequirement**, which should be set to `true` when your update action should not invalidate the cache.
149
149
150
-
+`ID` (read-only): The `ID` is the identifier of the container within the `annotationCache` storage account that will be used as the cache for this indexer. This cache will be unique to this indexer and if the indexer is deleted and recreated with the same name, the `ID` will be regenerated. The `ID` cannot be set, it is always generated by the service.
151
-
152
-
### Skillsets
153
-
154
-
+[Update Skillset](https://docs.microsoft.com/rest/api/searchservice/update-skillset) supports a new parameter on the request: `disableCacheReprocessingChangeDetection`, which should be set to `true` when you want no updates to existing documents based on the current action.
155
-
156
-
+[Reset Skills](preview-api-resetskills.md) is a new operation used to invalidate a skillset.
157
-
158
-
### Datasources
159
-
160
-
+ Some indexers retrieve data through queries. For queries that retrieve data, [Update Data Source](https://docs.microsoft.com/rest/api/searchservice/update-data-source) supports a new parameter on a request `ignoreResetRequirement`, which should be set to `true` when your update action should not invalidate the cache.
161
-
162
-
Use the `ignoreResetRequirement` sparingly as it could lead to unintended inconsistency in your data that will not be detected easily.
150
+
Use **ignoreResetRequirement** sparingly as it could lead to unintended inconsistency in your data that will not be detected easily.
163
151
164
152
## Next steps
165
153
166
-
Incremental enrichment is a powerful feature that extends change tracking to skillsets and AI enrichment. As skillsets evolve, incremental enrichment ensures the least possible work is done while still driving your documents to eventual consistency.
154
+
Incremental enrichment is a powerful feature that extends change tracking to skillsets and AI enrichment. AIncremental enrichment enables reuse of existing processed content as you iterate over skillset design.
167
155
168
-
Get started by adding a cache to an existing indexer or add the cache when defining a new indexer.
156
+
As a next step, enable caching on an existing indexer or add a cache when defining a new indexer.
169
157
170
158
> [!div class="nextstepaction"]
171
159
> [Configure caching for incremental enrichment](search-howto-incremental-index.md)
Copy file name to clipboardExpand all lines: articles/search/knowledge-store-concept-intro.md
+3-139Lines changed: 3 additions & 139 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -130,147 +130,11 @@ Once the enrichments exist in storage, any tool or technology that connects to A
130
130
131
131
## API reference
132
132
133
-
This section is a version of the [Create Skillset (REST API)](https://docs.microsoft.com/rest/api/searchservice/create-skillset) reference doc, modified to include a `knowledgeStore` definition.
133
+
REST API version `2019-05-06-Preview` provides knowledge store through additional definitions on skillsets. In addition to the reference, see [Create a knowledge store using Postman](knowledge-store-create-rest.md) for details on how to call the APIs.
134
134
135
-
### Example - knowledgeStore embedded in a Skillset
The following JSON specifies a `knowledgeStore`, which is part of a [`skillset`](https://docs.microsoft.com/rest/api/searchservice/create-skillset), which is invoked by an `indexer` (not shown). If you are already familiar with AI enrichment, a skillset determines the composition of an enriched document. A skillset must contain at least one skill, most likely a Shaper skill if you are modulating data structures.
214
-
215
-
The syntax for structuring the request payload is as follows.
216
-
217
-
```json
218
-
{
219
-
"name" : "Required for POST, optional for PUT requests which sets the name on the URI",
220
-
"description" : "Optional. Anything you want, or null",
221
-
"skills" : "Required. An array of skills. Each skill has an odata.type, name, input and output parameters",
222
-
"cognitiveServices": "A key to Cognitive Services, used for billing.",
A `knowledgeStore` has two properties: a `storageConnectionString` to an Azure Storage account, and `projections` that defines physical storage. You can use any storage account, but it's cost-effective to use services in the same region.
256
-
257
-
A `projections` collection contains projection objects. Each projection object must have `tables`, `objects`, `files` (one of each), which are either specified or null. The syntax above shows two objects, one fully specified and the other fully null. Within a projection object, once it is expressed in storage, any relationships among the data, if detected, are preserved.
258
-
259
-
Create as many projection objects as you need to support isolation and specific scenarios (for example, data structures used for exploration, versus those needed in a data science workload). You can get isolation and customization for specific scenarios by setting `source` and `storageContainer` or `table` to different values within an object. For more information and examples, see [Working with projections in a knowledge store](knowledge-store-projection-overview.md).
260
-
261
-
|Property | Applies to | Description|
262
-
|--------------|------------|------------|
263
-
|`storageConnectionString`| `knowledgeStore` | Required. In this format: `DefaultEndpointsProtocol=https;AccountName=<ACCOUNT-NAME>;AccountKey=<ACCOUNT-KEY>;EndpointSuffix=core.windows.net`|
264
-
|`projections`| `knowledgeStore` | Required. A collection of property objects consisting of `tables`, `objects`, `files` and their respective properties. Unused projections can be set to null.|
265
-
|`source`| All projections| The path to the node of the enrichment tree that is the root of the projection. This node is the output of any of the skills in the skillset. Paths start with `/document/`, representing the enriched document but can be extended to `/document/content/` or to nodes within the document tree. Examples: `/document/countries/*` (all countries), or `/document/countries/*/states/*` (all states in all countries). For more information on document paths, see [Skillset concepts and composition](cognitive-search-working-with-skillsets.md).|
266
-
|`tableName`| `tables`| A table to create in Azure Table storage. |
267
-
|`storageContainer`| `objects`, `files`| Name of a container to create in Azure Blob storage. |
268
-
|`generatedKeyName`| `tables`| A column created in the table that uniquely identifies a document. The enrichment pipeline populates this column with generated values.|
269
-
270
-
271
-
### Response
272
-
273
-
For a successful request, you should see status code "201 Created". By default, the response body will contain the JSON for the skillset definition that was created. Recall that the knowledge store is not created until you invoke an indexer that references this skillset.
0 commit comments