You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/search/search-howto-index-changed-deleted-blobs.md
+38-19Lines changed: 38 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,7 +9,7 @@ manager: nitinme
9
9
10
10
ms.service: cognitive-search
11
11
ms.topic: how-to
12
-
ms.date: 01/19/2022
12
+
ms.date: 09/09/2022
13
13
---
14
14
15
15
# Change and delete detection using indexers for Azure Storage in Azure Cognitive Search
@@ -37,35 +37,54 @@ There are two ways to implement a soft delete strategy:
37
37
For this deletion detection approach, Cognitive Search depends on the [native blob soft delete](../storage/blobs/soft-delete-blob-overview.md) feature in Azure Blob Storage to determine whether blobs have transitioned to a soft deleted state. When blobs are detected in this state, a search indexer uses this information to remove the corresponding document from the index.
38
38
39
39
> [!IMPORTANT]
40
-
> Support for native blob soft delete is in preview under [Supplemental Terms of Use](https://azure.microsoft.com/support/legal/preview-supplemental-terms/). The [REST API version 2020-06-30-Preview](./search-api-preview.md) provides this feature. There is currently no portal or .NET SDK support.
40
+
> Support for native blob soft delete is in preview under [Supplemental Terms of Use](https://azure.microsoft.com/support/legal/preview-supplemental-terms/). The [REST API version 2020-06-30-Preview](./search-api-preview.md) provides this feature. There is currently no .NET SDK support.
41
41
42
42
### Requirements for native soft delete
43
43
44
44
+[Enable soft delete for blobs](../storage/blobs/soft-delete-blob-enable.md).
45
45
+ Blobs must be in an Azure Blob Storage container. The Cognitive Search native blob soft delete policy is not supported for blobs in ADLS Gen2.
46
46
+ Document keys for the documents in your index must be mapped to either be a blob property or blob metadata.
47
-
+ You must use the preview REST API (`api-version=2020-06-30-Preview`) to configure support for soft delete.
47
+
+ You must use the preview REST API (`api-version=2020-06-30-Preview`) or the indexer Data Source configuration in your Cognitive Search Service from the Azure portal, to configure support for soft delete.
48
48
49
49
### How to configure deletion detection using native soft delete
50
50
51
51
1. In Blob storage, when enabling soft delete, set the retention policy to a value that's much higher than your indexer interval schedule. This way if there's an issue running the indexer or if you have a large number of documents to index, there's plenty of time for the indexer to eventually process the soft deleted blobs. Azure Cognitive Search indexers will only delete a document from the index if it processes the blob while it's in a soft deleted state.
52
52
53
-
1. In Cognitive Search, set a native blob soft deletion detection policy on the data source. An example is shown below. Because this feature is in preview, you must use the preview REST API.
53
+
1. In Cognitive Search, set a native blob soft deletion detection policy on the data source. You can do this either from the Azure portal or by using preview REST API (`api-version=2020-06-30-Preview`).
54
54
55
-
```http
56
-
PUT https://[service name].search.windows.net/datasources/blob-datasource?api-version=2020-06-30-Preview
1.[Sign in to Azure portal](https://portal.azure.com).
58
+
59
+
1. On the Cognitive Search service Overview page, go to **New Data Source**, a visual editor for specifying a data source definition.
60
+
61
+
The following screenshot shows where you can find this feature in the portal.
62
+
63
+
:::image type="content" source="media/search-indexing-changed-deleted-blobs/new-data-source.png" alt-text="Screenshot of portal data source." border="true":::
64
+
65
+
1. On the **New Data Source** form, fill out the required fields, select the **Track deletions** checkbox and choose **Native blob soft delete**. Then hit **Save** to enable the feature on Data Source creation.
66
+
67
+
:::image type="content" source="media/search-indexing-changed-deleted-blobs/native-soft-delete.png" alt-text="Screenshot of portal data source native soft delete." border="true":::
68
+
69
+
70
+
### [**REST**](#tab/rest-api)
71
+
72
+
An example of using REST API to set soft deletion detection policy on the data source is shown below.
73
+
74
+
```http
75
+
PUT https://[service name].search.windows.net/datasources/blob-datasource?api-version=2020-06-30-Preview
1.[Run the indexer](/rest/api/searchservice/run-indexer) or set the indexer to run [on a schedule](search-howto-schedule-indexers.md). When the indexer runs and processes a blob having a soft delete state, the corresponding search document will be removed from the index.
71
90
@@ -118,4 +137,4 @@ You can reverse a soft-delete if the original source file still physically exist
118
137
119
138
+ [Indexers in Azure Cognitive Search](search-indexer-overview.md)
120
139
+ [How to configure a blob indexer](search-howto-indexing-azure-blob-storage.md)
Copy file name to clipboardExpand all lines: articles/search/search-howto-index-sharepoint-online.md
+10-8Lines changed: 10 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,7 +9,7 @@ manager: liamca
9
9
10
10
ms.service: cognitive-search
11
11
ms.topic: how-to
12
-
ms.date: 08/25/2022
12
+
ms.date: 09/08/2022
13
13
---
14
14
15
15
# Index data from SharePoint document libraries
@@ -215,7 +215,7 @@ api-key: [admin key]
215
215
216
216
An indexer connects a data source with a target search index and provides a schedule to automate the data refresh. Once the index and data source have been created, you're ready to create the indexer.
217
217
218
-
During this section you’ll be asked to sign in with your organization credentials that have access to the SharePoint site. If possible, we recommend creating a new organizational user account and giving that new user the exact permissions that you want the indexer to have.
218
+
During this section you’ll be asked to sign in with your organization credentials that have access to the SharePoint site. If possible, we recommend creating a new organizational user account and giving that new user the exact permissions that you want the indexer to have.
219
219
220
220
There are a few steps to creating the indexer:
221
221
@@ -249,7 +249,8 @@ There are a few steps to creating the indexer:
249
249
"mappingFunction" : {
250
250
"name" : "base64Encode"
251
251
}
252
-
}
252
+
}
253
+
]
253
254
}
254
255
```
255
256
@@ -293,9 +294,9 @@ There are a few steps to creating the indexer:
293
294
"maxFailedItemsPerBatch": null,
294
295
"base64EncodeKeys": null,
295
296
"configuration:" {
296
-
"indexedFileNameExtensions" : null,
297
-
"excludedFileNameExtensions" : null,
298
-
"dataToExtract": "contentAndMetadata"
297
+
"dataToExtract": "contentAndMetadata",
298
+
"indexedFileNameExtensions" : ".pdf, .docx",
299
+
"excludedFileNameExtensions" : ".png, .jpg"
299
300
}
300
301
},
301
302
"schedule" : { },
@@ -306,7 +307,8 @@ There are a few steps to creating the indexer:
306
307
"mappingFunction" : {
307
308
"name" : "base64Encode"
308
309
}
309
-
}
310
+
}
311
+
]
310
312
}
311
313
```
312
314
@@ -467,4 +469,4 @@ You can also continue indexing if errors happen at any point of processing, eith
467
469
## See also
468
470
469
471
+[Indexers in Azure Cognitive Search](search-indexer-overview.md)
470
-
+[Content metadata properties used in Azure Cognitive Search](search-blob-metadata-properties.md)
472
+
+[Content metadata properties used in Azure Cognitive Search](search-blob-metadata-properties.md)
0 commit comments