You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/search/search-howto-index-changed-deleted-blobs.md
+8-12Lines changed: 8 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,7 +10,7 @@ ms.service: cognitive-search
10
10
ms.custom:
11
11
- ignite-2023
12
12
ms.topic: how-to
13
-
ms.date: 11/09/2022
13
+
ms.date: 12/19/2023
14
14
---
15
15
16
16
# Change and delete detection using indexers for Azure Storage in Azure AI Search
@@ -21,7 +21,7 @@ Although change detection is a given, deletion detection isn't. An indexer doesn
21
21
22
22
There are two ways to implement a soft delete strategy:
23
23
24
-
+[Native blob soft delete (preview)](#native-blob-soft-delete-preview), applies to Blob Storage only
24
+
+[Native blob soft delete](#native-blob-soft-delete), applies to Blob Storage only
25
25
+[Soft delete using custom metadata](#soft-delete-using-custom-metadata)
26
26
27
27
## Prerequisites
@@ -33,32 +33,28 @@ There are two ways to implement a soft delete strategy:
33
33
> [!NOTE]
34
34
> ADLS Gen2 allows directories to be renamed. When a directory is renamed, the timestamps for the blobs in that directory do not get updated. As a result, the indexer will not re-index those blobs. If you need the blobs in a directory to be reindexed after a directory rename because they now have new URLs, you will need to update the `LastModified` timestamp for all the blobs in the directory so that the indexer knows to re-index them during a future run. The virtual directories in Azure Blob Storage cannot be changed, so they do not have this issue.
35
35
36
-
## Native blob soft delete (preview)
36
+
## Native blob soft delete
37
37
38
38
For this deletion detection approach, Azure AI Search depends on the [native blob soft delete](../storage/blobs/soft-delete-blob-overview.md) feature in Azure Blob Storage to determine whether blobs have transitioned to a soft deleted state. When blobs are detected in this state, a search indexer uses this information to remove the corresponding document from the index.
39
39
40
-
> [!IMPORTANT]
41
-
> Support for native blob soft delete is in preview under [Supplemental Terms of Use](https://azure.microsoft.com/support/legal/preview-supplemental-terms/). The [REST API version 2020-06-30-Preview](./search-api-preview.md) provides this feature. There's currently no .NET SDK support.
42
-
43
40
### Requirements for native soft delete
44
41
45
-
+ Blobs must be in an Azure Blob Storage container. The Azure AI Search native blob soft delete policy isn't supported for blobs in ADLS Gen2.
42
+
+ Blobs must be in an Azure Blob Storage container. The Azure AI Search native blob soft delete policy isn't supported for blobs in ADLS Gen2 or Azure Files.
46
43
47
44
+[Enable soft delete for blobs](../storage/blobs/soft-delete-blob-enable.md).
48
45
49
46
+ Document keys for the documents in your index must be mapped to either be a blob property or blob metadata, such as "metadata_storage_path".
50
47
51
-
+ You must use the preview REST API (`api-version=2020-06-30-Preview`), or the indexer Data Source configuration in the Azure portal, to configure support for soft delete.
48
+
+ You must use the [REST API (`api-version=2023-11-01`)](/rest/api/searchservice/search-service-api-versions) or newer version, or the indexer Data Source configuration in the Azure portal, to configure support for soft delete.
52
49
53
50
+[Blob versioning](../storage/blobs/versioning-overview.md) must not be enabled in the storage account. Otherwise, native soft delete isn't supported by design.
54
51
55
52
56
-
57
53
### Configure native soft delete
58
54
59
55
In Blob storage, when enabling soft delete per the requirements, set the retention policy to a value that's much higher than your indexer interval schedule. If there's an issue running the indexer, or if you have a large number of documents to index, there's plenty of time for the indexer to eventually process the soft deleted blobs. Azure AI Search indexers will only delete a document from the index if it processes the blob while it's in a soft deleted state.
60
56
61
-
In Azure AI Search, set a native blob soft deletion detection policy on the data source. You can do this either from the Azure portal or by using preview REST API (`api-version=2020-06-30-Preview`). The following instructions explain how to set the delete detection policy in Azure portal or through REST APIs.
57
+
In Azure AI Search, set a native blob soft deletion detection policy on the data source. You can do this either from the Azure portal, by using REST API (`api-version=2023-11-01`). The following instructions explain how to set the delete detection policy in Azure portal or through REST APIs.
62
58
63
59
### [**Azure portal**](#tab/portal)
64
60
@@ -76,10 +72,10 @@ In Azure AI Search, set a native blob soft deletion detection policy on the data
76
72
77
73
### [**REST**](#tab/rest-api)
78
74
79
-
Set the soft deletion detection policy in the data source definition. Specify the preview API version when creating or updating the data source.
75
+
Set the soft deletion detection policy in the data source definition. Specify the API version when creating or updating the data source.
80
76
81
77
```http
82
-
PUT https://[service name].search.windows.net/datasources/blob-datasource?api-version=2020-06-30-Preview
78
+
PUT https://[service name].search.windows.net/datasources/blob-datasource?api-version=2023-11-01
0 commit comments