Skip to content

Commit 1d4db17

Browse files
Merge pull request #210621 from gmndrg/main
Removed directory created in wrong location
2 parents cf89eab + 68a4b98 commit 1d4db17

File tree

9 files changed

+48
-27
lines changed

9 files changed

+48
-27
lines changed
60.9 KB
Loading
26.9 KB
Loading

articles/search/search-howto-index-changed-deleted-blobs.md

Lines changed: 38 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ manager: nitinme
99

1010
ms.service: cognitive-search
1111
ms.topic: how-to
12-
ms.date: 01/19/2022
12+
ms.date: 09/09/2022
1313
---
1414

1515
# Change and delete detection using indexers for Azure Storage in Azure Cognitive Search
@@ -37,35 +37,54 @@ There are two ways to implement a soft delete strategy:
3737
For this deletion detection approach, Cognitive Search depends on the [native blob soft delete](../storage/blobs/soft-delete-blob-overview.md) feature in Azure Blob Storage to determine whether blobs have transitioned to a soft deleted state. When blobs are detected in this state, a search indexer uses this information to remove the corresponding document from the index.
3838

3939
> [!IMPORTANT]
40-
> Support for native blob soft delete is in preview under [Supplemental Terms of Use](https://azure.microsoft.com/support/legal/preview-supplemental-terms/). The [REST API version 2020-06-30-Preview](./search-api-preview.md) provides this feature. There is currently no portal or .NET SDK support.
40+
> Support for native blob soft delete is in preview under [Supplemental Terms of Use](https://azure.microsoft.com/support/legal/preview-supplemental-terms/). The [REST API version 2020-06-30-Preview](./search-api-preview.md) provides this feature. There is currently no .NET SDK support.
4141
4242
### Requirements for native soft delete
4343

4444
+ [Enable soft delete for blobs](../storage/blobs/soft-delete-blob-enable.md).
4545
+ Blobs must be in an Azure Blob Storage container. The Cognitive Search native blob soft delete policy is not supported for blobs in ADLS Gen2.
4646
+ Document keys for the documents in your index must be mapped to either be a blob property or blob metadata.
47-
+ You must use the preview REST API (`api-version=2020-06-30-Preview`) to configure support for soft delete.
47+
+ You must use the preview REST API (`api-version=2020-06-30-Preview`) or the indexer Data Source configuration in your Cognitive Search Service from the Azure portal, to configure support for soft delete.
4848

4949
### How to configure deletion detection using native soft delete
5050

5151
1. In Blob storage, when enabling soft delete, set the retention policy to a value that's much higher than your indexer interval schedule. This way if there's an issue running the indexer or if you have a large number of documents to index, there's plenty of time for the indexer to eventually process the soft deleted blobs. Azure Cognitive Search indexers will only delete a document from the index if it processes the blob while it's in a soft deleted state.
5252

53-
1. In Cognitive Search, set a native blob soft deletion detection policy on the data source. An example is shown below. Because this feature is in preview, you must use the preview REST API.
53+
1. In Cognitive Search, set a native blob soft deletion detection policy on the data source. You can do this either from the Azure portal or by using preview REST API (`api-version=2020-06-30-Preview`).
5454

55-
```http
56-
PUT https://[service name].search.windows.net/datasources/blob-datasource?api-version=2020-06-30-Preview
57-
Content-Type: application/json
58-
api-key: [admin key]
59-
{
60-
"name" : "blob-datasource",
61-
"type" : "azureblob",
62-
"credentials" : { "connectionString" : "<your storage connection string>" },
63-
"container" : { "name" : "my-container", "query" : null },
64-
"dataDeletionDetectionPolicy" : {
65-
"@odata.type" :"#Microsoft.Azure.Search.NativeBlobSoftDeleteDeletionDetectionPolicy"
66-
}
67-
}
68-
```
55+
### [**Azure portal**](#tab/portal)
56+
57+
1. [Sign in to Azure portal](https://portal.azure.com).
58+
59+
1. On the Cognitive Search service Overview page, go to **New Data Source**, a visual editor for specifying a data source definition.
60+
61+
The following screenshot shows where you can find this feature in the portal.
62+
63+
:::image type="content" source="media/search-indexing-changed-deleted-blobs/new-data-source.png" alt-text="Screenshot of portal data source." border="true":::
64+
65+
1. On the **New Data Source** form, fill out the required fields, select the **Track deletions** checkbox and choose **Native blob soft delete**. Then hit **Save** to enable the feature on Data Source creation.
66+
67+
:::image type="content" source="media/search-indexing-changed-deleted-blobs/native-soft-delete.png" alt-text="Screenshot of portal data source native soft delete." border="true":::
68+
69+
70+
### [**REST**](#tab/rest-api)
71+
72+
An example of using REST API to set soft deletion detection policy on the data source is shown below.
73+
74+
```http
75+
PUT https://[service name].search.windows.net/datasources/blob-datasource?api-version=2020-06-30-Preview
76+
Content-Type: application/json
77+
api-key: [admin key]
78+
{
79+
"name" : "blob-datasource",
80+
"type" : "azureblob",
81+
"credentials" : { "connectionString" : "<your storage connection string>" },
82+
"container" : { "name" : "my-container", "query" : null },
83+
"dataDeletionDetectionPolicy" : {
84+
"@odata.type" :"#Microsoft.Azure.Search.NativeBlobSoftDeleteDeletionDetectionPolicy"
85+
}
86+
}
87+
```
6988

7089
1. [Run the indexer](/rest/api/searchservice/run-indexer) or set the indexer to run [on a schedule](search-howto-schedule-indexers.md). When the indexer runs and processes a blob having a soft delete state, the corresponding search document will be removed from the index.
7190

@@ -118,4 +137,4 @@ You can reverse a soft-delete if the original source file still physically exist
118137
119138
+ [Indexers in Azure Cognitive Search](search-indexer-overview.md)
120139
+ [How to configure a blob indexer](search-howto-indexing-azure-blob-storage.md)
121-
+ [Blob indexing overview](search-blob-storage-integration.md)
140+
+ [Blob indexing overview](search-blob-storage-integration.md)

articles/search/search-howto-index-sharepoint-online.md

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ manager: liamca
99

1010
ms.service: cognitive-search
1111
ms.topic: how-to
12-
ms.date: 08/25/2022
12+
ms.date: 09/08/2022
1313
---
1414

1515
# Index data from SharePoint document libraries
@@ -215,7 +215,7 @@ api-key: [admin key]
215215

216216
An indexer connects a data source with a target search index and provides a schedule to automate the data refresh. Once the index and data source have been created, you're ready to create the indexer.
217217

218-
During this section you’ll be asked to sign in with your organization credentials that have access to the SharePoint site. If possible, we recommend creating a new organizational user account and giving that new user the exact permissions that you want the indexer to have.
218+
During this section you’ll be asked to sign in with your organization credentials that have access to the SharePoint site. If possible, we recommend creating a new organizational user account and giving that new user the exact permissions that you want the indexer to have.
219219

220220
There are a few steps to creating the indexer:
221221

@@ -249,7 +249,8 @@ There are a few steps to creating the indexer:
249249
"mappingFunction" : {
250250
"name" : "base64Encode"
251251
}
252-
}
252+
}
253+
]
253254
}
254255
```
255256
@@ -293,9 +294,9 @@ There are a few steps to creating the indexer:
293294
"maxFailedItemsPerBatch": null,
294295
"base64EncodeKeys": null,
295296
"configuration:" {
296-
"indexedFileNameExtensions" : null,
297-
"excludedFileNameExtensions" : null,
298-
"dataToExtract": "contentAndMetadata"
297+
"dataToExtract": "contentAndMetadata",
298+
"indexedFileNameExtensions" : ".pdf, .docx",
299+
"excludedFileNameExtensions" : ".png, .jpg"
299300
}
300301
},
301302
"schedule" : { },
@@ -306,7 +307,8 @@ There are a few steps to creating the indexer:
306307
"mappingFunction" : {
307308
"name" : "base64Encode"
308309
}
309-
}
310+
}
311+
]
310312
}
311313
```
312314
@@ -467,4 +469,4 @@ You can also continue indexing if errors happen at any point of processing, eith
467469
## See also
468470

469471
+ [Indexers in Azure Cognitive Search](search-indexer-overview.md)
470-
+ [Content metadata properties used in Azure Cognitive Search](search-blob-metadata-properties.md)
472+
+ [Content metadata properties used in Azure Cognitive Search](search-blob-metadata-properties.md)
Binary file not shown.

0 commit comments

Comments
 (0)