Merge pull request #278859 from HeidiSteen/heidist-june12

[azure search] cross-link security trimming to related content

Author: v-ccolin

articles/search/search-howto-index-sharepoint-online.md

@@ -58,7 +58,7 @@ Here are the limitations of this feature:
 
 + Renaming a SharePoint folder doesn't trigger incremental indexing. A renamed folder is treated as new content.
 
-+ SharePoint supports a granular authorization model that determines per-user access at the document level. The indexer doesn't pull these permissions into the index, and Azure AI Search doesn't support document-level authorization. When a document is indexed from SharePoint into a search service, the content is available to anyone who has read access to the index. If you require document-level permissions, you should consider [security filters to trim results](search-security-trimming-for-azure-search-with-aad.md) and automate copying the permissions at a file level to a field in the index.
++ SharePoint supports a granular authorization model that determines per-user access at the document level. The indexer doesn't pull these permissions into the index, and Azure AI Search doesn't support document-level authorization. When a document is indexed from SharePoint into a search service, the content is available to anyone who has read access to the index. If you require document-level permissions, you should consider [security filters to trim results](search-security-trimming-for-azure-search.md) and automate copying the permissions at a file level to a field in the index.
 
 + Indexing user-encrypted files, Information Rights Management (IRM) protected files, ZIP files with passwords or similar encrypted content isn't supported. For encrypted content to be processed, the user with proper permissions to the specific file must remove the encryption so the item can be indexed accordingly when the indexer runs the next scheduled iteration.
 
@@ -287,7 +287,7 @@ There are a few steps to creating the indexer:
     }
     ```
 
-If you're using application permissions, it's necessary to wait until the initial run is complete before starting to query your index. The following instructions provided in this step pertain specifically to delegated permissions, and are not applicable to application permissions.
+   If you're using application permissions, it's necessary to wait until the initial run is complete before starting to query your index. The following instructions provided in this step pertain specifically to delegated permissions, and are not applicable to application permissions.
 
 1. When you create the indexer for the first time, the [Create Indexer (preview)](/rest/api/searchservice/indexers/create-or-update?view=rest-searchservice-2023-10-01-preview&tabs=HTTP&preserve-view=true) request waits until you complete the next step. You must call [Get Indexer Status](/rest/api/searchservice/indexers/get-status?view=rest-searchservice-2023-10-01-preview&tabs=HTTP&preserve-view=true) to get the link and enter your new device code. 
 
@@ -299,7 +299,7 @@ If you're using application permissions, it's necessary to wait until the initia
 
     If you don’t run the [Get Indexer Status](/rest/api/searchservice/indexers/get-status?view=rest-searchservice-2023-10-01-preview&tabs=HTTP&preserve-view=true) within 10 minutes, the code expires and you’ll need to recreate the [data source](#create-data-source).
 
- 1. Copy the device login code from the [Get Indexer Status](/rest/api/searchservice/indexers/get-status?view=rest-searchservice-2023-10-01-preview&tabs=HTTP&preserve-view=true) response. The device login can be found in the "errorMessage".
+1. Copy the device login code from the [Get Indexer Status](/rest/api/searchservice/indexers/get-status?view=rest-searchservice-2023-10-01-preview&tabs=HTTP&preserve-view=true) response. The device login can be found in the "errorMessage".
 
     ```http
     {
@@ -309,6 +309,7 @@ If you're using application permissions, it's necessary to wait until the initia
         }
     }
     ```
+
 1. Provide the code that was included in the error message.
 
     :::image type="content" source="media/search-howto-index-sharepoint-online/enter-device-code.png" alt-text="Screenshot showing how to enter a device code.":::

articles/search/search-security-overview.md

@@ -156,12 +156,7 @@ For multitenancy solutions requiring security boundaries at the index level, it'
 
 User permissions at the document level, also known as *row-level security*, isn't natively supported in Azure AI Search. If you import data from an external system that provides row-level security, such as Azure Cosmos DB, those permissions won't transfer with the data as its being indexed by Azure AI Search.
 
-If you require permissioned access over content in search results, there's a technique for applying filters that include or exclude documents based on user identity. This workaround adds a string field in the data source that represents a group or user identity, which you can make filterable in your index. The following table describes two approaches for trimming search results of unauthorized content.
-
-| Approach | Description |
-|----------|-------------|
-|[Security trimming based on identity filters](search-security-trimming-for-azure-search.md)  | Documents the basic workflow for implementing user identity access control. It covers adding security identifiers to an index, and then explains filtering against that field to trim results of prohibited content. |
-|[Security trimming based on Microsoft Entra identities](search-security-trimming-for-azure-search-with-aad.md)  | This article expands on the previous article, providing steps for retrieving identities from Microsoft Entra ID, one of the [free services](https://azure.microsoft.com/free/) in the Azure cloud platform. |
+If you require permissioned access over content in search results, there's a technique for applying filters that include or exclude documents based on user identity. This workaround adds a string field in the data source that represents a group or user identity, which you can make filterable in your index. For more information about this pattern, see [Security trimming based on identity filters](search-security-trimming-for-azure-search.md).
 
 ## Data residency
 

articles/search/search-security-trimming-for-azure-search-with-aad.md

@@ -12,6 +12,7 @@ ms.date: 02/15/2024
 ms.custom:
   - devx-track-csharp
   - ignite-2023
+ROBOTS: NOINDEX,NOFOLLOW
 ---
 # Security filters for trimming Azure AI Search results using Microsoft Entra tenants and identities
 

articles/search/search-security-trimming-for-azure-search.md

@@ -1,5 +1,5 @@
 ---
-title: Security filters for trimming results
+title: Security filter pattern
 titleSuffix: Azure AI Search
 description: Learn how to implement security privileges at the document level for Azure AI Search search results, using security filters and user identities.
 
@@ -10,68 +10,73 @@ ms.service: cognitive-search
 ms.custom:
   - ignite-2023
 ms.topic: how-to
-ms.date: 01/10/2024
+ms.date: 06/20/2024
 ---
 
 # Security filters for trimming results in Azure AI Search
 
-Azure AI Search doesn't provide document-level permissions and can't vary search results from within the same index by user permissions. As a workaround, you can create a filter that trims search results based on a string containing a group or user identity.
+Azure AI Search doesn't provide native document-level permissions and can't vary search results from within the same index by user permissions. As a workaround, you can create a filter that trims search results based on a string containing a group or user identity.
 
-This article describes a pattern for security filtering that includes following steps:
+This article describes a pattern for security filtering having the following steps:
 
 > [!div class="checklist"]
 > * Assemble source documents with the required content
 > * Create a field for the principal identifiers 
 > * Push the documents to the search index for indexing
 > * Query the index with the `search.in` filter function
 
+It concludes with links to demos and examples that provide hands-on learning. We recommend reviewing this article first to understand the pattern.
+
 ## About the security filter pattern
 
-Although Azure AI Search doesn't integrate with security subsystems for access to content within an index, many customers who have document-level security requirements have found that filters can meet their needs.
+Although Azure AI Search doesn't integrate with security subsystems for access to content within an index, many customers who have document-level security requirements find that filters can meet their needs.
 
-In Azure AI Search, a security filter is a regular OData filter that includes or excludes a search result based on a matching value, except that in a security filter, the criteria is a string consisting of a security principal. There's no authentication or authorization through the security principal. The principal is just a string, used in a filter expression, to include or exclude a document from the search results.
+In Azure AI Search, a security filter is a regular OData filter that includes or excludes a search result based on a string consisting of a security principal. There's no authentication or authorization through the security principal. The principal is just a string, used in a filter expression, to include or exclude a document from the search results.
 
 There are several ways to achieve security filtering. One way is through a complicated disjunction of equality expressions: for example, `Id eq 'id1' or Id eq 'id2'`, and so forth. This approach is error-prone, difficult to maintain, and in cases where the list contains hundreds or thousands of values, slows down query response time by many seconds. 
 
 A better solution is using the `search.in` function for security filters, as described in this article. If you use `search.in(Id, 'id1, id2, ...')` instead of an equality expression, you can expect subsecond response times.
 
 ## Prerequisites
 
-* The field containing group or user identity must be a string with the filterable attribute. It should be a collection. It shouldn't allow nulls.
+* A string field containing a group or user identity, such as a Microsoft Entra object identifier.
 
-* Other fields in the same document should provide the content that's accessible to that group or user. In the following JSON documents, the "security_id" fields contain identities used in a security filter, and the name, salary, and marital status will be included if the identity of the caller matches the "security_id" of the document.
+* Other fields in the same document should provide the content that's accessible to that group or user. In the following JSON documents, the "security_id" fields contain identities used in a security filter, and the name, salary, and marital status are included if the identity of the caller matches the "security_id" of the document.
 
     ```json
     {  
         "Employee-1": {  
-            "id": "100-1000-10-1-10000-1",
+            "employee_id": "100-1000-10-1-10000-1",
             "name": "Abram",   
             "salary": 75000,   
             "married": true,
-            "security_id": "10011"
+            "security_id": "alphanumeric-object-id-for-employee-1"
         },
         "Employee-2": {  
-            "id": "200-2000-20-2-20000-2",
+            "employee_id": "200-2000-20-2-20000-2",
             "name": "Adams",   
             "salary": 75000,   
             "married": true,
-            "security_id": "20022"
+            "security_id": "alphanumeric-object-id-for-employee-2"
         } 
     }  
     ```
 
-   >[!NOTE]
-   > The process of retrieving the principal identifiers and injecting those strings into source documents that can be indexed by Azure AI Search isn't covered in this article. Refer to the documentation of your identity service provider for help with obtaining identifiers.
-
 ## Create security field
 
-In the search index, within the field collection, you need one field that contains the group or user identity, similar to the fictitious "security_id" field in the previous example.
+In the search index, within the fields collection, you need one field that contains the group or user identity, similar to the fictitious "security_id" field in the previous example.
+
+1. Add a security field as a `Collection(Edm.String)`.
+
+1. Set the field's `filterable` attribute set to `true`.
 
-1. Add a security field as a `Collection(Edm.String)`. Make sure it has a `filterable` attribute set to `true` so that search results are filtered based on the access the user has. For example, if you set the `group_ids` field to `["group_id1, group_id2"]` for the document with `file_name` "secured_file_b", only users that belong to group IDs "group_id1" or "group_id2" have read access to the file.
+1. Set the field's `retrievable` attribute to `false` so that it isn't returned as part of the search request.
 
-   Set the field's `retrievable` attribute to `false` so that it isn't returned as part of the search request.
+1. Indexes require a document key. The "file_id" field satisfies that requirement. 
 
-1. Indexes require a document key. The "file_id" field satisfies that requirement. Indexes should also contain searchable content. The "file_name" and "file_description" fields represent that in this example.  
+1. Indexes should also contain searchable and retrievable content. The "file_name" and "file_description" fields represent that in this example.
+
+   The following index schema satisfies the field requirements. Documents that you index on Azure AI Search should have values for all of these fields, including the "group_ids". For the document with `file_name` "secured_file_b", only users that belong to group IDs "group_id1" or "group_id2" have read access to the file.
 
    ```https
    POST https://[search service].search.windows.net/indexes/securedfiles/docs/index?api-version=2023-11-01
@@ -87,23 +92,25 @@ In the search index, within the field collection, you need one field that contai
    ```
 
 ## Push data into your index using the REST API
-  
-Send an HTTP POST request to the docs collection of your index's URL endpoint (see [Documents - Index](/rest/api/searchservice/documents/)). The body of the HTTP request is a JSON rendering of the documents to be indexed:
 
-```http
-POST https://[search service].search.windows.net/indexes/securedfiles/docs/index?api-version=2023-11-01
-```
+Populate your search index with documents that provide values for each field in the fields collection, including values for the security field. Azure AI Search doesn't provide APIs or features for populating the security field specifically. However, several of the examples listed at the end of this article explain techniques for populating this field.
 
-In the request body, specify the content of your documents:
+In Azure AI Search, the approaches for loading data are:
 
-```JSON
+* A single push or pull (indexer) operation that imports documents populated with all fields
+* Multiple push or pull operations. As long as secondary import operations target the right document identifier, you can load fields individually through multiple imports.
+
+The following example shows a single HTTP POST request to the docs collection of your index's URL endpoint (see [Documents - Index](/rest/api/searchservice/documents/)). The body of the HTTP request is a JSON rendering of the documents to be indexed:
+
+```http
+POST https://[search service].search.windows.net/indexes/securedfiles/docs/index?api-version=2023-11-01
 {
     "value": [
         {
             "@search.action": "upload",
             "file_id": "1",
             "file_name": "secured_file_a",
-            "file_description": "File access is restricted to the Human Resources.",
+            "file_description": "File access is restricted to Human Resources.",
             "group_ids": ["group_id1"]
         },
         {
@@ -147,17 +154,11 @@ For full details on searching documents using Azure AI Search, you can read [Sea
 
 This sample shows how to set up query using a POST request.
 
-Issue the HTTP POST request:
+Issue the HTTP POST request, specifying the filter in the request body:
 
 ```http
-POST https://[service name].search.windows.net/indexes/securedfiles/docs/search?api-version=2020-06-30
-Content-Type: application/json  
-api-key: [admin or query key]
-```
+POST https://[service name].search.windows.net/indexes/securedfiles/docs/search?api-version=2023-11-01
 
-Specify the filter in the request body:
-
-```JSON
 {
    "filter":"group_ids/any(g:search.in(g, 'group_id1, group_id2'))"  
 }
@@ -186,7 +187,8 @@ You should get the documents back where `group_ids` contains either "group_id1"
 
 This article describes a pattern for filtering results based on user identity and the `search.in()` function. You can use this function to pass in principal identifiers for the requesting user to match against principal identifiers associated with each target document. When a search request is handled, the `search.in` function filters out search results for which none of the user's principals have read access. The principal identifiers can represent things like security groups, roles, or even the user's own identity.
 
-For an alternative pattern based on Microsoft Entra ID, or to revisit other security features, see the following links.
+For more examples, demos, and videos:
 
-* [Security filters for trimming results using Microsoft Entra ID identities](search-security-trimming-for-azure-search-with-aad.md)
-* [Security in Azure AI Search](search-security-overview.md)
+* [Get started with chat document security in Python](/azure/developer/python/get-started-app-chat-document-security-trim)
+* [Set up optional sign in and document level access control (modifications to the AzureOpenAIDemo app)](https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/docs/login_and_acl.md)
+* [Video: Secure your Intelligent Applications with Microsoft Entra](https://build.microsoft.com/en-US/sessions/b5636ca7-64c2-493c-9b30-4a35852acfbe?source=/speakers/cc9b56a0-4af0-4b60-a2f3-8312c5b35ca2)