Merge pull request #188508 from HeidiSteen/heidist-fresh

[azure search] Cosmos DB topic updates

Author: LJSpiller

articles/search/TOC.yml

@@ -355,8 +355,10 @@
           href: search-howto-index-changed-deleted-blobs.md
       - name: Azure Cosmos DB
         items:
-        - name: SQL and MongoDB APIs
+        - name: SQL API
           href: search-howto-index-cosmosdb.md
+        - name: MongoDB API
+          href: search-howto-index-cosmosdb-mongodb.md
         - name: Gremlin API
           href: search-howto-index-cosmosdb-gremlin.md
       - name: Azure DB for MySQL

articles/search/search-howto-index-azure-data-lake-storage.md

@@ -127,7 +127,7 @@ In a [search index](search-what-is-an-index.md), add fields to accept the conten
 
    + A custom metadata property that you add to blobs. This option requires that your blob upload process adds that metadata property to all blobs. Since the key is a required property, any blobs that are missing a value will fail to be indexed. If you use a custom metadata property as a key, avoid making changes to that property. Indexers will add duplicate documents for the same blob if the key property changes.
 
-    Metadata properties often include characters, such as `/` and `-`, that are invalid for document keys. Because the indexer has a "base64EncodeKeys" property (true by default), it automatically encodes the metadata property, with no configuration or field mapping required.
+   Metadata properties often include characters, such as `/` and `-`, that are invalid for document keys. Because the indexer has a "base64EncodeKeys" property (true by default), it automatically encodes the metadata property, with no configuration or field mapping required.
 
 1. Add a "content" field to store extracted text from each file through the blob's "content" property. You aren't required to use this name, but doing so lets you take advantage of implicit field mappings. 
 

articles/search/search-howto-index-cosmosdb-gremlin.md

@@ -0,0 +1,236 @@
+---
+title: Azure Cosmos DB MongoDB indexer
+titleSuffix: Azure Cognitive Search
+description: Set up a search indexer to index data stored in Azure Cosmos DB for full text search in Azure Cognitive Search. This article explains how index data using the MongoDB API protocol.
+
+author: mgottein 
+ms.author: magottei
+ms.service: cognitive-search
+ms.topic: how-to
+ms.date: 02/15/2022
+---
+
+# Index data from Azure Cosmos DB using the MongoDB API
+
+> [!IMPORTANT] 
+> MongoDB API support is currently in public preview under [supplemental Terms of Use](https://azure.microsoft.com/support/legal/preview-supplemental-terms/). Currently, there is no SDK support.
+
+This article shows you how to configure an Azure Cosmos DB [indexer](search-indexer-overview.md) to extract content and make it searchable in Azure Cognitive Search. This workflow creates an Azure Cognitive Search index and loads it with existing text extracted from Azure Cosmos DB using the [MongoDB API](../cosmos-db/choose-api.md#api-for-mongodb).
+
+Because terminology can be confusing, it's worth noting that [Azure Cosmos DB indexing](../cosmos-db/index-overview.md) and [Azure Cognitive Search indexing](search-what-is-an-index.md) are different operations. Indexing in Cognitive Search creates and loads a search index on your search service.
+
+Although Cosmos DB indexing is easiest with the [Import data wizard](search-import-data-portal.md), this article uses the REST APIs to explain concepts and steps. 
+
+## Prerequisites
+
++ [Register for the preview](https://aka.ms/azure-cognitive-search/indexer-preview) to provide feedback and get help with any issues you encounter.
+
++ An [Azure Cosmos DB account, database, collection, and documents](../cosmos-db/sql/create-cosmosdb-resources-portal.md). Use the same region for both Cognitive Search and Cosmos DB for lower latency and to avoid bandwidth charges.
+
++ An [automatic indexing policy](../cosmos-db/index-policy.md) on the Cosmos DB collection, set to [Consistent](../cosmos-db/index-policy.md#indexing-mode). This is the default configuration. Lazy indexing isn't recommended and may result in missing data.
+
+Unfamiliar with indexers? Start with [**Create an indexer**](search-howto-create-indexers.md) for more background.
+
+## Define the data source
+
+The data source definition specifies the data to index, credentials, and policies for identifying changes in the data. A data source is defined as an independent resource so that it can be used by multiple indexers.
+
+For this call, specify a [preview REST API version](search-api-preview.md) (2020-06-30-Preview or 2021-04-30-Preview) to create a data source that connects using MongoDB API.
+
+1. [Create or update a data source](/rest/api/searchservice/preview-api/create-or-update-data-source) to set its definition: 
+
+    ```http
+    POST https://[service name].search.windows.net/datasources?api-version=2021-04-30-Preview
+    Content-Type: application/json
+    api-key: [Search service admin key]
+    {
+      "name": "[my-cosmosdb-mongodb-ds]",
+      "type": "cosmosdb",
+      "credentials": {
+        "connectionString": "AccountEndpoint=https://[cosmos-account-name].documents.azure.com;AccountKey=[cosmos-account-key];Database=[cosmos-database-name];ApiKind=MongoDb;"
+      },
+      "container": {
+        "name": "[cosmos-db-collection]",
+        "query": null
+      },
+      "dataChangeDetectionPolicy": {
+        "@odata.type": "#Microsoft.Azure.Search.HighWaterMarkChangeDetectionPolicy",
+        "highWaterMarkColumnName": "_ts"
+      },
+      "dataDeletionDetectionPolicy": null,
+      "encryptionKey": null,
+      "identity": null
+    }
+    ```
+
+1. Set "type" to `"cosmosdb"` (required).
+
+1. Set "credentials" to a connection string. The next section describes the supported formats.
+
+1. Set "container" to the collection. The "name" property is required and it specifies the ID of the database collection to be indexed. For the MongoDB API, "query" isn't supported. 
+
+1. [Set "dataChangeDetectionPolicy"](#DataChangeDetectionPolicy) if data is volatile and you want the indexer to pick up just the new and updated items on subsequent runs.
+
+1. [Set "dataDeletionDetectionPolicy"](#DataDeletionDetectionPolicy) if you want to remove search documents from a search index when the source item is deleted.
+
+<a name="credentials"></a>
+
+### Supported credentials and connection strings
+
+Indexers can connect to a collection using the following connections. For connections that target the [MongoDB API](../cosmos-db/mongodb/mongodb-introduction.md), be sure to include "ApiKind" in the connection string.
+
+Avoid port numbers in the endpoint URL. If you include the port number, the connection will fail.  
+
+| Full access connection string |
+|-----------------------------------------------|
+|`{ "connectionString" : "AccountEndpoint=https://<Cosmos DB account name>.documents.azure.com;AccountKey=<Cosmos DB auth key>;Database=<Cosmos DB database id>;ApiKind=MongoDb" }` |
+| You can get the connection string from the Cosmos DB account page in Azure portal by selecting **Keys** in the left navigation pane. Make sure to select a full connection string and not just a key.  |
+
+| Managed identity connection string |
+|------------------------------------|
+|`{ "connectionString" : "ResourceId=/subscriptions/<your subscription ID>/resourceGroups/<your resource group name>/providers/Microsoft.DocumentDB/databaseAccounts/<your cosmos db account name>/;(ApiKind=[api-kind];)" }`|
+|This connection string doesn't require an account key, but you must have previously configured a search service to [connect using a managed identity](search-howto-managed-identities-data-sources.md) and created a role assignment that grants **Cosmos DB Account Reader Role** permissions. See [Setting up an indexer connection to a Cosmos DB database using a managed identity](search-howto-managed-identities-cosmos-db.md) for more information. |
+
+## Add search fields to an index
+
+In a [search index](search-what-is-an-index.md), add fields to accept the source JSON documents or the output of your custom query projection. Ensure that the search index schema is compatible with source data. For content in Cosmos DB, your search index schema should correspond to the [Azure Cosmos DB items](../cosmos-db/account-databases-containers-items.md#azure-cosmos-items) in your data source.
+
+1. [Create or update an index](/rest/api/searchservice/create-index) to define search fields that will store data:
+
+    ```http
+    POST https://[service name].search.windows.net/indexes?api-version=2020-06-30
+    Content-Type: application/json
+    api-key: [Search service admin key]
+    
+    {
+        "name": "mysearchindex",
+        "fields": [{
+            "name": "id",
+            "type": "Edm.String",
+            "key": true,
+            "retrievable": true,
+            "searchable": false
+        }, {
+            "name": "description",
+            "type": "Edm.String",
+            "filterable": false,
+            "searchable": true,
+            "sortable": false,
+            "facetable": false,
+            "suggestions": true
+        }]
+    }
+    ```
+
+1. Create a document key field ("key": true). For MongoDB collections, Azure Cognitive Search automatically renames the `_id` property to `id` because field names can’t start with an underscore character. If `_id` contains characters that are invalid for search document keys, the `id` values are Base64 encoded.
+
+1. Create additional fields for more searchable content. See [Create an index](search-how-to-create-search-index.md) for details.
+
+### Mapping between JSON Data Types and Azure Cognitive Search Data Types
+
+| JSON data type | Compatible target index field types |
+| --- | --- |
+| Bool |Edm.Boolean, Edm.String |
+| Numbers that look like integers |Edm.Int32, Edm.Int64, Edm.String |
+| Numbers that look like floating-points |Edm.Double, Edm.String |
+| String |Edm.String |
+| Arrays of primitive types such as ["a", "b", "c"] |Collection(Edm.String) |
+| Strings that look like dates |Edm.DateTimeOffset, Edm.String |
+| GeoJSON objects such as { "type": "Point", "coordinates": [long, lat] } |Edm.GeographyPoint |
+| Other JSON objects |N/A |
+
+## Configure and run the Cosmos DB indexer
+
+Indexer configuration specifies the inputs, parameters, and properties controlling run time behaviors.
+
+1. [Create or update an indexer](/rest/api/searchservice/create-indexer) to use the predefined data source and search index.
+
+    ```http
+    POST https://[service name].search.windows.net/indexers?api-version=2020-06-30
+    Content-Type: application/json
+    api-key: [search service admin key]
+    {
+        "name" : "[my-cosmosdb-indexer]",
+        "dataSourceName" : "[my-cosmosdb-mongodb-ds]",
+        "targetIndexName" : "[my-search-index]",
+        "disabled": null,
+        "schedule": null,
+        "parameters": {
+        "batchSize": null,
+        "maxFailedItems": 0,
+        "maxFailedItemsPerBatch": 0,
+        "base64EncodeKeys": false,
+        "configuration": {}
+        },
+        "fieldMappings": [],
+        "encryptionKey": null
+    }
+    ```
+
+1. [Specify field mappings](search-indexer-field-mappings.md) if there are differences in field name or type, or if you need multiple versions of a source field in the search index.
+
+1. See [Create an indexer](search-howto-create-indexers.md) for more information about other properties.
+
+<a name="DataChangeDetectionPolicy"></a>
+
+## Indexing changed documents
+
+The purpose of a data change detection policy is to efficiently identify changed data items. Currently, the only supported policy is the [`HighWaterMarkChangeDetectionPolicy`](/dotnet/api/azure.search.documents.indexes.models.highwatermarkchangedetectionpolicy) using the `_ts` (timestamp) property provided by Azure Cosmos DB, which is specified in the data source definition as follows:
+
+```http
+"dataChangeDetectionPolicy": {
+    "@odata.type": "#Microsoft.Azure.Search.HighWaterMarkChangeDetectionPolicy",
+"  highWaterMarkColumnName": "_ts"
+},
+```
+
+Using this policy is highly recommended to ensure good indexer performance. 
+
+<a name="DataDeletionDetectionPolicy"></a>
+
+## Indexing deleted documents
+
+When rows are deleted from the collection, you normally want to delete those rows from the search index as well. The purpose of a data deletion detection policy is to efficiently identify deleted data items. Currently, the only supported policy is the `Soft Delete` policy (deletion is marked with a flag of some sort), which is specified in the data source definition as follows:
+
+```http
+"dataDeletionDetectionPolicy"": {
+    "@odata.type" : "#Microsoft.Azure.Search.SoftDeleteColumnDeletionDetectionPolicy",
+    "softDeleteColumnName" : "the property that specifies whether a document was deleted",
+    "softDeleteMarkerValue" : "the value that identifies a document as deleted"
+}
+```
+
+If you're using a custom query, make sure that the property referenced by `softDeleteColumnName` is projected by the query.
+
+The following example creates a data source with a soft-deletion policy:
+
+```http
+POST https://[service name].search.windows.net/datasources?api-version=2020-06-30
+Content-Type: application/json
+api-key: [Search service admin key]
+
+{
+    "name": ["my-cosmosdb-mongodb-ds]",
+    "type": "cosmosdb",
+    "credentials": {
+        "connectionString": "AccountEndpoint=https://[cosmos-account-name].documents.azure.com;AccountKey=[cosmos-account-key];Database=[cosmos-database-name];ApiKind=MongoDB"
+    },
+    "container": { "name": "[my-cosmos-collection]" },
+    "dataChangeDetectionPolicy": {
+        "@odata.type": "#Microsoft.Azure.Search.HighWaterMarkChangeDetectionPolicy",
+        "highWaterMarkColumnName": "_ts"
+    },
+    "dataDeletionDetectionPolicy": {
+        "@odata.type": "#Microsoft.Azure.Search.SoftDeleteColumnDeletionDetectionPolicy",
+        "softDeleteColumnName": "isDeleted",
+        "softDeleteMarkerValue": "true"
+    }
+}
+```
+
+## Next steps
+
+You can now control how you [run the indexer](search-howto-run-reset-indexers.md), [monitor status](search-howto-monitor-indexers.md), or [schedule indexer execution](search-howto-schedule-indexers.md). The following articles apply to indexers that pull content from Azure Cosmos DB:
+
++ [Set up an indexer connection to a Cosmos DB database using a managed identity](search-howto-managed-identities-cosmos-db.md)
++ [Index large data sets](search-howto-large-index.md)