Merge pull request #100645 from arv100kri/arjagann/ip-whitelisting

Add documentation about IP range restriction, NSG rules and service tag

Author: tfosmark

articles/search/search-howto-connecting-azure-sql-iaas-to-azure-search-using-indexers.md

@@ -68,8 +68,12 @@ The links below provide instructions on NSG configuration for VM deployments. Us
 
 IP addressing can pose a few challenges that are easily overcome if you are aware of the issue and potential workarounds. The remaining sections provide recommendations for handling issues related to IP addresses in the ACL.
 
-#### Restrict access to the search service IP address
-We strongly recommend that you restrict the access to the IP address of your search service in the ACL instead of making your SQL Azure VMs wide open to any connection requests. You can easily find out the IP address by pinging the FQDN (for example, `<your-search-service-name>.search.windows.net`) of your search service.
+#### Restrict access to the Azure Cognitive Search
+We strongly recommend that you restrict the access to the IP address of your search service and the IP address range of `AzureCognitiveSearch` [service tag](https://docs.microsoft.com/azure/virtual-network/service-tags-overview#available-service-tags) in the ACL instead of making your SQL Azure VMs open to all connection requests.
+
+You can find out the IP address by pinging the FQDN (for example, `<your-search-service-name>.search.windows.net`) of your search service.
+
+You can find out the IP address range of `AzureCognitiveSearch` [service tag](https://docs.microsoft.com/azure/virtual-network/service-tags-overview#available-service-tags) for the particular region in which your Azure Cognitive Search service is located by either using [Downloadable JSON files](https://docs.microsoft.com/azure/virtual-network/service-tags-overview#discover-service-tags-by-using-downloadable-json-files) or via the [Service Tag Discovery API](https://docs.microsoft.com/azure/virtual-network/service-tags-overview#use-the-service-tag-discovery-api-public-preview). The IP address range is updated weekly.
 
 #### Managing IP address fluctuations
 If your search service has only one search unit (that is, one replica and one partition), the IP address will change during routine service restarts, invalidating an existing ACL with your search service's IP address.

articles/search/search-howto-connecting-azure-sql-mi-to-azure-search-using-indexers.md

@@ -30,6 +30,13 @@ Check the Network Security Group has the correct **Inbound security rules** that
 
    ![NSG Inbound security rule](media/search-howto-connecting-azure-sql-mi-to-azure-search-using-indexers/nsg-rule.png "NSG Inbound security rule")
 
+> [!NOTE]
+> You can choose to be more restrictive in the inbound access to your managed SQL instance by replacing the current rule (`public_endpoint_inbound`) with 2 rules:
+>
+> * Allowing inbound access from the `AzureCognitiveSearch` [service tag](https://docs.microsoft.com/azure/virtual-network/service-tags-overview#available-service-tags) ("SOURCE" = `AzureCognitiveSearch`)
+>
+> * Allowing inbound access from the IP address of the search service, which can be obtained by pinging its fully qualified domain name (eg., `<your-search-service-name>.search.windows.net`). ("SOURCE" = `IP address`)
+
 ## Get public endpoint connection string
 Make sure you use the connection string for the **public endpoint** (port 3342, not port 1433).
 

articles/search/search-indexer-troubleshooting.md

@@ -15,33 +15,57 @@ ms.date: 11/04/2019
 
 Indexers can run into a number of issues when indexing data into Azure Cognitive Search. The main categories of failure include:
 
-* [Connecting to a data source](#data-source-connection-errors)
+* [Connecting to a data source or other resources](#connection-errors)
 * [Document processing](#document-processing-errors)
 * [Document ingestion to an index](#index-errors)
 
-## Data Source Connection Errors
+## Connection errors
 
-### Blob Storage
+> [!NOTE]
+> Indexers have limited support for accessing data sources and other resources that are secured by Azure network security mechanisms. Currently, indexers can only access data sources via corresponding IP address range restriction mechanisms or NSG rules when applicable. Details for accessing each supported data source can be found below.
+>
+> You can find out the IP address of your search service by pinging its fully qualified domain name (eg., `<your-search-service-name>.search.windows.net`).
+>
+> You can find out the IP address range of `AzureCognitiveSearch` [service tag](https://docs.microsoft.com/azure/virtual-network/service-tags-overview#available-service-tags) for the particular region in which your Azure Cognitive Search service is present by either using [Downloadable JSON files](https://docs.microsoft.com/azure/virtual-network/service-tags-overview#discover-service-tags-by-using-downloadable-json-files) or via the [Service Tag Discovery API](https://docs.microsoft.com/azure/virtual-network/service-tags-overview#use-the-service-tag-discovery-api-public-preview). The IP address range is updated weekly.
 
-#### Storage account firewall
+### Configure firewall rules
 
-Azure Storage provides a configurable firewall. By default, the firewall is disabled so Azure Cognitive Search can connect to your storage account.
+Azure Storage, CosmosDB and Azure SQL provide a configurable firewall. There's no specific error message when the firewall is enabled. Typically, firewall errors are generic and look like `The remote server returned an error: (403) Forbidden` or `Credentials provided in the connection string are invalid or have expired`.
 
-There's no specific error message when the firewall is enabled. Typically, firewall errors look like `The remote server returned an error: (403) Forbidden`.
+There are 2 options for allowing indexers to access these resources in such an instance:
 
-You can verify that the firewall is enabled in the [portal](https://docs.microsoft.com/azure/storage/common/storage-network-security#azure-portal). The only supported workaround is to disable the firewall by choosing to allow access from ['All networks'](https://docs.microsoft.com/azure/storage/common/storage-network-security#azure-portal).
+* Disable the firewall, by allowing access from **All Networks** (if feasible).
+* Alternatively, you can allow access for the IP address of your search service and the IP address range of `AzureCognitiveSearch` [service tag](https://docs.microsoft.com/azure/virtual-network/service-tags-overview#available-service-tags) in the firewall rules of your resource (IP address range restriction).
 
-If your indexer does not have an attached skillset, you _may_ attempt to [add an exception](https://docs.microsoft.com/azure/storage/common/storage-network-security#managing-ip-network-rules) for the IP addresses of your search service. However, this scenario is not supported and is not guaranteed to work.
+Details for configuring IP address range restrictions for each data source type can be found from the following links:
 
-You can find out the IP address of your search service by pinging its FQDN (`<your-search-service-name>.search.windows.net`).
+* [Azure Storage](https://docs.microsoft.com/azure/storage/common/storage-network-security#grant-access-from-an-internet-ip-range)
 
-### Cosmos DB
+* [Cosmos DB](https://docs.microsoft.com/azure/storage/common/storage-network-security#grant-access-from-an-internet-ip-range)
 
-#### Indexing isn't enabled
+* [Azure SQL](https://docs.microsoft.com/azure/sql-database/sql-database-firewall-configure#create-and-manage-ip-firewall-rules)
+
+**Limitation**: As stated in the documentation above for Azure Storage, IP address range restrictions will only work if your search service and your storage account are in different regions.
+
+Azure functions (that could be used as a [Custom Web Api skill](cognitive-search-custom-skill-web-api.md)) also support [IP address restrictions](https://docs.microsoft.com/azure/azure-functions/ip-addresses#ip-address-restrictions). The list of IP addresses to configure would be the IP address of your search service and the IP address range of `AzureCognitiveSearch` service tag.
+
+Details for accessing data in SQL server on an Azure VM are outlined [here](search-howto-connecting-azure-sql-iaas-to-azure-search-using-indexers.md)
+
+### Configure network security group (NSG) rules
+
+When accessing data in a SQL managed instance, or when an Azure VM is used as the web service URI for a [Custom Web Api skill](cognitive-search-custom-skill-web-api.md), customers need not be concerned with specific IP addresses.
+
+In such cases, the Azure VM, or the SQL managed instance can be configured to reside within a virtual network. Then a network security group can be configured to filter the type of network traffic that can flow in and out of the virtual network subnets and network interfaces.
+
+The `AzureCognitiveSearch` service tag can be directly used in the inbound [NSG rules](https://docs.microsoft.com/azure/virtual-network/manage-network-security-group#work-with-security-rules) without needing to look up its IP address range.
+
+More details for accessing data in a SQL managed instance are outlined [here](search-howto-connecting-azure-sql-mi-to-azure-search-using-indexers.md)
+
+### CosmosDB "Indexing" isn't enabled
 
 Azure Cognitive Search has an implicit dependency on Cosmos DB indexing. If you turn off automatic indexing in Cosmos DB, Azure Cognitive Search returns a successful state, but fails to index container contents. For instructions on how to check settings and turn on indexing, see [Manage indexing in Azure Cosmos DB](https://docs.microsoft.com/azure/cosmos-db/how-to-manage-indexing-policy#use-the-azure-portal).
 
-## Document Processing Errors
+## Document processing errors
 
 ### Unprocessable or unsupported documents
 
@@ -76,7 +100,7 @@ api-key: [admin key]
 }
 ```
 
-## Index Errors
+## Index errors
 
 ### Missing documents