Merge pull request #104714 from HeidiSteen/heidist-monitor

[Azure Cognitive Search] logging updates

Author: megvanhuygen

articles/search/TOC.yml

@@ -323,10 +323,10 @@
     items:
     - name: Fundamentals
       href: search-monitor-usage.md
-    - name: Monitor query activity
-      href: search-monitor-queries.md 
     - name: Diagnostic logging
       href: search-monitor-logs.md   
+    - name: Monitor query activity
+      href: search-monitor-queries.md 
     - name: Search traffic analytics
       href: search-traffic-analytics.md
   - name: Troubleshoot

articles/search/media/search-monitor-usage/azurediagnostics-table-sortby-duration.png

@@ -13,9 +13,11 @@ ms.date: 02/14/2020
 
 # Adjust capacity in Azure Cognitive Search
 
-Before [provisioning a search service](search-create-service-portal.md) and locking in a specific pricing tier, take a few minutes to understand the role of replicas and partitions in a service, whether you need proportionally larger or faster partitions, and how you might configure the service for expected load.
+Before [provisioning a search service](search-create-service-portal.md) and locking in a specific pricing tier, take a few minutes to understand the role of replicas and partitions in a service and how you might adjust a service to accommodate spikes and dips in resource demand.
 
-Capacity is a function of the [tier you choose](search-sku-tier.md) (tiers determine hardware characteristics), and the replica and partition combination necessary for projected workloads. This article focuses on replica and partition combinations and interactions.
+Capacity is a function of the [tier you choose](search-sku-tier.md) (tiers determine hardware characteristics), and the replica and partition combination necessary for projected workloads. Depending on the tier and the size of the adjustment, adding or reducing capacity can take anywhere from 15 minutes to several hours. 
+
+When modifying the allocation of replicas and partitions, we recommend using the Azure portal. The portal enforces limits on allowable combinations that stay below maximum limits of a tier. However, if you require a script-based or code-based provisioning approach, the [Azure PowerShell](search-manage-powershell.md) or the [Management REST API](https://docs.microsoft.com/rest/api/searchmanagement/services) are alternative solutions.
 
 ## Terminology: replicas and partitions
 
@@ -24,16 +26,18 @@ Capacity is a function of the [tier you choose](search-sku-tier.md) (tiers deter
 |*Partitions* | Provides index storage and I/O for read/write operations (for example, when rebuilding or refreshing an index). Each partition has a share of the total index. If you allocate three partitions, your index is divided into thirds. |
 |*Replicas* | Instances of the search service, used primarily to load balance query operations. Each replica is one copy of an index. If you allocate three replicas, you'll have three copies of an index available for servicing query requests.|
 
-## How to allocate replicas and partitions
+## When to add nodes
 
 Initially, a service is allocated a minimal level of resources consisting of one partition and one replica. 
 
-A single service must have sufficient resources to handle all workloads (indexing and queries). Neither workload runs in the background. You can schedule indexing for times when query requests are naturally less frequent, but the service will not otherwise prioritize one task over another.
-
-When modifying the allocation of replicas and partitions, we recommend using the Azure portal. The portal enforces limits on allowable combinations that stay below maximum limits of a tier. However, if you require a script-based or code-based provisioning approach, the [Azure PowerShell](search-manage-powershell.md) or the [Management REST API](https://docs.microsoft.com/rest/api/searchmanagement/services) are alternative solutions.
+A single service must have sufficient resources to handle all workloads (indexing and queries). Neither workload runs in the background. You can schedule indexing for times when query requests are naturally less frequent, but the service will not otherwise prioritize one task over another. Additionally, a certain amount of redundancy smooths out query performance when services or nodes are updated internally.
 
 As a general rule, search applications tend to need more replicas than partitions, particularly when the service operations are biased toward query workloads. The section on [high availability](#HA) explains why.
 
+Adding more replicas or partitions increases your cost of running the service. Be sure to check the [pricing calculator](https://azure.microsoft.com/pricing/calculator/) to understand the billing implications of adding more nodes. The [chart below](#chart) can help you cross-reference the number of search units required for a specific configuration.
+
+## How to allocate replicas and partitions
+
 1. Sign in to the [Azure portal](https://portal.azure.com/) and select the search service.
 
 1. In **Settings**, open the **Scale** page to modify replicas and partitions. 
@@ -110,15 +114,15 @@ Currently, there is no built-in mechanism for disaster recovery. Adding partitio
 
 ## Estimate replicas
 
-On a production service, you should allocate three replicas for SLA purposes. If you experience slow query performance, one remedy is to add replicas so that additional copies of the index are brought online to support bigger query workloads and to load balance the requests over the multiple replicas.
+On a production service, you should allocate three replicas for SLA purposes. If you experience slow query performance, you can add replicas so that additional copies of the index are brought online to support bigger query workloads and to load balance the requests over the multiple replicas.
 
 We do not provide guidelines on how many replicas are needed to accommodate query loads. Query performance depends on the complexity of the query and competing workloads. Although adding replicas clearly results in better performance, the result is not strictly linear: adding three replicas does not guarantee triple throughput.
 
 For guidance in estimating QPS for your solution, see [Scale for performance](search-performance-optimization.md)and [Monitor queries](search-monitor-queries.md)
 
 ## Estimate partitions
 
-The [tier you choose](search-sku-tier.md) determines partition size and speed, and each tier is optimized around a set of characteristics that fit various scenarios. If you choose a higher-end tier, you might need fewer partitions than if you go with S1.
+The [tier you choose](search-sku-tier.md) determines partition size and speed, and each tier is optimized around a set of characteristics that fit various scenarios. If you choose a higher-end tier, you might need fewer partitions than if you go with S1. One of the questions you'll need to answer through self-directed testing is whether a larger and more expensive partition yields better performance than two cheaper partitions on a service provisioned at a lower tier.
 
 Search applications that require near real-time data refresh will need proportionally more partitions than replicas. Adding partitions spreads read/write operations across a larger number of compute resources. It also gives you more disk space for storing additional indexes and documents.
 

articles/search/media/search-monitor-usage/azurediagnostics-table.png

@@ -8,22 +8,22 @@ author: HeidiSteen
 ms.author: heidist
 ms.service: cognitive-search
 ms.topic: conceptual
-ms.date: 02/11/2020
+ms.date: 02/18/2020
 ---
 
 # Collect and analyze log data for Azure Cognitive Search
 
-Diagnostic or operational logs provide insight into the detailed operations of Azure Cognitive Search and are useful for monitoring service and workload processes. Internally, logs exist on the backend for a short period of time, sufficient for investigation and analysis if you file a support ticket. However, if you want self-direction over operational data, you should configure a diagnostic setting to specify where logging information is collected. 
+Diagnostic or operational logs provide insight into the detailed operations of Azure Cognitive Search and are useful for monitoring service and workload processes. Internally, logs exist on the backend for a short period of time, sufficient for investigation and analysis if you file a support ticket. However, if you want self-direction over operational data, you should configure a diagnostic setting to specify where logging information is collected.
 
 Setting up logs is useful for diagnostics and preserving operational history. After you enable logging, you can run queries or build reports for structured analysis.
 
 The following table enumerates options for collecting and persisting data.
 
 | Resource | Used for |
 |----------|----------|
-| [Send to Log Analytics workspace](https://docs.microsoft.com/azure/azure-monitor/learn/tutorial-resource-logs) | Logged events and query metrics, based on the schemas below. Events are logged to a Log Analytics workspace. Using Log Analytics, you can run queries to return detailed information. For more information, see [Get started with Azure Monitor logs](https://docs.microsoft.com/azure/azure-monitor/learn/tutorial-viewdata) |
-| [Archive with Blob storage](https://docs.microsoft.com/azure/storage/blobs/storage-blobs-overview) | Logged events and query metrics, based on the schemas below. Events are logged to a Blob container and stored in JSON files. Logs can be quite granular (by the hour/minute), useful for researching a specific incident but not for open-ended investigation. Use a JSON editor to view a log file.|
-| [Stream to Event Hub](https://docs.microsoft.com/azure/event-hubs/) | Logged events and query metrics, based on the schemas documented in this article. Choose this as an alternative data collection service for very large logs. |
+| [Send to Log Analytics workspace](https://docs.microsoft.com/azure/azure-monitor/learn/tutorial-resource-logs) | Events and metrics are sent to a Log Analytics workspace, which can be queried in the portal to return detailed information. For an introduction, see [Get started with Azure Monitor logs](https://docs.microsoft.com/azure/azure-monitor/learn/tutorial-viewdata) |
+| [Archive with Blob storage](https://docs.microsoft.com/azure/storage/blobs/storage-blobs-overview) | Events and metrics are archived to a Blob container and stored in JSON files. Logs can be quite granular (by the hour/minute), useful for researching a specific incident but not for open-ended investigation. Use a JSON editor to view a raw log file or Power BI to aggregate and visualize log data.|
+| [Stream to Event Hub](https://docs.microsoft.com/azure/event-hubs/) | Events and metrics are streamed to an Azure Event Hubs service. Choose this as an alternative data collection service for very large logs. |
 
 Both Azure Monitor logs and Blob storage are available as a free service so that you can try it out at no charge for the lifetime of your Azure subscription. Application Insights is free to sign up and use as long as application data size is under certain limits (see the [pricing page](https://azure.microsoft.com/pricing/details/monitor/) for details).
 
@@ -33,38 +33,59 @@ If you are using Log Analytics or Azure Storage, you can create resources in adv
 
 + [Create a log analytics workspace](https://docs.microsoft.com/azure/azure-monitor/learn/quick-create-workspace)
 
-+ [Create a storage account](https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account) if you require a log archive.
++ [Create a storage account](https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account)
 
-## Create a log
+## Enable data collection
 
-Diagnostic settings define data collection. A setting specifies how and what is collected. 
+Diagnostic settings specify how logged events and metrics are collected.
 
 1. Under **Monitoring**, select **Diagnostic settings**.
 
    ![Diagnostic settings](./media/search-monitor-usage/diagnostic-settings.png "Diagnostic settings")
 
 1. Select **+ Add diagnostic setting**
 
-1. Choose the data you want to export: Logs, Metrics or both. You can collect data in a storage account, a log analytics workspace, or stream it to Event Hub.
-
-   Log analytics is recommended because you can query the workspace in the portal.
-
-   If you are also using Blob storage, containers and blobs will be created as-needed when log data is exported.
+1. Check **Log Analytics**, select your workspace, and select **OperationLogs** and **AllMetrics**.
 
    ![Configure data collection](./media/search-monitor-usage/configure-storage.png "Configure data collection")
 
 1. Save the setting.
 
-1. Test by creating or deleting objects (creates log events) and by submitting queries (generates metrics). 
+1. After logging has been enabled, use your search service to start generating logs and metrics. It will take time before logged events and metrics become available.
+
+For Log Analytics, it will be several minutes before data is available, after which you can run Kusto queries to return data. For more information, see [Monitor query requests](search-monitor-logs.md).
+
+For Blob storage, it takes one hour before the containers will appear in Blob storage. There is one blob, per hour, per container. Containers are only created when there is an activity to log or measure. When the data is copied to a storage account, the data is formatted as JSON and placed in two containers:
+
++ insights-logs-operationlogs: for search traffic logs
++ insights-metrics-pt1m: for metrics
+
+## Query log information
+
+In diagnostic logs, two tables contain logs and metrics for Azure Cognitive Search: **AzureDiagnostics** and **AzureMetrics**.
+
+1. Under **Monitoring**, select **Logs**.
+
+1. Enter **AzureMetrics** in the query window. Run this simple query to get acquainted with the data collected in this table. Scroll across the table to view metrics and values. Notice the record count at the top, and if your service has been collecting metrics for a while, you might want to adjust the time interval to get a manageable data set.
+
+   ![AzureMetrics table](./media/search-monitor-usage/azuremetrics-table.png "AzureMetrics table")
+
+1. Enter the following query to return a tabular result set.
 
-In Blob storage, containers are only created when there is an activity to log or measure. When the data is copied to a storage account, the data is formatted as JSON and placed in two containers:
+   ```
+   AzureMetrics
+    | project MetricName, Total, Count, Maximum, Minimum, Average
+   ```
 
-* insights-logs-operationlogs: for search traffic logs
-* insights-metrics-pt1m: for metrics
+1. Repeat the previous steps, starting with **AzureDiagnostics** to return all columns for informational purposes, followed by a more selective query that extracts more interesting information.
 
-**It takes one hour before the containers will appear in Blob storage. There is one blob, per hour, per container.**
+   ```
+   AzureDiagnostics
+   | project OperationName, resultSignature_d, DurationMs, Query_s, Documents_d, IndexName_s
+   | where OperationName == "Query.Search" 
+   ```
 
-Logs are archived for every hour in which activity occurs. The following path is an example of one log file created on January 12 2020 at 9:00 a.m. where each `/` is a folder: `resourceId=/subscriptions/<subscriptionID>/resourcegroups/<resourceGroupName>/providers/microsoft.search/searchservices/<searchServiceName>/y=2020/m=01/d=12/h=09/m=00/name=PT1H.json`
+   ![AzureDiagnostics table](./media/search-monitor-usage/azurediagnostics-table.png "AzureDiagnostics table")
 
 ## Log schema
 
@@ -119,7 +140,7 @@ For the **Search Queries Per Second** metric, minimum is the lowest value for se
 
 For **Throttled Search Queries Percentage**, minimum, maximum, average and total, all have the same value: the percentage of search queries that were throttled, from the total number of search queries during one minute.
 
-## View log files
+## View raw log files
 
 Blob storage is used for archiving log files. You can use any JSON editor to view the log file. If you don't have one, we recommend [Visual Studio Code](https://code.visualstudio.com/download).