Merge pull request #213785 from bwren/cost-management

Cost optimization rewrite with WAF

Author: jborsecnik

articles/azure-monitor/agents/data-collection-rule-azure-monitor-agent.md

@@ -34,7 +34,6 @@ To send data to Log Analytics, create the data collection rule in the *same regi
 1. Enter a **Rule name** and specify a **Subscription**, **Resource Group**, **Region**, and **Platform Type**:
 
     - **Region** specifies where the DCR will be created. The virtual machines and their associations can be in any subscription or resource group in the tenant.
-
     - **Platform Type** specifies the type of resources this rule can apply to. The **Custom** option allows for both Windows and Linux types.
 
     [ ![Screenshot that shows the Basics tab of the Data Collection Rule screen.](media/data-collection-rule-azure-monitor-agent/data-collection-rule-basics-updated.png) ](media/data-collection-rule-azure-monitor-agent/data-collection-rule-basics-updated.png#lightbox)
@@ -107,9 +106,12 @@ This capability is enabled as part of the Azure CLI monitor-control-service exte
 For sample templates, see [Azure Resource Manager template samples for data collection rules in Azure Monitor](./resource-manager-data-collection-rules.md).
 
 ---
+
 ## Filter events using XPath queries
 
-You're charged for any data you collect in a Log Analytics workspace, so collect only the data you need. The basic configuration in the Azure portal provides you with a limited ability to filter out events.
+Since you're charged for any data you collect in a Log Analytics workspace, you should limit data collection from your agent to only the event data that you need. The basic configuration in the Azure portal provides you with a limited ability to filter out events.
+
+[!INCLUDE [azure-monitor-cost-optimization](../../../includes/azure-monitor-cost-optimization.md)]
 
 To specify more filters, use custom configuration and specify an XPath that filters out the events you don't need. XPath entries are written in the form `LogName!XPathQuery`. For example, you might want to return only events from the Application event log with an event ID of 1035. The `XPathQuery` for these events would be `*[System[EventID=1035]]`. Because you want to retrieve the events from the Application event log, the XPath is `Application!*[System[EventID=1035]]`
 
@@ -145,6 +147,7 @@ Examples of using a custom XPath to filter events:
 | Collect all Critical, Error, Warning, and Information events from the System event log except for Event ID = 6 (Driver loaded) |  `System!*[System[(Level=1 or Level=2 or Level=3) and (EventID != 6)]]` |
 | Collect all success and failure Security events except for Event ID 4624 (Successful logon) |  `Security!*[System[(band(Keywords,13510798882111488)) and (EventID != 4624)]]` |
 
+
 ## Next steps
 
 - [Collect text logs by using Azure Monitor Agent](data-collection-text-log.md).

articles/azure-monitor/agents/data-collection-rule-sample-agent.md

@@ -24,7 +24,7 @@ The sample [data collection rule](../essentials/data-collection-rule-overview.md
   - Sends all data to a Log Analytics workspace named centralWorkspace.
 
 > [!NOTE]
-> For an explanation of XPaths that are used to specify event collection in data collection rules, see [Limit data collection with custom XPath queries](../agents/data-collection-rule-azure-monitor-agent.md#filter-events-using-xpath-queries)
+> For an explanation of XPaths that are used to specify event collection in data collection rules, see [Limit data collection with custom XPath queries](../agents/data-collection-rule-azure-monitor-agent.md#filter-events-using-xpath-queries).
 
 ## Sample DCR
 

articles/azure-monitor/best-practices-cost.md

@@ -14,9 +14,10 @@ This article provides pricing guidance for Container insights to help you unders
 * Measure costs after Container insights has been enabled for one or more containers.
 * Control the collection of data and make cost reductions.
 
-Azure Monitor Logs collects, indexes, and stores data generated by your Kubernetes cluster.
+[!INCLUDE [azure-monitor-cost-optimization](../../../includes/azure-monitor-cost-optimization.md)]
 
-The Azure Monitor pricing model is primarily based on the amount of data ingested in gigabytes per day into your Log Analytics workspace. The cost of a Log Analytics workspace isn't based only on the volume of data collected. It's also dependent on the plan selected and how long you chose to store data generated from your clusters.
+
+The Azure Monitor pricing model is primarily based on the amount of data ingested in gigabytes per day into your Log Analytics workspace. The cost of a Log Analytics workspace isn't based only on the volume of data collected, it is also dependent on the plan selected, and how long you chose to store data generated from your clusters.
 
 >[!NOTE]
 >All sizes and pricing are for sample estimation only. See the Azure Monitor [pricing](https://azure.microsoft.com/pricing/details/monitor/) page for the most recent pricing based on your Azure Monitor Log Analytics pricing model and Azure region.
@@ -29,62 +30,7 @@ The following types of data collected from a Kubernetes cluster with Container i
 - Active scraping of Prometheus metrics
 - [Diagnostic log collection](../../aks/monitor-aks.md#configure-monitoring) of Kubernetes main node logs in your Azure Kubernetes Service (AKS) cluster to analyze log data generated by main components, such as `kube-apiserver` and `kube-controller-manager`.
 
-## What's collected from Kubernetes clusters?
-
-Container insights includes a predefined set of metrics and inventory items that are collected and written as log data in your Log Analytics workspace. All the metrics listed here are collected every minute.
-
-### Node metrics collected
-
-The 24 metrics per node that are collected:
-
-- cpuUsageNanoCores
-- cpuCapacityNanoCores
-- cpuAllocatableNanoCores
-- memoryRssBytes
-- memoryWorkingSetBytes
-- memoryCapacityBytes
-- memoryAllocatableBytes
-- restartTimeEpoch
-- used (disk)
-- free (disk)
-- used_percent (disk)
-- io_time (diskio)
-- writes (diskio)
-- reads (diskio)
-- write_bytes (diskio)
-- write_time (diskio)
-- iops_in_progress (diskio)
-- read_bytes (diskio)
-- read_time (diskio)
-- err_in (net)
-- err_out (net)
-- bytes_recv (net)
-- bytes_sent (net)
-- Kubelet_docker_operations (kubelet)
-
-### Container metrics
-
-The eight metrics per container that are collected:
-
-- cpuUsageNanoCores
-- cpuRequestNanoCores
-- cpuLimitNanoCores
-- memoryRssBytes
-- memoryWorkingSetBytes
-- memoryRequestBytes
-- memoryLimitBytes
-- restartTimeEpoch
-
-### Cluster inventory
-
-The cluster inventory data that's collected by default:
-
-- KubePodInventory: 1 per pod per minute
-- KubeNodeInventory: 1 per node per minute
-- KubeServices: 1 per service per minute
-- ContainerInventory: 1 per container per minute
-
-## Estimate costs to monitor your AKS cluster
+## Estimating costs to monitor your AKS cluster
 
 The following estimation is based on an AKS cluster with the following sizing example. The estimate applies only for metrics and inventory data collected. For container logs like stdout, stderr, and environmental variables, the estimate varies based on the log sizes generated by the workload. They're excluded from our estimation.
 
@@ -187,10 +133,29 @@ If you use [Prometheus metric scraping](container-insights-prometheus.md), make
 
 ### Configure Basic Logs
 
-You can save on data ingestion costs by configuring certain tables in your Log Analytics workspace that you primarily use for debugging, troubleshooting, and auditing as Basic Logs. For more information, including the limitations of Basic Logs, see [Configure Basic Logs](../best-practices-cost.md#configure-basic-logs). ContainerLogV2 is the configured version of Basic Logs that Container Insights uses. ContainerLogV2 includes verbose text-based log records.
+You can save on data ingestion costs by configuring certain tables in your Log Analytics workspace that you primarily use for debugging, troubleshooting, and auditing as Basic Logs. For more information, including the limitations of Basic Logs, see [Configure Basic Logs in Azure Monitor](../logs/basic-logs-configure.md). ContainerLogV2 is the configured version of Basic Logs that Container Insights uses. ContainerLogV2 includes verbose text-based log records.
 
 You must be on the ContainerLogV2 schema to configure Basic Logs. For more information, see [Enable the ContainerLogV2 schema (preview)](container-insights-logging-v2.md).
 
+## Data collected from Kubernetes clusters
+
+### Metric data
+Container insights includes a predefined set of metrics and inventory items collected that are written as log data in your Log Analytics workspace. All metrics in the following table are collected every one minute.
+
+
+| Type | Metrics |
+|:---|:---|
+| Node metrics | `cpuUsageNanoCores`<br>`cpuCapacityNanoCores`<br>`cpuAllocatableNanoCores`<br>`memoryRssBytes`<br>`memoryWorkingSetBytes`<br>`memoryCapacityBytes`<br>`memoryAllocatableBytes`<br>`restartTimeEpoch`<br>`used` (disk)<br>`free` (disk)<br>`used_percent` (disk)<br>`io_time` (diskio)<br>`writes` (diskio)<br>`reads` (diskio)<br>`write_bytes` (diskio)<br>`write_time` (diskio)<br>`iops_in_progress` (diskio)<br>`read_bytes` (diskio)<br>`read_time` (diskio)<br>`err_in` (net)<br>`err_out` (net)<br>`bytes_recv` (net)<br>`bytes_sent` (net)<br>`Kubelet_docker_operations` (kubelet)
+| Container metrics | `cpuUsageNanoCores`<br>`cpuRequestNanoCores`<br>`cpuLimitNanoCores`<br>`memoryRssBytes`<br>`memoryWorkingSetBytes`<br>`memoryRequestBytes`<br>`memoryLimitBytes`<br>`restartTimeEpoch`
+
+### Cluster inventory
+
+The following list is the cluster inventory data collected by default:
+
+- KubePodInventory – 1 per pod per minute
+- KubeNodeInventory – 1 per node per minute
+- KubeServices – 1 per service per minute
+- ContainerInventory – 1 per container per minute
 ## Next steps
 
 To help you understand what the costs are likely to be based on recent usage patterns from data collected with Container insights, see [Analyze usage in a Log Analytics workspace](../logs/analyze-usage.md).

articles/azure-monitor/containers/container-insights-cost.md

@@ -10,33 +10,16 @@ ms.reviwer: nikeist
 # Data collection transformations in Azure Monitor (preview)
 Transformations in Azure Monitor allow you to filter or modify incoming data before it's sent to a Log Analytics workspace. This article provides a basic description of transformations and how they are implemented. It provides links to other content for actually creating a transformation.
 
-## When to use transformations
-Transformations are useful for a variety of scenarios, including those described below. 
+## Why to use transformations
+The following table describes the different goals that transformations can be used to achieve.
 
-### Reduce data costs
-Since you're charged ingestion cost for any data sent to a Log Analytics workspace, you want to filter out any data that you don't require to reduce your costs.
-
-- **Remove entire rows.** For example, you might have a diagnostic setting to collect resource logs from a particular resource but not require all of the log entries that it generates. Create a transformation that filters out records that match a certain criteria. 
-
-- **Remove a column from each row.** For example, your data may include columns with data that's redundant or has minimal value. Create a transformation that filters out columns that aren't required.
-
-- **Parse important data from a column.** You may have a table with valuable data buried in a particular column. Use a transformation to parse the valuable data into a new column and remove the original.
-
-
-### Remove sensitive data
-You may have a data source that sends information you don't want stored for privacy or compliancy reasons.
-
-- **Filter sensitive information.** Filter out entire rows or just particular columns that contain sensitive information.
- 
-- **Obfuscate sensitive information**. For example, you might replace digits with a common character in an IP address or telephone number.
-
-
-### Enrich data with additional or calculated information
-Use a transformation to add information to data that provides business context or simplifies querying the data later.
+| Category | Details |
+|:---|:---|
+| Remove sensitive data | You may have a data source that sends information you don't want stored for privacy or compliancy reasons.<br><br>**Filter sensitive information.** Filter out entire rows or just particular columns that contain sensitive information.<br><br>**Obfuscate sensitive information**. For example, you might replace digits with a common character in an IP address or telephone number. |
+| Enrich data with additional or calculated information | Use a transformation to add information to data that provides business context or simplifies querying the data later.<br><br>**Add a column with additional information.** For example, you might add a column identifying whether an IP address in another column is internal or external.<br><br>**Add business specific information.** For example, you might add a column indicating a company division based on location information in other columns. |
+| Reduce data costs | Since you're charged ingestion cost for any data sent to a Log Analytics workspace, you want to filter out any data that you don't require to reduce your costs.<br><br>**Remove entire rows.** For example, you might have a diagnostic setting to collect resource logs from a particular resource but not require all of the log entries that it generates. Create a transformation that filters out records that match a certain criteria.<br><br>**Remove a column from each row.** For example, your data may include columns with data that's redundant or has minimal value. Create a transformation that filters out columns that aren't required.<br><br>**Parse important data from a column.** You may have a table with valuable data buried in a particular column. Use a transformation to parse the valuable data into a new column and remove the original. |
 
-- **Add a column with additional information.** For example, you might add a column identifying whether an IP address in another column is internal or external.
 
-- **Add business specific information.** For example, you might add a column indicating a company division based on location information in other columns. 
 
 ## Supported tables
 Transformations may be applied to the following tables in a Log Analytics workspace. 

articles/azure-monitor/essentials/data-collection-transformations.md

@@ -123,6 +123,19 @@ The following table provides unique requirements for each destination including
 | Event Hubs | The shared access policy for the namespace defines the permissions that the streaming mechanism has. Streaming to Event Hubs requires Manage, Send, and Listen permissions. To update the diagnostic setting to include streaming, you must have the ListKey permission on that Event Hubs authorization rule.<br><br>The event hub namespace needs to be in the same region as the resource being monitored if the resource is regional. <br><br> Diagnostic settings can't access Event Hubs resources when virtual networks are enabled. You must enable **Allow trusted Microsoft services** to bypass this firewall setting in Event Hubs so that the Azure Monitor diagnostic settings service is granted access to your Event Hubs resources.|
 | Partner integrations | The solutions vary by partner. Check the [Azure Monitor partner integrations documentation](../../partner-solutions/overview.md) for details.
 
+## Controlling costs
+
+There is a cost for collecting data in a Log Analytics workspace, so you should only collect the categories you require for each service. The data volume for resource logs varies significantly between services, 
+
+You might also not want to collect platform metrics from Azure resources because this data is already being collected in Metrics. Only configure your diagnostic data to collect metrics if you need metric data in the workspace for more complex analysis with log queries.
+
+Diagnostic settings don't allow granular filtering of resource logs. You might require certain logs in a particular category but not others. Or you may want to remove unneeded columns from the data. In these cases, use [transformations](data-collection-transformations.md) on the workspace to filter logs that you don't require. 
+
+
+You can also use transformations to lower the storage requirements for records you want by removing columns without useful information. For example, you might have error events in a resource log that you want for alerting. But you might not require certain columns in those records that contain a large amount of data. You can create a transformation for the table that removes those columns.
+
+[!INCLUDE [azure-monitor-cost-optimization](../../../includes/azure-monitor-cost-optimization.md)]
+
 ## Create diagnostic settings
 
 You can create and edit diagnostic settings by using multiple methods.

articles/azure-monitor/essentials/diagnostic-settings.md

@@ -498,7 +498,7 @@ sections:
         answer: |
           Yes, for experimental use. In the basic pricing plan, your application can send a certain allowance of data each month free of charge. The free allowance is large enough to cover development, and publishing an app for a few users. You can set a cap to prevent more than a specified amount of data from being processed.
           
-          Larger volumes of telemetry are charged by the Gb. We provide some tips on how to [limit your charges](best-practices-cost.md#application-insights).
+          Larger volumes of telemetry are charged by the Gb. We provide some tips on how to [limit your charges](best-practices-cost.md#data-collection).
           
           The Enterprise plan incurs a charge for each day that each web server node sends telemetry. It's suitable if you want to use Continuous Export on a large scale.