|
| 1 | +--- |
| 2 | +title: Monitoring cost for Azure Monitor for containers | Microsoft Docs |
| 3 | +description: This article describes the monitoring cost for metrics & inventory data collected by Azure Monitor for containers to help customers manage their usage and associated costs. |
| 4 | +ms.topic: conceptual |
| 5 | +ms.date: 05/27/2020 |
| 6 | + |
| 7 | +--- |
| 8 | +# Understand monitoring costs for Azure Monitor for containers |
| 9 | + |
| 10 | +This article provides pricing guidance for Azure Monitor for containers to help you understand the following: |
| 11 | + |
| 12 | +* How to estimate costs up-front before you enable this Insight |
| 13 | + |
| 14 | +* How to measure costs after Azure Monitor for containers has been enabled for one or more containers |
| 15 | + |
| 16 | +* How to control the collection of data and make cost reductions |
| 17 | + |
| 18 | +Azure Monitor Logs collects, indexes, and stores data generated by your Kubernetes cluster. |
| 19 | + |
| 20 | +The Azure Monitor pricing model is primarily based on the amount of data ingested in gigabytes per day into your Log Analytics workspace. The cost of a Log Analytics workspace isn't based only on the volume of data collected, it is also dependent on the plan selected, and how long you chose to store data generated from your clusters. |
| 21 | + |
| 22 | +>[!NOTE] |
| 23 | +>All sizes and pricing are for sample estimation only. Please refer to the Azure Monitor [pricing](https://azure.microsoft.com/pricing/details/monitor/) page for the most recent pricing based on your Azure Monitor Log Analytics pricing model and Azure region. |
| 24 | +
|
| 25 | +The following is a summary of what types of data are collected from a Kubernetes cluster with Azure Monitor for containers that influences cost and can be customized based on your usage: |
| 26 | + |
| 27 | +- Stdout, stderr container logs from every monitored container in every Kubernetes namespace in the cluster |
| 28 | + |
| 29 | +- Container environment variables from every monitored container in the cluster |
| 30 | + |
| 31 | +- Completed Kubernetes jobs/pods in the cluster that does not require monitoring |
| 32 | + |
| 33 | +- Active scraping of Prometheus metrics |
| 34 | + |
| 35 | +- [Diagnostic log collection](../../aks/view-master-logs.md) of Kubernetes master node logs in your AKS cluster to analyze log data generated by master components such as the *kube-apiserver* and *kube-controller-manager*. |
| 36 | + |
| 37 | +## What is collected from Kubernetes clusters |
| 38 | + |
| 39 | +Azure Monitor for containers includes a predefined set of metrics and inventory items collected that are written as log data in your Log Analytics workspace. All metrics listed below are collected by default every one minute. |
| 40 | + |
| 41 | +### Node metrics collected |
| 42 | + |
| 43 | +The following list is the 24 metrics per node that are collected: |
| 44 | + |
| 45 | +- cpuUsageNanoCores |
| 46 | +- cpuCapacityNanoCores |
| 47 | +- cpuAllocatableNanoCores |
| 48 | +- memoryRssBytes |
| 49 | +- memoryWorkingSetBytes |
| 50 | +- memoryCapacityBytes |
| 51 | +- memoryAllocatableBytes |
| 52 | +- restartTimeEpoch |
| 53 | +- used (disk) |
| 54 | +- free (disk) |
| 55 | +- used_percent (disk) |
| 56 | +- io_time (diskio) |
| 57 | +- writes (diskio) |
| 58 | +- reads (diskio) |
| 59 | +- write_bytes (diskio) |
| 60 | +- write_time (diskio) |
| 61 | +- iops_in_progress (diskio) |
| 62 | +- read_bytes (diskio) |
| 63 | +- read_time (diskio) |
| 64 | +- err_in (net) |
| 65 | +- err_out (net) |
| 66 | +- bytes_recv (net) |
| 67 | +- bytes_sent (net) |
| 68 | +- Kubelet_docker_operations (kubelet) |
| 69 | + |
| 70 | +### Container metrics |
| 71 | + |
| 72 | +The following list is the eight metrics per container collected: |
| 73 | + |
| 74 | +- cpuUsageNanoCores |
| 75 | +- cpuRequestNanoCores |
| 76 | +- cpuLimitNanoCores |
| 77 | +- memoryRssBytes |
| 78 | +- memoryWorkingSetBytes |
| 79 | +- memoryRequestBytes |
| 80 | +- memoryLimitBytes |
| 81 | +- restartTimeEpoch |
| 82 | + |
| 83 | +### Cluster inventory |
| 84 | + |
| 85 | +The following list is the cluster inventory data collected by default: |
| 86 | + |
| 87 | +- KubePodInventory – 1 per minute per container |
| 88 | +- KubeNodeInventory – 1 per node per minute |
| 89 | +- KubeServices – 1 per service per minute |
| 90 | +- ContainerInventory – 1 per container per minute |
| 91 | + |
| 92 | +## Estimating costs to monitor your AKS cluster |
| 93 | + |
| 94 | +The estimation below is based on an Azure Kubernetes Service (AKS) cluster with the following sizing example. Also, the estimate applies only for metrics and inventory data collected. For container logs (stdout, stderr, and environmental variables), it varies based on the log sizes generated by the workload, and they are excluded from our estimation. |
| 95 | + |
| 96 | +If you enabled monitoring of an AKS cluster configured as follows, |
| 97 | + |
| 98 | +- Three nodes |
| 99 | +- Two disks per node |
| 100 | +- One network interface per node |
| 101 | +- 20 pods (one container in each pod = 20 containers in total) |
| 102 | +- Two Kubernetes namespaces |
| 103 | +- Five Kubernetes services (includes kube-system pods, services, and namespace) |
| 104 | +- Collection frequency = 60 secs (default) |
| 105 | + |
| 106 | +You can see the tables and volume of data generated per hour in the assigned Log Analytics workspace. For more information about each of these tables, see [Container records](container-insights-log-search.md#container-records). |
| 107 | + |
| 108 | +|Table | Size estimate (MB/hour) | |
| 109 | +|------|---------------| |
| 110 | +|Perf | 12.9 | |
| 111 | +|InsightsMetrics | 11.3 | |
| 112 | +|KubePodInventory | 1.5 | |
| 113 | +|KubeNodeInventory | 0.75 | |
| 114 | +|KubeServices | 0.13 | |
| 115 | +|ContainerInventory | 3.6 | |
| 116 | +|KubeHealth | 0.1 | |
| 117 | +|KubeMonAgentEvents |0.005 | |
| 118 | + |
| 119 | +Total = 31 MB/Hour = 23.1 GB/month (one month = 31 days) |
| 120 | + |
| 121 | +Using the default [pricing](https://azure.microsoft.com/pricing/details/monitor/) for Log Analytics, which is a Pay-As-You-Go model, you can estimate the Azure Monitor cost per month. After including a capacity reservation, the price would be higher per month depending on the reservation selected. |
| 122 | + |
| 123 | +## Controlling ingestion to reduce cost |
| 124 | + |
| 125 | +Consider a scenario where your organization's different business unit shares Kubernetes infrastructure and a Log Analytics workspace. With each business unit separated by a Kubernetes namespace. You can visualize how much data is ingested in each workspace using a workbook recently released. The **Container Insights Usage** workbook, found in the [workbooks gallery](../platform/workbooks-overview.md#getting-started), helps you to visualize the source of your data without having to build your own library of queries from what we share in our documentation. In this workbook, there are charts with which you can view billable data from such perspectives as: |
| 126 | + |
| 127 | +- Total billable data ingested in GB by solution |
| 128 | + |
| 129 | +- Billable data ingested by Container logs(application logs) |
| 130 | + |
| 131 | +- Billable container logs data ingested per by Kubernetes namespace |
| 132 | + |
| 133 | +- Billable container logs data ingested segregated by Cluster name |
| 134 | + |
| 135 | +- Billable container log data ingested by logsource entry |
| 136 | + |
| 137 | +- Billable diagnostic data ingested by diagnostic master node logs |
| 138 | + |
| 139 | +After completing your analysis to determine which source or sources are generating the most data or more data that are exceeding your requirements, you can reconfigure data collection. Details on configuring collection of stdout, stderr, and environmental variables is described in the [Configure agent data collection settings](container-insights-agent-config.md) article. |
| 140 | + |
| 141 | +The following are examples of what changes you can apply to your cluster by modifying the ConfigMap file to help control cost. |
| 142 | + |
| 143 | +1. Disable stdout logs across all namespaces in the cluster by modifying the following in the ConfigMap file: |
| 144 | + |
| 145 | + ``` |
| 146 | + [log_collection_settings] |
| 147 | + [log_collection_settings.stdout] |
| 148 | + enabled = false |
| 149 | + ``` |
| 150 | +
|
| 151 | +2. Disable collecting stderr logs from your development namespace (for example, **dev-test**), and continue collecting stderr logs from other namespaces (for example, **prod** and **default**) by modifying the following in the ConfigMap file: |
| 152 | +
|
| 153 | + >[!NOTE] |
| 154 | + >The kube-system log collection is disabled by default. The default setting is retained, adding **dev-test** namespace to the list of exclusion namespaces is applied to stderr log collection. |
| 155 | +
|
| 156 | + ``` |
| 157 | + [log_collection_settings.stderr] |
| 158 | + enabled = true |
| 159 | + exclude_namespaces = ["kube-system", "dev-test"] |
| 160 | + ``` |
| 161 | +
|
| 162 | +3. Disable environment variable collection across the cluster by modifying the following in the ConfigMap file. This is applicable to all containers in every Kubernetes namespace. |
| 163 | +
|
| 164 | + ``` |
| 165 | + [log_collection_settings.env_var] |
| 166 | + enabled = false |
| 167 | + ``` |
| 168 | +
|
| 169 | +4. To clean up completed jobs, specify the cleanup policy in the job definition by modifying the following in the ConfigMap file: |
| 170 | +
|
| 171 | + ``` |
| 172 | + apiVersion: batch/v1 |
| 173 | + kind: Job |
| 174 | + metadata: |
| 175 | + name: pi-with-ttl |
| 176 | + spec: |
| 177 | + ttlSecondsAfterFinished: 100 |
| 178 | + ``` |
| 179 | +
|
| 180 | +After applying one or more of these changes to your ConfigMaps, see [Applying updated ConfigMap](container-insights-prometheus-integration.md#applying-updated-configmap) to apply it to your cluster. |
| 181 | +
|
| 182 | +### Prometheus metrics scraping |
| 183 | +
|
| 184 | +If you are utilizing [Prometheus metric scraping](container-insights-prometheus-integration.md), ensure you consider the following to limit the number of metrics that you collect from your cluster: |
| 185 | +
|
| 186 | +- Ensure scraping frequency is set optimally (the default is 60 seconds). While you can increase the frequency to 15 seconds, you need to ensure that the metrics you are scraping are published at that frequency. Otherwise there will be many duplicate metrics scraped and sent to your Log Analytics workspace at intervals adding to data ingestion and retention costs, but are of less value. |
| 187 | +
|
| 188 | +- Azure Monitor for containers supports exclusion & inclusion lists by metric name. For example, if you are scraping **kubedns** metrics in your cluster, there might be hundreds of them that gets scraped by default, but you are most likely only interested in a subset. Confirm you specified a list of metrics to scrape, or exclude others except a few to save on data ingestion volume. It is easy to enable scraping and not use many of those metrics, which will only add additional charges to your Log Analytics bill. |
| 189 | +
|
| 190 | +- When scraping through pod annotations, ensure you filter by namespace so that you exclude scraping of pod metrics from namespaces that you don't use (for example, **dev-test** namespace). |
| 191 | +
|
| 192 | +## Next steps |
| 193 | +
|
| 194 | +For more information about how to understand what the costs are likely to be based on recent usage patterns from data collected with Azure Monitor for containers, see [Manage your usage and estimate costs](../platform/manage-cost-storage.md). |
0 commit comments