You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/azure-monitor/containers/container-insights-data-collection-configure.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -195,7 +195,7 @@ The following table describes the parameters you need to provide values for in e
195
195
|`dataCollectionInterval`| Determines how often the agent collects data. Valid values are 1m - 30m in 1m intervals The default value is 1m. If the value is outside the allowed range, then it defaults to *1 m*. |
196
196
| `namespaceFilteringModeForDataCollection` | *Include*: Collects only data from the values in the *namespaces* field.<br>*Exclude*: Collects data from all namespaces except for the values in the *namespaces* field.<br>*Off*: Ignores any *namespace* selections and collect data on all namespaces.
197
197
|`namespacesForDataCollection`| Array of comma separated Kubernetes namespaces to collect inventory and perf data based on the _namespaceFilteringMode_.<br>For example, *namespaces = ["kube-system", "default"]* with an _Include_ setting collects only these two namespaces. With an _Exclude_ setting, the agent collects data from all other namespaces except for _kube-system_ and _default_. With an _Off_ setting, the agent collects data from all namespaces including _kube-system_ and _default_. Invalid and unrecognized namespaces are ignored. |
198
-
|`streams`| An array of container insights table streams. See [Stream values in DCR](#stream-values-in-dcr) for a list of the valid streams and their corresponding tables.To enable [high scale mode](./container-insights-high-scale.md) for container logs, use `Microsoft-ContainerLogV2-HighScale`. |
198
+
|`streams`| An array of container insights table streams. See [Stream values in DCR](#stream-values-in-dcr) for a list of the valid streams and their corresponding tables.<br><br>To enable [high scale mode](./container-insights-high-scale.md) for container logs, use `Microsoft-ContainerLogV2-HighScale`. |
199
199
|`useAzureMonitorPrivateLinkScope`| Specifies whether to use private link for the cluster connection to Azure Monitor. |
200
200
|`azureMonitorPrivateLinkScopeResourceId`| If private link is used, resource ID of the private link scope. |
Copy file name to clipboardExpand all lines: articles/azure-monitor/containers/container-insights-high-scale.md
+37-38Lines changed: 37 additions & 38 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,23 +6,24 @@ ms.date: 08/06/2024
6
6
---
7
7
8
8
# High scale logs collection in Container Insights (Preview)
9
-
High scale mode is a feature in Container Insights that enables you to collect container console (stdout & stderr) logs with high throughput from your AKS cluster nodes. This feature is intended for Azure Kubernetes Service clusters sending more than 5,000 logs/sec per node.
9
+
High scale mode is a feature in Container Insights that enables you to collect container console (stdout & stderr) logs with high throughput from your Azure Kubernetes Service (AKS) cluster nodes. This feature is intended for AKS clusters sending more than 5,000 logs/sec per node.
10
10
11
11
## Overview
12
12
When high scale mode is enabled, Container Insights performs multiple configuration changes resulting in a higher overall throughput. This includes using an upgraded agent and Azure Monitor data pipeline with scale improvements. These changes are all made in the background by Azure Monitor and don't require input or configuration after the feature is enabled.
13
13
14
-
High scale logs collection is suited for environments sending more than 5,000 logs/sec per node in their Kubernetes clusters and has been designed and tested for sending up to 50,000 logs/sec per node. High scale mode impacts only the data collection layer. The rest of the Container insights experience remains the same, with logs being ingested into same `ContainerLogV2` table. Existing queries and alerts continue to work since the same data is being collected.
14
+
High scale mode impacts only the data collection layer. The rest of the Container insights experience remains the same, with logs being ingested into same `ContainerLogV2` table. Existing queries and alerts continue to work since the same data is being collected.
15
15
16
16
To achieve the maximum supported logs throughput, you should use high-end VM SKUs with 16 CPU cores or more for your AKS cluster nodes. Using low end VM SKUs will impact your logs throughput.
17
17
18
18
## Does my cluster qualify?
19
-
Use the following [log queries](../logs/log-query-overview.md) to determine whether your cluster is suitable for high scale logs collection.
19
+
High scale logs collection is suited for environments sending more than 5,000 logs/sec per node in their Kubernetes clusters and has been designed and tested for sending up to 50,000 logs/sec per node. Use the following [log queries](../logs/log-query-overview.md) to determine whether your cluster is suitable for high scale logs collection.
20
+
20
21
21
22
**Logs per second and per node**
22
23
23
-
```kql
24
+
```kusto
24
25
ContainerLogV2
25
-
| where _ResourceId = "<AzureResourceIdoftheAKSCluster>"
26
+
| where _ResourceId = "<cluster-resource-id>"
26
27
| summarize count() by bin(TimeGenerated, 1s), Computer
27
28
| render timechart
28
29
```
@@ -31,7 +32,7 @@ ContainerLogV2
31
32
32
33
```kusto
33
34
ContainerLogV2
34
-
| where _ResourceId = "<AzureResourceIdoftheAKSCluster>"
35
+
| where _ResourceId = "<cluster-resource-id>"
35
36
| summarize BillableDataMB = sum(_BilledSize)/1024/1024 by bin(TimeGenerated, 1s), Computer
36
37
| render timechart
37
38
```
@@ -41,11 +42,24 @@ ContainerLogV2
41
42
- Azure CLI version 2.63.0 or higher.
42
43
- AKS-preview CLI extension version must be 7.0.0b4 or higher if an aks-preview CLI extension is installed.
43
44
- Cluster schema must be [configured for ContainerLogV2](./container-insights-logs-schema.md#enable-the-containerlogv2-schema).
44
-
- See [Network and firewall requirements](#network-and-firewall-requirements).
45
+
46
+
## Network firewall requirements
47
+
In addition to the [network firewall requirements](./kubernetes-monitoring-firewall.md) for monitoring a Kubernetes cluster, additional configurations are needed for enabling High scale Mode.
48
+
49
+
Get the **Logs Ingestion** endpoint from the data collection endpoint (DCE) for the data collection rule (DCR) used by the cluster. The DCR name is in the form `MSCI-<region>-<clusterName>`.
50
+
51
+
The endpoint has a different format depending on the cloud as shown in the following table.
52
+
53
+
| Cloud | Endpoint | Port |
54
+
|:---|:--|:--|
55
+
| Azure Public Cloud |`<dce-name>-<suffix>.<cluster-region-name>-<suffix>.ingest.monitor.azure.com`| 443 |
56
+
| Microsoft Azure operated by 21Vianet cloud |`<dce-name>-<suffix>.<cluster-region-name>-<suffix>.ingest.monitor.azure.cn`| 443 |
57
+
| Azure Government cloud |`<dce-name>-<suffix>.<cluster-region-name>-<suffix>.ingest.monitor.azure.us`| 443 |
58
+
45
59
46
60
## Limitations
47
61
48
-
The following scenarios aren't supported during the Preview release. These will be addressed when the feature becomes generally available.
62
+
The following scenarios aren't supported during the preview release. These will be addressed when the feature becomes generally available.
49
63
50
64
- AKS Clusters with Arm64 nodes
51
65
- Azure Arc-enabled Kubernetes
@@ -55,10 +69,10 @@ The following scenarios aren't supported during the Preview release. These will
55
69
- Automatic migration from existing Container Insights
56
70
57
71
## Enable high scale logs collection
58
-
Use the following steps to enable high scale logs collection for your cluster.
72
+
The following two steps are required to enable high scale mode for your cluster.
59
73
60
74
### Update configmap
61
-
The first step is to update configmap for the cluster to instruct the container insights ama-logs deamonset pods to run in high log scale mode.
75
+
The first step is to update configmap for the cluster to instruct the container insights ama-logs deamonset pods to run in high scale mode.
62
76
63
77
Follow the guidance in [Configure and deploy ConfigMap](./container-insights-data-collection-configmap.md#configure-and-deploy-configmap) to download and update ConfigMap for the cluster. The only change you need to make for high scale logs is to add the following entry under `agent-settings`:
64
78
@@ -67,57 +81,42 @@ Follow the guidance in [Configure and deploy ConfigMap](./container-insights-dat
67
81
enabled = true
68
82
```
69
83
70
-
After applying this configmap, `ama-logs-*` pods will get restarted automatically and configure the ama-logs daemonset pods to run in high scale log mode.
84
+
After applying this configmap, `ama-logs-*` pods will get restarted automatically and configure the ama-logs daemonset pods to run in high scale mode.
85
+
86
+
### Enable high scale mode for Monitoring add-on
87
+
Once the ama-logs pods are running in high log scale mode, you can enable the Monitoring Add-on with high scale mode. Use the following Azure CLI commands to enable high scale logs mode for the Monitoring add-on depending on your AKS configuration.
88
+
89
+
> [!NOTE]
90
+
> Instead of CLI, you can use an ARM template to enable high scale mode for the Monitoring add-on. See [Enable Container insights](./kubernetes-monitoring-enable.md?tabs=arm#enable-container-insights) for guidance on enabling Container Insights using an ARM template. To enable high scale mode, use `Microsoft-ContainerLogV2-HighScale` in the `streams` parameter.
71
91
72
-
### Enable high scale log mode for Monitoring add-on
73
-
Once the ama-logs pods are running in high log scale mode, you can enable the Monitoring Add-on with high scale log mode. You have the option to perform this configuration using either Azure CLI or an ARM Template.
92
+
> Don't use both `Microsoft-ContainerLogV2` and `Microsoft-ContainerLogV2-HighScale` in the `streams` parameter. This will result in logs being collected in the standard mode.
74
93
75
-
### [CLI](#tab/cli)
76
-
Use the following Azure CLI commands to enable high scale logs mode for the Monitoring add-on depending on your AKS configuration.
77
94
78
-
#### Existing AKS cluster
95
+
**Existing AKS cluster**
79
96
80
97
```azurecli
81
98
az aks enable-addons -a monitoring -g <resource-group-name> -n <cluster-name> --enable-high-log-scale-mode
82
99
```
83
100
84
-
#### Existing AKS Private cluster
101
+
**Existing AKS Private cluster**
85
102
86
103
```azurecli
87
104
az aks enable-addons -a monitoring -g <resource-group-name> -n <cluster-name> --enable-high-scale-mode --ampls-resource-id /subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/microsoft.insights/privatelinkscopes/<resourceName>
88
105
```
89
106
90
-
#### New AKS cluster
107
+
**New AKS cluster**
91
108
92
109
```azurecli
93
110
az aks create -g <cluster-name> -n <cluster-name> enable-addons -a monitoring --enable-high-log-scale-mode
94
111
```
95
112
96
-
#### New AKS Private cluster
97
-
See [Create a private Azure Kubernetes Service (AKS) cluster](/azure/aks/private-clusters?tabs=azure-portal) for details on creating an AKS Private cluster. Use the additional parameters `--enable-high-scale-mode` and `--ampls-resource-id` to configure high log scale mode with Azure Monitor Private Link Scope Resource ID.
98
-
99
-
113
+
**New AKS Private cluster**
100
114
101
-
### [ARM](#tab/arm)
102
-
See [Enable Container insights](./kubernetes-monitoring-enable.md?tabs=arm#enable-container-insights) for guidance on enabling Container Insights using an ARM template. To enable high scale logs mode, use `Microsoft-ContainerLogV2-HighScale` in the `streams` parameter.
115
+
See [Create a private Azure Kubernetes Service (AKS) cluster](/azure/aks/private-clusters?tabs=azure-portal) for details on creating an AKS Private cluster. Use the additional parameters `--enable-high-scale-mode` and `--ampls-resource-id` to configure high log scale mode with Azure Monitor Private Link Scope Resource ID.
103
116
104
-
> [!WARNING]
105
-
> Don't use both `Microsoft-ContainerLogV2` and `Microsoft-ContainerLogV2-HighScale` in the `streams` parameter. This will result in logs being collected in the standard mode.
106
117
107
118
---
108
119
109
-
## Network and firewall requirements
110
-
In addition to the [network firewall requirements](See [Network firewall requirements for monitoring Kubernetes cluster](./kubernetes-monitoring-firewall.md)) for monitoring a Kubernetes cluster, additional configurations are needed for enabling High scale Mode.
111
-
112
-
Get the **Logs Ingestion** endpoint from the data collection endpoint (DCE) for the data collection rule (DCR) used by the cluster. The DCR name is in the form `MSCI-<region>-<clusterName>`.
113
-
114
-
The endpoint has a different format depending on the cloud as shown in the following table.
115
-
116
-
| Cloud | Endpoint | Port |
117
-
|:---|:--|:--|
118
-
| Azure Public Cloud |`<dce-name>-<suffix>.<cluster-region-name>-<suffix>.ingest.monitor.azure.com`| 443 |
119
-
| Microsoft Azure operated by 21Vianet cloud |`<dce-name>-<suffix>.<cluster-region-name>-<suffix>.ingest.monitor.azure.cn`| 443 |
120
-
| Azure Government cloud |`<dce-name>-<suffix>.<cluster-region-name>-<suffix>.ingest.monitor.azure.us`| 443 |
0 commit comments