Skip to content

Commit 53c6254

Browse files
committed
final updates
1 parent f359a68 commit 53c6254

File tree

2 files changed

+38
-39
lines changed

2 files changed

+38
-39
lines changed

articles/azure-monitor/containers/container-insights-data-collection-configure.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -195,7 +195,7 @@ The following table describes the parameters you need to provide values for in e
195195
| `dataCollectionInterval` | Determines how often the agent collects data. Valid values are 1m - 30m in 1m intervals The default value is 1m. If the value is outside the allowed range, then it defaults to *1 m*. |
196196
| `namespaceFilteringModeForDataCollection` | *Include*: Collects only data from the values in the *namespaces* field.<br>*Exclude*: Collects data from all namespaces except for the values in the *namespaces* field.<br>*Off*: Ignores any *namespace* selections and collect data on all namespaces.
197197
| `namespacesForDataCollection` | Array of comma separated Kubernetes namespaces to collect inventory and perf data based on the _namespaceFilteringMode_.<br>For example, *namespaces = ["kube-system", "default"]* with an _Include_ setting collects only these two namespaces. With an _Exclude_ setting, the agent collects data from all other namespaces except for _kube-system_ and _default_. With an _Off_ setting, the agent collects data from all namespaces including _kube-system_ and _default_. Invalid and unrecognized namespaces are ignored. |
198-
| `streams` | An array of container insights table streams. See [Stream values in DCR](#stream-values-in-dcr) for a list of the valid streams and their corresponding tables. To enable [high scale mode](./container-insights-high-scale.md) for container logs, use `Microsoft-ContainerLogV2-HighScale`. |
198+
| `streams` | An array of container insights table streams. See [Stream values in DCR](#stream-values-in-dcr) for a list of the valid streams and their corresponding tables.<br><br>To enable [high scale mode](./container-insights-high-scale.md) for container logs, use `Microsoft-ContainerLogV2-HighScale`. |
199199
| `useAzureMonitorPrivateLinkScope` | Specifies whether to use private link for the cluster connection to Azure Monitor. |
200200
| `azureMonitorPrivateLinkScopeResourceId` | If private link is used, resource ID of the private link scope. |
201201

articles/azure-monitor/containers/container-insights-high-scale.md

Lines changed: 37 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -6,23 +6,24 @@ ms.date: 08/06/2024
66
---
77

88
# High scale logs collection in Container Insights (Preview)
9-
High scale mode is a feature in Container Insights that enables you to collect container console (stdout & stderr) logs with high throughput from your AKS cluster nodes. This feature is intended for Azure Kubernetes Service clusters sending more than 5,000 logs/sec per node.
9+
High scale mode is a feature in Container Insights that enables you to collect container console (stdout & stderr) logs with high throughput from your Azure Kubernetes Service (AKS) cluster nodes. This feature is intended for AKS clusters sending more than 5,000 logs/sec per node.
1010

1111
## Overview
1212
When high scale mode is enabled, Container Insights performs multiple configuration changes resulting in a higher overall throughput. This includes using an upgraded agent and Azure Monitor data pipeline with scale improvements. These changes are all made in the background by Azure Monitor and don't require input or configuration after the feature is enabled.
1313

14-
High scale logs collection is suited for environments sending more than 5,000 logs/sec per node in their Kubernetes clusters and has been designed and tested for sending up to 50,000 logs/sec per node. High scale mode impacts only the data collection layer. The rest of the Container insights experience remains the same, with logs being ingested into same `ContainerLogV2` table. Existing queries and alerts continue to work since the same data is being collected.
14+
High scale mode impacts only the data collection layer. The rest of the Container insights experience remains the same, with logs being ingested into same `ContainerLogV2` table. Existing queries and alerts continue to work since the same data is being collected.
1515

1616
To achieve the maximum supported logs throughput, you should use high-end VM SKUs with 16 CPU cores or more for your AKS cluster nodes. Using low end VM SKUs will impact your logs throughput.
1717

1818
## Does my cluster qualify?
19-
Use the following [log queries](../logs/log-query-overview.md) to determine whether your cluster is suitable for high scale logs collection.
19+
High scale logs collection is suited for environments sending more than 5,000 logs/sec per node in their Kubernetes clusters and has been designed and tested for sending up to 50,000 logs/sec per node. Use the following [log queries](../logs/log-query-overview.md) to determine whether your cluster is suitable for high scale logs collection.
20+
2021

2122
**Logs per second and per node**
2223

23-
```kql
24+
```kusto
2425
ContainerLogV2
25-
| where _ResourceId = "<AzureResourceIdoftheAKSCluster>"
26+
| where _ResourceId = "<cluster-resource-id>"
2627
| summarize count() by bin(TimeGenerated, 1s), Computer
2728
| render timechart
2829
```
@@ -31,7 +32,7 @@ ContainerLogV2
3132

3233
```kusto
3334
ContainerLogV2
34-
| where _ResourceId = "<AzureResourceIdoftheAKSCluster>"
35+
| where _ResourceId = "<cluster-resource-id>"
3536
| summarize BillableDataMB = sum(_BilledSize)/1024/1024 by bin(TimeGenerated, 1s), Computer
3637
| render timechart
3738
```
@@ -41,11 +42,24 @@ ContainerLogV2
4142
- Azure CLI version 2.63.0 or higher.
4243
- AKS-preview CLI extension version must be 7.0.0b4 or higher if an aks-preview CLI extension is installed.
4344
- Cluster schema must be [configured for ContainerLogV2](./container-insights-logs-schema.md#enable-the-containerlogv2-schema).
44-
- See [Network and firewall requirements](#network-and-firewall-requirements).
45+
46+
## Network firewall requirements
47+
In addition to the [network firewall requirements](./kubernetes-monitoring-firewall.md) for monitoring a Kubernetes cluster, additional configurations are needed for enabling High scale Mode.
48+
49+
Get the **Logs Ingestion** endpoint from the data collection endpoint (DCE) for the data collection rule (DCR) used by the cluster. The DCR name is in the form `MSCI-<region>-<clusterName>`.
50+
51+
The endpoint has a different format depending on the cloud as shown in the following table.
52+
53+
| Cloud | Endpoint | Port |
54+
|:---|:--|:--|
55+
| Azure Public Cloud | `<dce-name>-<suffix>.<cluster-region-name>-<suffix>.ingest.monitor.azure.com` | 443 |
56+
| Microsoft Azure operated by 21Vianet cloud | `<dce-name>-<suffix>.<cluster-region-name>-<suffix>.ingest.monitor.azure.cn` | 443 |
57+
| Azure Government cloud | `<dce-name>-<suffix>.<cluster-region-name>-<suffix>.ingest.monitor.azure.us` | 443 |
58+
4559

4660
## Limitations
4761

48-
The following scenarios aren't supported during the Preview release. These will be addressed when the feature becomes generally available.
62+
The following scenarios aren't supported during the preview release. These will be addressed when the feature becomes generally available.
4963

5064
- AKS Clusters with Arm64 nodes
5165
- Azure Arc-enabled Kubernetes
@@ -55,10 +69,10 @@ The following scenarios aren't supported during the Preview release. These will
5569
- Automatic migration from existing Container Insights
5670

5771
## Enable high scale logs collection
58-
Use the following steps to enable high scale logs collection for your cluster.
72+
The following two steps are required to enable high scale mode for your cluster.
5973

6074
### Update configmap
61-
The first step is to update configmap for the cluster to instruct the container insights ama-logs deamonset pods to run in high log scale mode.
75+
The first step is to update configmap for the cluster to instruct the container insights ama-logs deamonset pods to run in high scale mode.
6276

6377
Follow the guidance in [Configure and deploy ConfigMap](./container-insights-data-collection-configmap.md#configure-and-deploy-configmap) to download and update ConfigMap for the cluster. The only change you need to make for high scale logs is to add the following entry under `agent-settings`:
6478

@@ -67,57 +81,42 @@ Follow the guidance in [Configure and deploy ConfigMap](./container-insights-dat
6781
enabled = true
6882
```
6983

70-
After applying this configmap, `ama-logs-*` pods will get restarted automatically and configure the ama-logs daemonset pods to run in high scale log mode.
84+
After applying this configmap, `ama-logs-*` pods will get restarted automatically and configure the ama-logs daemonset pods to run in high scale mode.
85+
86+
### Enable high scale mode for Monitoring add-on
87+
Once the ama-logs pods are running in high log scale mode, you can enable the Monitoring Add-on with high scale mode. Use the following Azure CLI commands to enable high scale logs mode for the Monitoring add-on depending on your AKS configuration.
88+
89+
> [!NOTE]
90+
> Instead of CLI, you can use an ARM template to enable high scale mode for the Monitoring add-on. See [Enable Container insights](./kubernetes-monitoring-enable.md?tabs=arm#enable-container-insights) for guidance on enabling Container Insights using an ARM template. To enable high scale mode, use `Microsoft-ContainerLogV2-HighScale` in the `streams` parameter.
7191
72-
### Enable high scale log mode for Monitoring add-on
73-
Once the ama-logs pods are running in high log scale mode, you can enable the Monitoring Add-on with high scale log mode. You have the option to perform this configuration using either Azure CLI or an ARM Template.
92+
> Don't use both `Microsoft-ContainerLogV2` and `Microsoft-ContainerLogV2-HighScale` in the `streams` parameter. This will result in logs being collected in the standard mode.
7493
75-
### [CLI](#tab/cli)
76-
Use the following Azure CLI commands to enable high scale logs mode for the Monitoring add-on depending on your AKS configuration.
7794

78-
#### Existing AKS cluster
95+
**Existing AKS cluster**
7996

8097
```azurecli
8198
az aks enable-addons -a monitoring -g <resource-group-name> -n <cluster-name> --enable-high-log-scale-mode
8299
```
83100

84-
#### Existing AKS Private cluster
101+
**Existing AKS Private cluster**
85102

86103
```azurecli
87104
az aks enable-addons -a monitoring -g <resource-group-name> -n <cluster-name> --enable-high-scale-mode --ampls-resource-id /subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/microsoft.insights/privatelinkscopes/<resourceName>
88105
```
89106

90-
#### New AKS cluster
107+
**New AKS cluster**
91108

92109
```azurecli
93110
az aks create -g <cluster-name> -n <cluster-name> enable-addons -a monitoring --enable-high-log-scale-mode
94111
```
95112

96-
#### New AKS Private cluster
97-
See [Create a private Azure Kubernetes Service (AKS) cluster](/azure/aks/private-clusters?tabs=azure-portal) for details on creating an AKS Private cluster. Use the additional parameters `--enable-high-scale-mode` and `--ampls-resource-id` to configure high log scale mode with Azure Monitor Private Link Scope Resource ID.
98-
99-
113+
**New AKS Private cluster**
100114

101-
### [ARM](#tab/arm)
102-
See [Enable Container insights](./kubernetes-monitoring-enable.md?tabs=arm#enable-container-insights) for guidance on enabling Container Insights using an ARM template. To enable high scale logs mode, use `Microsoft-ContainerLogV2-HighScale` in the `streams` parameter.
115+
See [Create a private Azure Kubernetes Service (AKS) cluster](/azure/aks/private-clusters?tabs=azure-portal) for details on creating an AKS Private cluster. Use the additional parameters `--enable-high-scale-mode` and `--ampls-resource-id` to configure high log scale mode with Azure Monitor Private Link Scope Resource ID.
103116

104-
> [!WARNING]
105-
> Don't use both `Microsoft-ContainerLogV2` and `Microsoft-ContainerLogV2-HighScale` in the `streams` parameter. This will result in logs being collected in the standard mode.
106117

107118
---
108119

109-
## Network and firewall requirements
110-
In addition to the [network firewall requirements](See [Network firewall requirements for monitoring Kubernetes cluster](./kubernetes-monitoring-firewall.md)) for monitoring a Kubernetes cluster, additional configurations are needed for enabling High scale Mode.
111-
112-
Get the **Logs Ingestion** endpoint from the data collection endpoint (DCE) for the data collection rule (DCR) used by the cluster. The DCR name is in the form `MSCI-<region>-<clusterName>`.
113-
114-
The endpoint has a different format depending on the cloud as shown in the following table.
115-
116-
| Cloud | Endpoint | Port |
117-
|:---|:--|:--|
118-
| Azure Public Cloud | `<dce-name>-<suffix>.<cluster-region-name>-<suffix>.ingest.monitor.azure.com` | 443 |
119-
| Microsoft Azure operated by 21Vianet cloud | `<dce-name>-<suffix>.<cluster-region-name>-<suffix>.ingest.monitor.azure.cn` | 443 |
120-
| Azure Government cloud | `<dce-name>-<suffix>.<cluster-region-name>-<suffix>.ingest.monitor.azure.us` | 443 |
121120

122121

123122
## Next steps

0 commit comments

Comments
 (0)