|
| 1 | +--- |
| 2 | +title: High scale logs collection in Container Insights (Preview) |
| 3 | +description: Enable high scale logs collection in Container Insights (Preview). |
| 4 | +ms.topic: conceptual |
| 5 | +ms.date: 08/06/2024 |
| 6 | +--- |
| 7 | + |
| 8 | +# High scale logs collection in Container Insights (Preview) |
| 9 | +High scale mode is a feature in Container Insights that enables you to collect container console (stdout & stderr) logs with high throughput from your Azure Kubernetes Service (AKS) cluster nodes. This feature enables you to collect up to 50,000 logs/sec per node. |
| 10 | + |
| 11 | +## Overview |
| 12 | +When high scale mode is enabled, Container Insights performs multiple configuration changes resulting in a higher overall throughput. This includes using an upgraded agent and Azure Monitor data pipeline with scale improvements. These changes are all made in the background by Azure Monitor and don't require input or configuration after the feature is enabled. |
| 13 | + |
| 14 | +High scale mode impacts only the data collection layer. The rest of the Container insights experience remains the same, with logs being ingested into same `ContainerLogV2` table. Existing queries and alerts continue to work since the same data is being collected. |
| 15 | + |
| 16 | +To achieve the maximum supported logs throughput, you should use high-end VM SKUs with 16 CPU cores or more for your AKS cluster nodes. Using low end VM SKUs will impact your logs throughput. |
| 17 | + |
| 18 | +## Does my cluster qualify? |
| 19 | +High scale logs collection is suited for environments sending more than 2,000 logs/sec (or 2 MB/sec) per node in their Kubernetes clusters and has been designed and tested for sending up to 50,000 logs/sec per node. Use the following [log queries](../logs/log-query-overview.md) to determine whether your cluster is suitable for high scale logs collection. |
| 20 | + |
| 21 | + |
| 22 | +**Logs per second and per node** |
| 23 | + |
| 24 | +```kusto |
| 25 | +ContainerLogV2 |
| 26 | +| where _ResourceId = "<cluster-resource-id>" |
| 27 | +| summarize count() by bin(TimeGenerated, 1s), Computer |
| 28 | +| render timechart |
| 29 | +``` |
| 30 | + |
| 31 | +**Log size (in MB) per second per node** |
| 32 | + |
| 33 | +```kusto |
| 34 | + ContainerLogV2 |
| 35 | +| where _ResourceId = "<cluster-resource-id>" |
| 36 | +| summarize BillableDataMB = sum(_BilledSize)/1024/1024 by bin(TimeGenerated, 1s), Computer |
| 37 | +| render timechart |
| 38 | +``` |
| 39 | + |
| 40 | +## Prerequisites |
| 41 | + |
| 42 | +- Azure CLI version 2.63.0 or higher. |
| 43 | +- AKS-preview CLI extension version must be 7.0.0b4 or higher if an aks-preview CLI extension is installed. |
| 44 | +- Cluster schema must be [configured for ContainerLogV2](./container-insights-logs-schema.md#enable-the-containerlogv2-schema). |
| 45 | +- If the default resource limits (CPU and memory) on ama-logs daemon set container doesn't meet your log scale requirements, please contact the Microsoft support channel to increase the resource limits of your ama-logs container. |
| 46 | + |
| 47 | +## Network firewall requirements |
| 48 | +In addition to the [network firewall requirements](./kubernetes-monitoring-firewall.md) for monitoring a Kubernetes cluster, additional configurations in the following table are needed for enabling high scale mode depending on your cloud. |
| 49 | + |
| 50 | +| Cloud | Endpoint | Port | |
| 51 | +|:---|:--|:--| |
| 52 | +| Azure Public Cloud | `<dce-name>-<suffix>.<cluster-region-name>-<suffix>.ingest.monitor.azure.com` | 443 | |
| 53 | +| Microsoft Azure operated by 21Vianet cloud | `<dce-name>-<suffix>.<cluster-region-name>-<suffix>.ingest.monitor.azure.cn` | 443 | |
| 54 | +| Azure Government cloud | `<dce-name>-<suffix>.<cluster-region-name>-<suffix>.ingest.monitor.azure.us` | 443 | |
| 55 | + |
| 56 | +The endpoint is the **Logs Ingestion** endpoint from the data collection endpoint (DCE) for the data collection rule (DCR) used by the cluster. This DCE is created when you enable high scale mode for the cluster and will start with the prefix `MSCI-ingest`. |
| 57 | + |
| 58 | +:::image type="content" source="media/container-insights-high-scale/logs-ingestion-endpoint.png" alt-text="Screenshot of logs ingestion endpoint for DCE." lightbox="media/container-insights-high-scale/logs-ingestion-endpoint.png" ::: |
| 59 | + |
| 60 | + |
| 61 | +## Limitations |
| 62 | + |
| 63 | +The following scenarios aren't supported during the preview release. These will be addressed when the feature becomes generally available. |
| 64 | + |
| 65 | +- AKS Clusters with Arm64 nodes |
| 66 | +- Azure Arc-enabled Kubernetes |
| 67 | +- HTTP proxy with trusted certificate |
| 68 | +- Onboarding through Azure portal, Azure Policy, Terraform and Bicep |
| 69 | +- Configuring through **Monitor Settings** in the AKS Insights portal experience |
| 70 | +- Automatic migration from existing Container Insights |
| 71 | + |
| 72 | +## Enable high scale logs collection |
| 73 | +Follow the two steps in the following sections to enable high scale mode for your cluster. |
| 74 | + |
| 75 | +> [!NOTE] |
| 76 | +> High log scale mode requires a [data collection endpoint (DCE)](../essentials/data-collection-endpoint-overview.md) for ingestion. An ingestion DCE is created with the prefix `MSCI-ingest` for each cluster when you onboard them. If Azure Monitor private link scope is configured, then there will also be configuration DCE created with the prefix `MSCI-config`. |
| 77 | +
|
| 78 | +### Update configmap |
| 79 | +The first step is to update configmap for the cluster to instruct the container insights ama-logs deamonset pods to run in high scale mode. |
| 80 | + |
| 81 | +Follow the guidance in [Configure and deploy ConfigMap](./container-insights-data-collection-configmap.md#configure-and-deploy-configmap) to download and update ConfigMap for the cluster. The only change you need to make for high scale logs is to enable `agent_settings.high_log_scale` under `agent-settings` as in the following: |
| 82 | + |
| 83 | +```yml |
| 84 | +[agent_settings.high_log_scale] |
| 85 | + enabled = true |
| 86 | +``` |
| 87 | + |
| 88 | +After applying this configmap, `ama-logs-*` pods will get restarted automatically and configure the ama-logs daemonset pods to run in high scale mode. |
| 89 | + |
| 90 | +### Enable high scale mode for Monitoring add-on |
| 91 | +Enable the Monitoring Add-on with high scale mode using the following Azure CLI commands to enable high scale logs mode for the Monitoring add-on depending on your AKS configuration. |
| 92 | + |
| 93 | +> [!NOTE] |
| 94 | +> Instead of CLI, you can use an ARM template to enable high scale mode for the Monitoring add-on. See [Enable Container insights](./kubernetes-monitoring-enable.md?tabs=arm#enable-container-insights) for guidance on enabling Container Insights using an ARM template. To enable high scale mode, use `Microsoft-ContainerLogV2-HighScale` instead of `Microsoft-ContainerLogV2` in the `streams` parameter as described in [Configure DCR with ARM templates](./container-insights-data-collection-configure.md?tabs=arm#configure-dcr-with-arm-templates). |
| 95 | +
|
| 96 | + |
| 97 | +**Existing AKS cluster** |
| 98 | + |
| 99 | +```azurecli |
| 100 | +az aks enable-addons -a monitoring -g <resource-group-name> -n <cluster-name> --enable-high-log-scale-mode |
| 101 | +``` |
| 102 | + |
| 103 | +**Existing AKS Private cluster** |
| 104 | + |
| 105 | +```azurecli |
| 106 | +az aks enable-addons -a monitoring -g <resource-group-name> -n <cluster-name> --enable-high-scale-mode --ampls-resource-id /subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/microsoft.insights/privatelinkscopes/<resourceName> |
| 107 | +``` |
| 108 | + |
| 109 | +**New AKS cluster** |
| 110 | + |
| 111 | +```azurecli |
| 112 | +az aks create -g <cluster-name> -n <cluster-name> enable-addons -a monitoring --enable-high-log-scale-mode |
| 113 | +``` |
| 114 | + |
| 115 | +**New AKS Private cluster** |
| 116 | + |
| 117 | +See [Create a private Azure Kubernetes Service (AKS) cluster](/azure/aks/private-clusters?tabs=azure-portal) for details on creating an AKS Private cluster. Use the additional parameters `--enable-high-scale-mode` and `--ampls-resource-id` to configure high log scale mode with Azure Monitor Private Link Scope Resource ID. |
| 118 | + |
| 119 | +## Migration |
| 120 | +If Container insights is already enabled for your cluster, then you need to disable it and then re-enable it with high scale mode. |
| 121 | + |
| 122 | +- Since high scale mode uses a different data pipeline, you must ensure that pipeline endpoints are not blocked by a firewall or other network connections. |
| 123 | +- High scale mode requires a data collection endpoint (DCE) for ingestion in addition to the standard DCR for data collection. If you've created any DCRs that use `Microsoft.ContainerLogV2`, you must replace this with `Microsoft.ContainerLogV2-HighScale` or data will be duplicated. You should also create a DCE for ingestion and link it to the DCR if the DCR isn't already using one. Refer to Container Insights onboarding through Azure Resource Manager for reference for the dependencies. |
| 124 | + |
| 125 | + |
| 126 | +## Next steps |
| 127 | +- Share any feedback or issues with High Scale mode at [https://aka.ms/cihsfeedback](https://aka.ms/cihsfeedback). |
| 128 | + |
0 commit comments