|
| 1 | +--- |
| 2 | +title: Configure Kafka integration for Prometheus metrics in Azure Monitor |
| 3 | +description: Describes how to configure Kafka monitoring using Prometheus metrics in Azure Monitor to Kubernetes cluster. |
| 4 | +ms.topic: conceptual |
| 5 | +ms.date: 3/19/2024 |
| 6 | +ms.reviewer: rashmy |
| 7 | +ms.service: azure-monitor |
| 8 | +ms.subservice: containers |
| 9 | +--- |
| 10 | +# Apache Kafka |
| 11 | +Apache Kafka is an open-source distributed event streaming platform used by high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. |
| 12 | +This article describes how to configure Azure Managed Prometheus with Azure Kubernetes Service(AKS) to monitor kafka clusters by scraping prometheus metrics. |
| 13 | + |
| 14 | +## Prerequisites |
| 15 | + |
| 16 | ++ Kafka cluster running on AKS |
| 17 | ++ Azure Managed prometheus enabled on the AKS cluster - [Enable Azure Managed Prometheus on AKS](kubernetes-monitoring-enable.md#enable-prometheus-and-grafana) |
| 18 | + |
| 19 | + |
| 20 | +### Install Kafka Exporter |
| 21 | +Install the [Kafka Exporter](https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-kafka-exporter) using the helm chart. |
| 22 | + |
| 23 | +```bash |
| 24 | +helm install azmon-kafka-exporter --namespace=azmon-kafka-exporter --create-namespace --version 2.10.0 prometheus-community/prometheus-kafka-exporter --set kafkaServer="{kafka-server.namespace.svc:9092,.....}" --set prometheus.serviceMonitor.enabled=true --set prometheus.serviceMonitor.apiVersion=azmonitoring.coreos.com/v1 |
| 25 | +``` |
| 26 | + |
| 27 | +> [!NOTE] |
| 28 | +> Managed prometheus pod/service monitor configuration with helm chart installation is only supported with the helm chart version >=2.10.0. |
| 29 | +> |
| 30 | +> The [prometheus kafka exporter](https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-kafka-exporter) helm chart can be configured with [values](https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus-kafka-exporter/values.yaml) yaml. |
| 31 | +Please specify the right server addresses where the kafka servers can be reached. Set the server address(es) using the argument "kafkaServer". |
| 32 | +> |
| 33 | +> If you want to configure any other service or pod monitors, please follow the instructions [here](prometheus-metrics-scrape-crd.md#create-a-pod-or-service-monitor). |
| 34 | +
|
| 35 | + |
| 36 | +### Import the Grafana Dashboard |
| 37 | + |
| 38 | +To import the Grafana Dashboards using the ID or JSON, follow the instructions to [Import a dashboard from Grafana Labs](../../managed-grafana/how-to-create-dashboard.md#import-a-grafana-dashboard). </br> |
| 39 | + |
| 40 | +[Kafka Exporter Grafana Dashboard](https://grafana.com/grafana/dashboards/7589-kafka-exporter-overview/)(ID-7589) |
| 41 | + |
| 42 | +### Deploy Rules |
| 43 | +1. Download the template and parameter files |
| 44 | + |
| 45 | + **Alerting Rules** |
| 46 | + - [Template file](https://github.com/Azure/prometheus-collector/blob/main/Azure-ARM-templates/Workload-Rules/Kafka/kafka-alerting-rules.json) |
| 47 | + - [Parameter file](https://github.com/Azure/prometheus-collector/blob/main/Azure-ARM-templates/Workload-Rules/Alert-Rules-Parameters.json) |
| 48 | + |
| 49 | + |
| 50 | +2. Edit the following values in the parameter files. Retrieve the resource ID of the resources from the **JSON View** of their **Overview** page. |
| 51 | + |
| 52 | + | Parameter | Value | |
| 53 | + |:---|:---| |
| 54 | + | `azureMonitorWorkspace` | Resource ID for the Azure Monitor workspace. Retrieve from the **JSON view** on the **Overview** page for the Azure Monitor workspace. | |
| 55 | + | `location` | Location of the Azure Monitor workspace. Retrieve from the **JSON view** on the **Overview** page for the Azure Monitor workspace. | |
| 56 | + | `clusterName` | Name of the AKS cluster. Retrieve from the **JSON view** on the **Overview** page for the cluster. | |
| 57 | + | `actionGroupId` | Resource ID for the alert action group. Retrieve from the **JSON view** on the **Overview** page for the action group. Learn more about [action groups](../alerts/action-groups.md) | |
| 58 | + |
| 59 | +3. Deploy the template by using any standard methods for installing ARM templates. For guidance, see [ARM template samples for Azure Monitor](../resource-manager-samples.md). |
| 60 | + |
| 61 | +4. Once deployed, you can view the rules in the Azure portal as described in - [Prometheus Alerts](../essentials/prometheus-rule-groups.md#view-prometheus-rule-groups) |
| 62 | + |
| 63 | +> [!Note] |
| 64 | +> Review the alert thresholds to make sure it suits your cluster/workloads and update it accordingly. |
| 65 | +> |
| 66 | +> Please note that the above rules are not scoped to a cluster. If you would like to scope the rules to a specific cluster, see [Limiting rules to a specific cluster](../essentials/prometheus-rule-groups.md#limiting-rules-to-a-specific-cluster) for more details. |
| 67 | +> |
| 68 | +> Learn more about [Prometheus Alerts](../essentials/prometheus-rule-groups.md). |
| 69 | +> |
| 70 | +> If you want to use any other OSS prometheus alerting/recording rules please use the converter here to create the azure equivalent prometheus rules [az-prom-rules-converter](https://aka.ms/az-prom-rules-converter) |
| 71 | +
|
| 72 | + |
| 73 | +### More jmx_exporter metrics using strimzi |
| 74 | +If you are using the [strimzi operator](https://github.com/strimzi/strimzi-kafka-operator.git) for deploying the kafka clusters, deploy the pod monitors to get more jmx_exporter metrics. |
| 75 | +> [!Note] |
| 76 | +> Metrics need to be exposed by the kafka cluster deployments like the examples [here](https://github.com/strimzi/strimzi-kafka-operator/tree/main/examples/metrics). Refer to the kafka-.*-metrics.yaml files to configure metrics to be exposed. |
| 77 | +> |
| 78 | +>The pod monitors here also assume that the namespace where the kafka workload is deployed in 'kafka'. Update it accordingly if the workloads are deployed in another namespace. |
| 79 | +
|
| 80 | +```yaml |
| 81 | +apiVersion: azmonitoring.coreos.com/v1 |
| 82 | +kind: PodMonitor |
| 83 | +metadata: |
| 84 | + name: azmon-cluster-operator-metrics |
| 85 | + labels: |
| 86 | + app: strimzi |
| 87 | +spec: |
| 88 | + selector: |
| 89 | + matchLabels: |
| 90 | + strimzi.io/kind: cluster-operator |
| 91 | + namespaceSelector: |
| 92 | + matchNames: |
| 93 | + - kafka |
| 94 | + podMetricsEndpoints: |
| 95 | + - path: /metrics |
| 96 | + port: http |
| 97 | +--- |
| 98 | +apiVersion: azmonitoring.coreos.com/v1 |
| 99 | +kind: PodMonitor |
| 100 | +metadata: |
| 101 | + name: azmon-entity-operator-metrics |
| 102 | + labels: |
| 103 | + app: strimzi |
| 104 | +spec: |
| 105 | + selector: |
| 106 | + matchLabels: |
| 107 | + app.kubernetes.io/name: entity-operator |
| 108 | + namespaceSelector: |
| 109 | + matchNames: |
| 110 | + - kafka |
| 111 | + podMetricsEndpoints: |
| 112 | + - path: /metrics |
| 113 | + port: healthcheck |
| 114 | +--- |
| 115 | +apiVersion: azmonitoring.coreos.com/v1 |
| 116 | +kind: PodMonitor |
| 117 | +metadata: |
| 118 | + name: azmon-bridge-metrics |
| 119 | + labels: |
| 120 | + app: strimzi |
| 121 | +spec: |
| 122 | + selector: |
| 123 | + matchLabels: |
| 124 | + strimzi.io/kind: KafkaBridge |
| 125 | + namespaceSelector: |
| 126 | + matchNames: |
| 127 | + - kafka |
| 128 | + podMetricsEndpoints: |
| 129 | + - path: /metrics |
| 130 | + port: rest-api |
| 131 | +--- |
| 132 | +apiVersion: azmonitoring.coreos.com/v1 |
| 133 | +kind: PodMonitor |
| 134 | +metadata: |
| 135 | + name: azmon-kafka-resources-metrics |
| 136 | + labels: |
| 137 | + app: strimzi |
| 138 | +spec: |
| 139 | + selector: |
| 140 | + matchExpressions: |
| 141 | + - key: "strimzi.io/kind" |
| 142 | + operator: In |
| 143 | + values: ["Kafka", "KafkaConnect", "KafkaMirrorMaker", "KafkaMirrorMaker2"] |
| 144 | + namespaceSelector: |
| 145 | + matchNames: |
| 146 | + - kafka |
| 147 | + podMetricsEndpoints: |
| 148 | + - path: /metrics |
| 149 | + port: tcp-prometheus |
| 150 | + relabelings: |
| 151 | + - separator: ; |
| 152 | + regex: __meta_kubernetes_pod_label_(strimzi_io_.+) |
| 153 | + replacement: $1 |
| 154 | + action: labelmap |
| 155 | + - sourceLabels: [__meta_kubernetes_namespace] |
| 156 | + separator: ; |
| 157 | + regex: (.*) |
| 158 | + targetLabel: namespace |
| 159 | + replacement: $1 |
| 160 | + action: replace |
| 161 | + - sourceLabels: [__meta_kubernetes_pod_name] |
| 162 | + separator: ; |
| 163 | + regex: (.*) |
| 164 | + targetLabel: kubernetes_pod_name |
| 165 | + replacement: $1 |
| 166 | + action: replace |
| 167 | + - sourceLabels: [__meta_kubernetes_pod_node_name] |
| 168 | + separator: ; |
| 169 | + regex: (.*) |
| 170 | + targetLabel: node_name |
| 171 | + replacement: $1 |
| 172 | + action: replace |
| 173 | + - sourceLabels: [__meta_kubernetes_pod_host_ip] |
| 174 | + separator: ; |
| 175 | + regex: (.*) |
| 176 | + targetLabel: node_ip |
| 177 | + replacement: $1 |
| 178 | + action: replace |
| 179 | +``` |
| 180 | +
|
| 181 | +#### Alerts with strimzi |
| 182 | +Rich set of alerts based off of strimzi metrics can also be configured by refering to the [examples](https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/prometheus-install/prometheus-rules.yaml). |
| 183 | +
|
| 184 | +> [!NOTE] |
| 185 | +> If using any other way of exposing the jmx_exporter on your kafka cluster, please follow the instructions [here](prometheus-metrics-scrape-crd.md) on how to configure the pod or service monitors accordingly. |
| 186 | +
|
| 187 | +### Grafana Dashboards for more jmx metrics with strimzi |
| 188 | +Please also see the [grafana-dashboards-for-strimzi](https://github.com/strimzi/strimzi-kafka-operator/tree/main/examples/metrics/grafana-dashboards) to view dashboards for metrics exposed by strimzi operator. |
| 189 | +
|
| 190 | +
|
| 191 | +### Troubleshooting |
| 192 | +When the service monitors or pod monitors are successfully applied, if you want to make sure that the service monitor targets get picked up by the addon, follow the instructions [here](prometheus-metrics-troubleshoot.md#prometheus-interface). |
| 193 | +
|
0 commit comments