Inconsistency between queries in dashboard examples/metrics/grafana-dashboards/strimzi-kafka-connect.json and metrics published by Strimzi Kafka Connect

[play-io]

Hi.

Problem:
Dashboard examples/metrics/grafana-dashboards/strimzi-kafka-connect.json shows N/A or nothing only.

Observations:

• By opening dashboard examples/metrics/grafana-dashboards/strimzi-kafka-connect.json you could observe prometheus query
  sum(kafka_connect_worker_connector_count{strimzi_io_kind=~"KafkaConnect.*",strimzi_io_cluster="$strimzi_connect_cluster_name"})

• By querying Prometheus on scraped metrics from Connect you could see metrics with name kafka_connect_worker_connector_count but it have neither tag strimzi_io_kind nor strimzi_io_cluster

This is what could be observed

kafka_connect_worker_connector_count{container="strimzi-kafka-connect-connect", endpoint="tcp-prometheus", instance="10.44.0.71:9404", job="develop/strimzi-kafka-connect-metrics", namespace="develop", pod="strimzi-kafka-connect-connect-6b9c774fdc-wn6s5"}

• There same could be observed by exec'ing into kafka connect and curl metrics

bash-4.4$ curl localhost:9404/metrics -s | grep kafka_connect_worker_connector_count
# HELP kafka_connect_worker_connector_count Kafka Connect JMX metric worker
# TYPE kafka_connect_worker_connector_count gauge
kafka_connect_worker_connector_count 2.0

Metrics config is as following

  metrics-config.yml: |-
    # Inspired by kafka-connect rules
    # https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-connect.yml
    # See https://github.com/prometheus/jmx_exporter for more info about JMX Prometheus Exporter metrics
    lowercaseOutputName: true
    lowercaseOutputLabelNames: true
    rules:
    #kafka.connect:type=app-info,client-id="{clientid}"
    #kafka.consumer:type=app-info,client-id="{clientid}"
    #kafka.producer:type=app-info,client-id="{clientid}"
    - pattern: 'kafka.(.+)<type=app-info, client-id=(.+)><>start-time-ms'
      name: kafka_$1_start_time_seconds
      labels:
        clientId: "$2"
      help: "Kafka $1 JMX metric start time seconds"
      type: GAUGE
      valueFactor: 0.001
    - pattern: 'kafka.(.+)<type=app-info, client-id=(.+)><>(commit-id|version): (.+)'
      name: kafka_$1_$3_info
      value: 1
      labels:
        clientId: "$2"
        $3: "$4"
      help: "Kafka $1 JMX metric info version and commit-id"
      type: GAUGE
    #kafka.producer:type=producer-topic-metrics,client-id="{clientid}",topic="{topic}"", partition="{partition}"
    #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}",topic="{topic}"", partition="{partition}"
    - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), topic=(.+), partition=(.+)><>(.+-total|compression-rate|.+-avg|.+-replica|.+-lag|.+-lead)
      name: kafka_$2_$6
      labels:
        clientId: "$3"
        topic: "$4"
        partition: "$5"
      help: "Kafka $1 JMX metric type $2"
      type: GAUGE
    #kafka.producer:type=producer-topic-metrics,client-id="{clientid}",topic="{topic}"
    #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}",topic="{topic}"", partition="{partition}"
    - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), topic=(.+)><>(.+-total|compression-rate|.+-avg)
      name: kafka_$2_$5
      labels:
        clientId: "$3"
        topic: "$4"
      help: "Kafka $1 JMX metric type $2"
      type: GAUGE
    #kafka.connect:type=connect-node-metrics,client-id="{clientid}",node-id="{nodeid}"
    #kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id="{nodeid}"
    - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), node-id=(.+)><>(.+-total|.+-avg)
      name: kafka_$2_$5
      labels:
        clientId: "$3"
        nodeId: "$4"
      help: "Kafka $1 JMX metric type $2"
      type: UNTYPED
    #kafka.connect:type=kafka-metrics-count,client-id="{clientid}"
    #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}"
    #kafka.consumer:type=consumer-coordinator-metrics,client-id="{clientid}"
    #kafka.consumer:type=consumer-metrics,client-id="{clientid}"
    - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.*)><>(.+-total|.+-avg|.+-bytes|.+-count|.+-ratio|.+-age|.+-flight|.+-threads|.+-connectors|.+-tasks|.+-ago)
      name: kafka_$2_$4
      labels:
        clientId: "$3"
      help: "Kafka $1 JMX metric type $2"
      type: GAUGE
    #kafka.connect:type=connector-metrics,connector="{connector}"
    - pattern: 'kafka.(.+)<type=connector-metrics, connector=(.+)><>(connector-class|connector-type|connector-version|status): (.+)'
      name: kafka_connect_connector_$3
      value: 1
      labels:
        connector: "$2"
        $3: "$4"
      help: "Kafka Connect $3 JMX metric type connector"
      type: GAUGE
    #kafka.connect:type=connector-task-metrics,connector="{connector}",task="{task}<> status"
    - pattern: 'kafka.connect<type=connector-task-metrics, connector=(.+), task=(.+)><>status: ([a-z-]+)'
      name: kafka_connect_connector_task_status
      value: 1
      labels:
        connector: "$1"
        task: "$2"
        status: "$3"
      help: "Kafka Connect JMX Connector task status"
      type: GAUGE
    #kafka.connect:type=task-error-metrics,connector="{connector}",task="{task}"
    #kafka.connect:type=source-task-metrics,connector="{connector}",task="{task}"
    #kafka.connect:type=sink-task-metrics,connector="{connector}",task="{task}"
    #kafka.connect:type=connector-task-metrics,connector="{connector}",task="{task}"
    - pattern: kafka.connect<type=(.+)-metrics, connector=(.+), task=(.+)><>(.+-total|.+-count|.+-ms|.+-ratio|.+-avg|.+-failures|.+-requests|.+-timestamp|.+-logged|.+-errors|.+-retries|.+-skipped)
      name: kafka_connect_$1_$4
      labels:
        connector: "$2"
        task: "$3"
      help: "Kafka Connect JMX metric type $1"
      type: GAUGE
    #kafka.connect:type=connector-metrics,connector="{connector}"
    #kafka.connect:type=connect-worker-metrics,connector="{connector}"
    - pattern: kafka.connect<type=connect-worker-metrics, connector=(.+)><>([a-z-]+)
      name: kafka_connect_worker_$2
      labels:
        connector: "$1"
      help: "Kafka Connect JMX metric $1"
      type: GAUGE
    #kafka.connect:type=connect-worker-metrics
    - pattern: kafka.connect<type=connect-worker-metrics><>([a-z-]+)
      name: kafka_connect_worker_$1
      help: "Kafka Connect JMX metric worker"
      type: GAUGE
    #kafka.connect:type=connect-worker-rebalance-metrics
    - pattern: kafka.connect<type=connect-worker-rebalance-metrics><>([a-z-]+)
      name: kafka_connect_worker_rebalance_$1
      help: "Kafka Connect JMX metric rebalance information"
      type: GAUGE

[scholzj]

I think this is something you need to configure in your Prometheus to attach these labels when scraping the configs. Is that right @ppatierno @sknot-rh?

[play-io]

Yes, this
https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/prometheus-install/strimzi-pod-monitor.yaml#L71-L74
helped.

Originally I created my own custom PodMonitor and not the one from the link because I overlooked the one dedicated for Kafka Connect and created my own.

[ghakfoort]

the Kafka-connect Grafana dashboard ( https://github.com/strimzi/strimzi-kafka-operator/blob/release-0.32.x/examples/metrics/grafana-dashboards/strimzi-kafka-connect.json ) still does not work in the 0.32.x release branch. Kafka-connect exposes metrics with different names. does anyone know what's the status of it?

[scholzj]

@ghakfoort Why does not work for you?

[aj84276]

@scholzj for similar case, i am trying to add few annotations to kafka connect pods but unable to add them how to do it?

I tried below but this are added to kind kafkaconnect but not to pods of it

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: test
  annotations:
    strimzi.io/use-connector-resources: "true"
    prometheus.io/path: /
    prometheus.io/port: '9404'
    prometheus.io/scrape: 'true'

[scholzj]

If you want to set custom annotations to the Connect Pods, the best way to do it is in the KafkaConnect custom resource in the .spec.template.pod section. There should be more details on the structure in the docs. You can use the template section to also add the custom annotations to the service etc. if needed.