kafka dashboard empty due to missing a key metrics

[scottc-git]

What could be the reason the kafka_server_replicamanager_leadercount metric is missing in Prometheus? That metric is critical to render the drop down menus in the Strimzi Kafka dashboard. Since it's missing the dashboard is currently empty.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: strimzi
spec:
  kafka:
    version: 2.8.0
    replicas: 3
    jmxOptions:
      authentication:
        type: "password"
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics*
          key: kafka-metrics-config.yml

  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"
    logging: debug

  jmxTrans^:
    outputDefinitions:
      - outputType: "com.googlecode.jmxtrans.model.output.StdOutWriter"
        name: "standardOut"
      - outputType: "com.googlecode.jmxtrans.model.output.GraphiteOutputWriter"
        host: "mylogstash.com"
        port: 31028
        flushDelayInSeconds: 5
        name: "logstash"
    kafkaQueries:
      - targetMBean: "kafka.server:type=BrokerTopicMetrics,name=*"
        attributes:  ["Count"]
        outputs: ["standardOut"]
    

*: The kafka-metrics config map is exactly as in https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/kafka-metrics.yaml

^: I enabled jmxTrans but I believe it is irrelevant to the issue.

Strimzi: 0.27.1
Kafka: 2.8

[scholzj]

Did you checked your Prometheus if you have any other metrics from Kafka? I'm not using JMXTrans - I would expect it to be unrelated, but you never know. It both uses JMX.

[scottc-git]

I see these ⬇️ but not kafka_server*, @scholzj

. . .
kafka_brokers
kafka_consumergroup_current_offset
kafka_consumergroup_current_offset_sum
kafka_consumergroup_lag
kafka_consumergroup_lag_sum
kafka_consumergroup_members
kafka_exporter_build_info
kafka_topic_partition_current_offset
kafka_topic_partition_in_sync_replica
kafka_topic_partition_leader
kafka_topic_partition_leader_is_preferred
kafka_topic_partition_oldest_offset
kafka_topic_partition_replicas
kafka_topic_partition_under_replicated_partition
. . .

p.s. Per your suggestion, I also removed jmxTrans and saw the same result.

[scholzj]

These look like Kafka Exporter metrics. Not Kafka broker metrics. So maybe your Prometheus does not scrape the metrics from the Kafka brokers?

[scottc-git]

@scholzj When I port-forward the broker: 9404, I can see the missing metrics.
This means Prometheus does not know about this target and does not scrape. Only difference from your example is that I have a separate yaml file defining the 'kafka-metrics' config map. Also, I see the kafka pods do not have the prometheus annotations. Should they? What's the discovery mechanism?

[scholzj]

The examples have the PodMonitor resources for the Prometheus operator. But it is really up to you to configure and manage your Prometheus. Strimzi really just provides the Prometheus export, we are not Prometheus experts to help you with your Prometheus installation. In general, you can either use the things such as the PodMonitro resources, or you can sue the template section of the Kafka CR to add some annotations if you want etc.

[scottc-git]

That was it. I added prometheus annotations using the template section and it worked like a charm. Thanks again @scholzj !

• Happy Strimzier!