Kafka connect metrics, how to expose kafka connector status ?

[ialexivy]

Hi,

I'm using the strimzi operator and created a Kafka Connect cluster with some connectors and exposed metrics as described in this example:
https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/kafka-connect-metrics.yaml
There is a metric stat that exposes the connector task status:

  #kafka.connect:type=connector-task-metrics,connector="{connector}",task="{task}<> status"
  - pattern: 'kafka.connect<type=connector-task-metrics, connector=(.+), task=(.+)><>status: ([a-z-]+)'
    name: kafka_connect_connector_status
    value: 1
    labels:
      connector: "$1"
      task: "$2"
      status: "$3"
    help: "Kafka Connect JMX Connector status"
    type: GAUGE

However when the connector fails to start none of the metrics being exposed, in Kafka Connect rest API you can expose connector status that returns connector state:

GET /connectors/my-connector/status
https://docs.confluent.io/platform/current/connect/references/restapi.html#get--connectors-(string-name)-status
"connector": { "state": "RUNNING", "worker_id": "fakehost:8083" }

I'm wondering if it's possible to get the same metric from the JMX exporter, looking in confluent official docs(https://docs.confluent.io/home/connect/monitoring.html#connector-metrics), looks like that there is MBean kafka.connect:type=connector-metrics,connector=”{([-.w]+) which should export it, can anyone help me to expose this metric ?

[scholzj]

The JMX Exporter just exposes the existing Kafka JMX metrics. That means that just because something is available on the REST API it might not be available as JMX metric. But if Kafka provides such metric and it is not currently exposed by the example JMX Exporter configuration, you can update the exporter configuration to expose it (and open a PR with it ideally) or open an enhancement issue and someone else might have a look at it.

[ialexivy]

@scholzj Of course(i just mentioned the REST API so it would be clear what I'm trying to achieve here). But that is exactly my issue I can't get it to work, according to Kafka docs this should be available in JMX, maybe I need some additional configuration to make it work, I tried multiple patterns but nothing has been exported, so I'm not sure what I'm missing here

[scholzj]

Hmm ... Well, I can try to have a look if I find something out. Did you found it also somewhere in Apache Kafka documentation? Or only in the Confluent docs? Maybe it is some private Confluent feature not available in Apache Kafka?

[ialexivy]

I was only able to find these docs https://docs.confluent.io/home/connect/monitoring.html#connector-metrics. You have a good point confluent use their own metric exporter (metric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter) and strimzi by default uses JMX. But as all other metrics that are described in confluent docs being exposed, i don't believe that that is the case, also is not clear as is such basic metrics not available, how other people are monitoring the state of connector today

[ialexivy]

@scholzj found the correct pattern and added new metrics, created pull request below #5565