Exposing Debezium plugin JMX metrics on kafka connect cluster in Strimzi

[davidnateberg]

Hello, I am currently deploying strimzi in a kubernetes cluster, with the following configuration.

strimzi version: 0.32.0
k8s cluster version: 1.23
helm version: 3.7.1

I have installed the operator with the default values, and everything behaves as expected, except that I am trying to expose metrics from a Debezium postgresql connector connect plugin that is running in the kafka connect cluster. https://debezium.io/documentation/reference/stable/connectors/postgresql.html.

Here is the Debezium documentation for exposing metrics. https://debezium.io/documentation/reference/stable/operations/monitoring.html. I have tried setting the various environment variables on the connect pods as mentioned in the documentation with no luck, though I'm not even sure if that is necessary. I thought these mBeans should be exposed on the connect cluster automatically.

The metrics config I have is the default connect metrics config in the strimzi examples folder, with patters for the debezium plugin included. It looks like this

kind: ConfigMap
apiVersion: v1
metadata:
  name: connect-metrics
  labels:
    app: strimzi
data:
  metrics-config.yml: |
    # Inspired by kafka-connect rules
    # https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-connect.yml
    # See https://github.com/prometheus/jmx_exporter for more info about JMX Prometheus Exporter metrics
    # Engine Application Versioning Info
    lowercaseOutputName: true
    lowercaseOutputLabelNames: true
    startDelaySeconds: 0
    rules:
    #kafka.connect:type=app-info,client-id="{clientid}"
    #kafka.consumer:type=app-info,client-id="{clientid}"
    #kafka.producer:type=app-info,client-id="{clientid}"
    - pattern: 'kafka.(.+)<type=app-info, client-id=(.+)><>start-time-ms'
      name: kafka_$1_start_time_seconds
      labels:
        clientId: "$2"
      help: "Kafka $1 JMX metric start time seconds"
      type: GAUGE
      valueFactor: 0.001
    - pattern: 'kafka.(.+)<type=app-info, client-id=(.+)><>(commit-id|version): (.+)'
      name: kafka_$1_$3_info
      value: 1
      labels:
        clientId: "$2"
        $3: "$4"
      help: "Kafka $1 JMX metric info version and commit-id"
      type: GAUGE
    #kafka.producer:type=producer-topic-metrics,client-id="{clientid}",topic="{topic}"", partition="{partition}"
    #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}",topic="{topic}"", partition="{partition}"
    - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), topic=(.+), partition=(.+)><>(.+-total|compression-rate|.+-avg|.+-replica|.+-lag|.+-lead)
      name: kafka_$2_$6
      labels:
        clientId: "$3"
        topic: "$4"
        partition: "$5"
      help: "Kafka $1 JMX metric type $2"
      type: GAUGE
    #kafka.producer:type=producer-topic-metrics,client-id="{clientid}",topic="{topic}"
    #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}",topic="{topic}"", partition="{partition}"
    - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), topic=(.+)><>(.+-total|compression-rate|.+-avg)
      name: kafka_$2_$5
      labels:
        clientId: "$3"
        topic: "$4"
      help: "Kafka $1 JMX metric type $2"
      type: GAUGE
    #kafka.connect:type=connect-node-metrics,client-id="{clientid}",node-id="{nodeid}"
    #kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id="{nodeid}"
    - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), node-id=(.+)><>(.+-total|.+-avg)
      name: kafka_$2_$5
      labels:
        clientId: "$3"
        nodeId: "$4"
      help: "Kafka $1 JMX metric type $2"
      type: UNTYPED
    #kafka.connect:type=kafka-metrics-count,client-id="{clientid}"
    #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}"
    #kafka.consumer:type=consumer-coordinator-metrics,client-id="{clientid}"
    #kafka.consumer:type=consumer-metrics,client-id="{clientid}"
    - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.*)><>(.+-total|.+-avg|.+-bytes|.+-count|.+-ratio|.+-age|.+-flight|.+-threads|.+-connectors|.+-tasks|.+-ago)
      name: kafka_$2_$4
      labels:
        clientId: "$3"
      help: "Kafka $1 JMX metric type $2"
      type: GAUGE
    #kafka.connect:type=connector-metrics,connector="{connector}"
    - pattern: 'kafka.(.+)<type=connector-metrics, connector=(.+)><>(connector-class|connector-type|connector-version|status): (.+)'
      name: kafka_connect_connector_$3
      value: 1
      labels:
        connector: "$2"
        $3: "$4"
      help: "Kafka Connect $3 JMX metric type connector"
      type: GAUGE
    #kafka.connect:type=connector-task-metrics,connector="{connector}",task="{task}<> status"
    - pattern: 'kafka.connect<type=connector-task-metrics, connector=(.+), task=(.+)><>status: ([a-z-]+)'
      name: kafka_connect_connector_task_status
      value: 1
      labels:
        connector: "$1"
        task: "$2"
        status: "$3"
      help: "Kafka Connect JMX Connector task status"
      type: GAUGE
    #kafka.connect:type=task-error-metrics,connector="{connector}",task="{task}"
    #kafka.connect:type=source-task-metrics,connector="{connector}",task="{task}"
    #kafka.connect:type=sink-task-metrics,connector="{connector}",task="{task}"
    #kafka.connect:type=connector-task-metrics,connector="{connector}",task="{task}"
    - pattern: kafka.connect<type=(.+)-metrics, connector=(.+), task=(.+)><>(.+-total|.+-count|.+-ms|.+-ratio|.+-seq-no|.+-rate|.+-max|.+-avg|.+-failures|.+-requests|.+-timestamp|.+-logged|.+-errors|.+-retries|.+-skipped)
      name: kafka_connect_$1_$4
      labels:
        connector: "$2"
        task: "$3"
      help: "Kafka Connect JMX metric type $1"
      type: GAUGE
    #kafka.connect:type=connector-metrics,connector="{connector}"
    #kafka.connect:type=connect-worker-metrics,connector="{connector}"
    - pattern: kafka.connect<type=connect-worker-metrics, connector=(.+)><>([a-z-]+)
      name: kafka_connect_worker_$2
      labels:
        connector: "$1"
      help: "Kafka Connect JMX metric $1"
      type: GAUGE
    #kafka.connect:type=connect-worker-metrics
    - pattern: kafka.connect<type=connect-worker-metrics><>([a-z-]+)
      name: kafka_connect_worker_$1
      help: "Kafka Connect JMX metric worker"
      type: GAUGE
    #kafka.connect:type=connect-worker-rebalance-metrics
    - pattern: kafka.connect<type=connect-worker-rebalance-metrics><>([a-z-]+)
      name: kafka_connect_worker_rebalance_$1
      help: "Kafka Connect JMX metric rebalance information"
      type: GAUGE
    - pattern : "kafka.connect<type=connect-worker-metrics>([^:]+):"
      name: "kafka_connect_worker_metrics_$1"
    - pattern : "kafka.connect<type=connect-metrics, client-id=([^:]+)><>([^:]+)"
      name: "kafka_connect_metrics_$2"
      labels:
        client: "$1"
    - pattern: "debezium.([^:]+)<type=connector-metrics, context=([^,]+), server=([^,]+), key=([^>]+)><>RowsScanned"
      name: "debezium_metrics_RowsScanned"
      labels:
        plugin: "$1"
        name: "$3"
        context: "$2"
        table: "$4"
    - pattern: "debezium.([^:]+)<type=connector-metrics, context=([^,]+), server=([^>]+)>([^:]+)"
      name: "debezium_metrics_$4"
      labels:
        plugin: "$1"
        name: "$3"
        context: "$2"
    

I am able to see all the kafka connect metrics in prometheus as expected, or if exec into a pod and curl localhost:9094/metrics I can see them there as well. However, there are no Debzium metrics exposed. I copied the patterns for the Debezium plugin pattern from their documentation here https://github.com/debezium/debezium-examples/blob/main/monitoring/debezium-jmx-exporter/config.yml

The kafka connect resource manifest looks like this, though I dont think its an issue with the cluster because everything else works as expected

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: snowpipe-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  image: {{ .Values.kafkaConnect.image }}
  replicas: 3
  bootstrapServers: {{ .Values.bootstrapServers }}
  config:
    rest.port: {{ .Values.kafkaConnect.restPort | default "8083" }}
    group.id: {{ .Values.kafkaConnect.groupId | default "snowpipe" }}
    config.storage.topic: {{ .Values.kafkaConnect.configStorageTopic | default "config" }}
    offset.storage.topic: {{ .Values.kafkaConnect.offsetStorageTopic | default "offset" }}
    status.storage.topic: {{ .Values.kafkaConnect.statusStorageTopic| default "status" }}
    key.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter: org.apache.kafka.connect.json.JsonConverter
    internal.key.converter: org.apache.kafka.connect.json.JsonConverter
    internal.value.converter: org.apache.kafka.connect.json.JsonConverter
    config.storage.replication.factor: {{ .Values.kafkaConnect.configStorageReplicationFactor | default "3" }}
    offset.storage.replication.factor: {{ .Values.kafkaConnect.offsetStorageReplicationFactor | default "3" }}
    status.storage.replication.factor: {{ .Values.kafkaConnect.statusStorageReplicationFactor | default "3" }}
    offset.flush.internval.ms: {{ .Values.kafkaConnect.offsetFlushInterval | default "1000" }}
    config.providers: env
    config.providers.env.class: io.strimzi.kafka.EnvVarConfigProvider
    connector.client.config.override.policy: All
  externalConfiguration:
    env:
      - name: WRITE_DB_HOST
        valueFrom:
          secretKeyRef:
            name: strimzi-kafka-connect
            key: DB_WRITE_HOST
      - name: READ_DB_HOST
        valueFrom:
          secretKeyRef:
            name: strimzi-kafka-connect
            key: DB_READ_HOST
      - name: DEBEZIUM_PG_USER
        valueFrom:
          secretKeyRef:
            name: strimzi-kafka-connect
            key: DEBEZIUM_PG_USER
      - name: DEBEZIUM_PG_PASS
        valueFrom:
          secretKeyRef:
            name: strimzi-kafka-connect
            key: DEBEZIUM_PG_PASS
      - name: AWS_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: strimzi-kafka-connect
            key: AWS_ACCESS_KEY_ID
      - name: AWS_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: strimzi-kafka-connect
            key: AWS_SECRET_ACCESS_KEY
  resources:
    requests:
      cpu: "1"
      memory: 2Gi
    limits:
      cpu: "1"
      memory: 2Gi
  metricsConfig:
    type: jmxPrometheusExporter
    valueFrom:
      configMapKeyRef:
        name: connect-metrics
        key: metrics-config.yml

Any help would be greatly appreciated. To reiterate, I am trying to get JMX metrics from a kafka connect plugin, in this case the Debeziuim postgres plugin, exposed on the kafka connect cluster using the Prometheus JMX exporter. Thank you.

[scholzj]

Have you tried to not mix the conflicting configurations and add just the debezium.* metrics to the Strimzi metrics configuration? Or use only the Debezium metrics configuration? You can also check the operator logs to see if they report something.

[davidnateberg]

I can look at the operator logs. I did try just deploying the debezium rules. I also tried deploying with no rules so that it should expose everything available. But no luck.

[davidnateberg]

I get the feeling this issue is to niche/vague for anyone to necessarily help with, I did look at operator logs, and kafka connect cluster logs, with the debug log level, but didn't find any logs related to JMX or those metrics

[scholzj]

I guess the question is whether you have no exceptions in the operator logs and whether your KafkaConnect CR is ready and has the .status.observedGeneration equal to .metadata.generation which means it is up to date. There could be some unrelated issue or there can be some issue parsing the YAML with the metrics configuration etc. I'm afraid I do not know much about Debezium. So I cannot help much beyond that.

[ee07dazn]

Hi @davidnateberg : I am working on the same stuff and have come across the exact same issue. In my case everything seems to be running normally ie .status.observedGeneration = .metadata.gemeration = 1 . So no issues on that side, I will try Jakub's suggestion on just adding the debezium config to see if this is due to some contradiction or something.
Since we are working on the same issue, might be good if we can update each other if we make some progress.

[davidnateberg]

@ee07dazn well that makes me feel a little better! I've been banging my head against the wall with this issue for days. At this point I've given up on Debezium metrics because I needed to get moving on our deployment, at least having the kafka connect metrics is nice. I tried deploying with only the Debezium config, however, I truly don't think it is an issue with the metrics config. I have deployed with the following pattern

rules:
- pattern: ".*"

which shouldn't filter out any metrics, so anything thats exposed should be visible, including Debezium. But there are no debezium metrics if I curl localhost:9404 on the kafka connect cluster pod. So this leads me to believe it is an issue with Debezium itself, not exposing metrics. I've tried setting the variables specified here on the connect pods using the templating option, but still no luck.
https://debezium.io/documentation/reference/stable/operations/monitoring.html#kafka-connect-jmx-environment-variables
@scholzj are you able to confirm for that, the way these plugins are run with strimzi, it would be considered bare metal yes? It's been difficult for me to track what script is being run to configure the connect pods, and what env variables that sets. But I believe it would be this one

which sets this env variable

# enabling Prometheus JMX exporter as Java agent
if [ "$KAFKA_CONNECT_METRICS_ENABLED" = "true" ]; then
    KAFKA_OPTS="${KAFKA_OPTS} -javaagent:$(ls "$KAFKA_HOME"/libs/jmx_prometheus_javaagent*.jar)=9404:$KAFKA_HOME/custom-config/metrics-config.json"
    export KAFKA_OPTS
fi

but doesn't set any of the other environment variables that Debezium claims to require to expose metrics, i.e.
JMX_PORT and KAFKA_JMX_OPTS

I don't know if that's related, I'm really at a loss here

[scholzj]

@scholzj are you able to confirm for that, the way these plugins are run with strimzi, it would be considered bare metal yes? It's been difficult for me to track what script is being run to configure the connect pods, and what env variables that sets. But I believe it would be this one

which sets this env variable

# enabling Prometheus JMX exporter as Java agent
if [ "$KAFKA_CONNECT_METRICS_ENABLED" = "true" ]; then
    KAFKA_OPTS="${KAFKA_OPTS} -javaagent:$(ls "$KAFKA_HOME"/libs/jmx_prometheus_javaagent*.jar)=9404:$KAFKA_HOME/custom-config/metrics-config.json"
    export KAFKA_OPTS
fi

but doesn't set any of the other environment variables that Debezium claims to require to expose metrics, i.e. JMX_PORT and KAFKA_JMX_OPTS

I don't know if that's related, I'm really at a loss here

I'm not sure what do you mean with being considered bare metal. Strimzi runs on Kubernetes. You have two options in Strimzi's Kafka Connect:

• Configure the Prometheus metrics with the Prometheus JMX Exporter
• Expose JMX via the KafkaConnect CR (check the Strimzi docs for that). Then you can use some JMX client to connect to it.

In neither of those ways are you expected to set any environment variables. You just configure the custom resources. I'm afraid I do not know how it is implemented in Debezium -> there is no real metrics unification for Kafka Connect connectors. That is currently only proposed in KIP-877. But right now it is up to each connector.

[davidnateberg]

@scholzj are you able to confirm for that, the way these plugins are run with strimzi, it would be considered bare metal yes? It's been difficult for me to track what script is being run to configure the connect pods, and what env variables that sets. But I believe it would be this one

which sets this env variable

enabling Prometheus JMX exporter as Java agent

if [ "$KAFKA_CONNECT_METRICS_ENABLED" = "true" ]; then

KAFKA_OPTS="${KAFKA_OPTS} -javaagent:$(ls "$KAFKA_HOME"/libs/jmx_prometheus_javaagent*.jar)=9404:$KAFKA_HOME/custom-config/metrics-config.json"

export KAFKA_OPTS

fi

but doesn't set any of the other environment variables that Debezium claims to require to expose metrics, i.e. JMX_PORT and KAFKA_JMX_OPTS

I don't know if that's related, I'm really at a loss here

I'm not sure what do you mean with being considered bare metal. Strimzi runs on Kubernetes. You have two options in Strimzi's Kafka Connect:

• Configure the Prometheus metrics with the Prometheus JMX Exporter

• Expose JMX via the KafkaConnect CR (check the Strimzi docs for that). Then you can use some JMX client to connect to it.

In neither of those ways are you expected to set any environment variables. You just configure the custom resources. I'm afraid I do not know how it is implemented in Debezium -> there is no real metrics unification for Kafka Connect connectors. That is currently only proposed in KIP-877. But right now it is up to each connector.

Yeah it was kind of a silly question. But I was just trying to sus out which set of env variables would be appropriate based on the Debezium docs I linked. Those docs reference a specific docker image for debezium and tell you which env variables need to be set to expose metrics for that plugin. As a theory I thought maybe those have to be set on the Kafka connect pod to properly expose the plug-in metrics. I have set the config as you mentioned using the connect CR. It works as expected but only seems to expose Kafka connect metrics and metrics for plugins. This is not necessarily Debezium specific. At least I don't think so. But I understand it's a random plug-in, and I wouldn't expect you to troubleshoot that. But I guess I would be curious if there's any examples of people exposing other metrics from plugins besides the Kafka connect metrics.

[ee07dazn]

Hi @davidnateberg : literally just got today to look into it after I last pinged but basically, with ".*" option, I am able to see all the debezium metrics. I will continue to look into how i can replcae ".*" with something sensible. But for now, some progress :)

[davidnateberg]

Interesting, do you mean you set a rule to expose all the metrics. So your only rule is ".*"? Because I tried that as well with no luck. But I may have been misunderstanding how to view what metrics were exposed in Prometheus and it was actually there and I just missed it.

[davidnateberg]

@ee07dazn I'm also curious if you're deploying a kafka cluster with your strimzi setup as well, because i am pointing my kafka connect cluster at a kafka cluster hosted elsewhere, in AWS, and I'm wondering if that has anything to do with it.

[ee07dazn]

@davidnateberg : yup, just .* (see attached)

I am not using strimzi kafka cluster but redpanda BYOC version deployed in GCP.

[ee07dazn]

@davidnateberg - couple of things :

1. try and use the latest version of deb - the one I am using is 2.1.2
2. with this, even "debezium.*" works

[ee07dazn]

[jslusher]

@ee07dazn and @davidnateberg I'm running a Debezium connector in a Strimzi KafkaConnect and my Kafka cluster is also managed by the Strimzi operator. I have the wide open configuration for my jmx configuration and I see the other connect metrics but I do not see any Debezium metrics.

  kafka-connect-metrics.yml: |
    lowercaseOutputName: true
    lowercaseOutputLabelNames: true
    rules:
    - pattern: ".*"

Searching for them on the connect pods using this command:

curl http://localhost:9404/metrics | grep debezium

I'm also on the latest stable version of Debezium. @davidnateberg did you ever get the metrics to surface?

[jslusher]

We're using Datadog and their implementation of Prometheus called openmetrics, which I'm under the impression should do the same thing as a Prometheus server which is scrape metrics from a given endpoint. I would think that if the metrics aren't showing up in the curl output on port 9404 then Prometheus/openmetrics would also not find them there. I'll try this configuration and let you know if it works.

[jslusher]

It turns out in my case that I was looking at the wrong connect pod when trying to curl for the debezium metrics. The metrics will only surface on the pod hosting the worker hosting the connector, and since I only have one connector so far, the metrics were only on one pod/worker and it was not the pod/worker I was looking at. Anyone who finds this in the future, be sure you're looking at the pod/worker that is hosting your debezium connector, which you can see in the output of kubectl get kctr -o yaml. Thanks for the help @davidnateberg @scholzj !

[scholzj]

You would definitely need to see them in the curl command on the Pod regardless what is your Prometheus implementation. But keep in mind that if you have multiple Connect Pods, each has its own metrics. So you need to check the one where the connector is running.

[scholzj]

The metrics will only surface on the pod hosting the worker hosting the connector, and since I only have one connector so far, the metrics were only on one pod/worker and it was not the pod/worker I was looking at.

Hehe, it came to my mind few seconds later :-D.

[davidnateberg]

It turns out in my case that I was looking at the wrong connect pod when trying to curl for the debezium metrics. The metrics will only surface on the pod hosting the worker hosting the connector, and since I only have one connector so far, the metrics were only on one pod/worker and it was not the pod/worker I was looking at. Anyone who finds this in the future, be sure you're looking at the pod/worker that is hosting your debezium connector, which you can see in the output of kubectl get kctr -o yaml. Thanks for the help @davidnateberg @scholzj !

@jslusher ah that makes complete sense 🤦 And I wasn't saying our Prometheus deployment was the reason I could see metrics, just that I could in Prometheus so I stopped caring too much about why I couldn't see them at the pod level. But this makes complete sense, I'm sure I was just looking at the metrics on the wrong pod. It makes sense why the would be in Prometheus though because Prometheus was monitoring all the connect pods in the cluster.

[davidnateberg]

Closing this discussion after @jslusher pointed out the metrics only being available on certain pods. I think that's the underlying issue here if anyone else comes along and reads this discussion

https://github.com/orgs/strimzi/discussions/7914#discussioncomment-5676210