Kafka connect exposing jmx metrics

[hafizmujadidKhalid]

Hello lovely Strimzi community!
I have deployed kafka connect using strimzi and that works undoubtedly. Now, I am trying to enable monitoring for kafka connect cluster. I read jmxOptions can be enabled it. I did it but I can't access any metric on the 9999 port of the kafka-connect pod.
Here is my yaml configuration:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: ingestion-metric
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  replicas: 1
  version: 2.6.0
  image: <ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com/kafka-connect:2.1.0
  bootstrapServers: kafka-broker:12345
  metricsConfig:
    type: jmxPrometheusExporter
    valueFrom:
      configMapKeyRef:
        name: ingestion-metric
        key: ingestion-metric-config.yml
  resources:
    requests:
      cpu: 250m
      memory: 512Mi
    limits:
      cpu: 500m
      memory: 1Gi
  jmxOptions: {}
  jvmOptions:
    gcLoggingEnabled: false
  config:
    group.id: ingestion-metric

    key.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter: org.apache.kafka.connect.json.JsonConverter
    offset.storage.topic: ingestion-metric-offsets
    config.storage.topic: ingestion-metric-configs
    status.storage.topic: ingestion-metric-status
    config.storage.replication.factor: 3
    offset.storage.replication.factor: 3
    status.storage.replication.factor: 3
  logging:
    type: inline
    loggers:
      logger.name: "DEBUG"

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: ingestion-metric
  labels:
    app: strimzi
data:
  ingestion-metric-config.yml: |
    lowercaseOutputName: true
    lowercaseOutputLabelNames: true
    rules:
    #kafka.connect:type=connect-node-metrics,client-id="{clientid}",node-id="{nodeid}"
    #kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id="{nodeid}"
    - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), node-id=(.+)><>(.+-total|.+-avg)
      name: kafka_$2_$5
      labels:
        clientId: "$3"
        nodeId: "$4"
      help: "Kafka $1 JMX metric type $2"
      type: UNTYPED

    #kafka.connect:type=kafka-metrics-count,client-id="{clientid}"
    #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}"
    #kafka.consumer:type=consumer-coordinator-metrics,client-id="{clientid}"
    #kafka.consumer:type=consumer-metrics,client-id="{clientid}"
    - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.*)><>(.+-total|.+-avg|.+-bytes|.+-count|.+-ratio|.+-age|.+-flight|.+-threads|.+-connectors|.+-tasks|.+-ago)
      name: kafka_$2_$4
      labels:
        clientId: "$3"
      help: "Kafka $1 JMX metric type $2"
      type: GAUGE

    #kafka.connect:type=connector-metrics,connector="{connector}"
    - pattern: 'kafka.(.+)<type=connector-metrics, connector=(.+)><>(connector-class|connector-type|connector-version|status): (.+)'
      name: kafka_connect_connector_$3
      value: 1
      labels:
        connector: "$2"
        $3: "$4"
      help: "Kafka Connect $3 JMX metric type connector"
      type: GAUGE

    #kafka.connect:type=connector-task-metrics,connector="{connector}",task="{task}<> status"
    - pattern: 'kafka.connect<type=connector-task-metrics, connector=(.+), task=(.+)><>status: ([a-z-]+)'
      name: kafka_connect_connector_task_status
      value: 1
      labels:
        connector: "$1"
        task: "$2"
        status: "$3"
      help: "Kafka Connect JMX Connector task status"
      type: GAUGE

    #kafka.connect:type=task-error-metrics,connector="{connector}",task="{task}"
    #kafka.connect:type=source-task-metrics,connector="{connector}",task="{task}"
    #kafka.connect:type=sink-task-metrics,connector="{connector}",task="{task}"
    #kafka.connect:type=connector-task-metrics,connector="{connector}",task="{task}"
    - pattern: kafka.connect<type=(.+)-metrics, connector=(.+), task=(.+)><>(.+-total|.+-count|.+-ms|.+-ratio|.+-seq-no|.+-rate|.+-max|.+-avg|.+-failures|.+-requests|.+-timestamp|.+-logged|.+-errors|.+-retries|.+-skipped)
      name: kafka_connect_$1_$4
      labels:
        connector: "$2"
        task: "$3"
      help: "Kafka Connect JMX metric type $1"
      type: GAUGE

    #kafka.connect:type=connector-metrics,connector="{connector}"
    #kafka.connect:type=connect-worker-metrics,connector="{connector}"
    - pattern: kafka.connect<type=connect-worker-metrics, connector=(.+)><>([a-z-]+)
      name: kafka_connect_worker_$2
      labels:
        connector: "$1"
      help: "Kafka Connect JMX metric $1"
      type: GAUGE

    #kafka.connect:type=connect-worker-metrics
    - pattern: kafka.connect<type=connect-worker-metrics><>([a-z-]+)
      name: kafka_connect_worker_$1
      help: "Kafka Connect JMX metric worker"
      type: GAUGE

    #kafka.connect:type=connect-worker-rebalance-metrics
    - pattern: kafka.connect<type=connect-worker-rebalance-metrics><>([a-z-]+)
      name: kafka_connect_worker_rebalance_$1
      help: "Kafka Connect JMX metric rebalance information"
      type: GAUGE

The pod looks fine from strimzi operator logs and kafka connect logs. But still I can't see any metrics. I entered into kafkaconnect pod and executed the curl localhost:9999/metrics command. It says connection refused. I don't have any Prometheus yet. Am I missing something? Do I need to install something else? Thanks

[scholzj]

I think you are mixing two different things:

• jmxOptions configures JMX access on port 9999. If you enable it, you can access the JMX MBeans using VisualVM or Jconsole.
• metricsConfig enables Prometheus metrics on port 9404. That is what you can scrape using Prometheus or using curl (but on port 9404).

So you would normally pick one. If you want metrics, we recommend Prometheus over JMX -> so you can remove the jmxOptions, keep the metricsConfig and get the metrics on 9404.

[hafizmujadidKhalid]

So, metrics are available inside the pod on port 9404 and i should use prometheous scraper to scrape pod for metrics? Out of curiosity, Even though, I enabled both, shouldn't jmx metrics be available on 9999 port of service? thanks already for quick response.

[scholzj]

The JMX 9999 port is IMHO added to the service (well, I did not tried it right now as I do not use JMX - but the code say it should be added). Although I think it is not really too useful if you have more Connect nodes because the service will lead you to one random pod basically.

The 9404 port for the Prometheus metrics is not exposed on the service.

[hafizmujadidKhalid]

Thanks, I have checked service definition and jmx port 9999 was leading me to access jmx mbeans on 9999.

[scholzj]

Yes, it would give you the JMX access (i.e. it matches the jmxOptions).