KafkaConnect crashes with java.io.FileNotFoundException: /opt/kafka/custom-config/metrics-config.yml

[lkoniecz]

Describe the bug
Upgraded strimzi 0.22 to 0.26

KafkaConnect pod fails with

Exception in thread "main" java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
Caused by: java.io.FileNotFoundException: /opt/kafka/custom-config/metrics-config.yml (No such file or directory)
	at java.base/java.io.FileInputStream.open0(Native Method)
	at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
	at java.base/java.io.FileReader.<init>(FileReader.java:75)
	at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.<init>(JmxCollector.java:74)
	at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:29)
	... 6 more
*** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at ./src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422
FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed

Issue was reported earlier: #5512 but was left unresolved.

To Reproduce
Upgrade strimzi to 0.26

Expected behavior
KafkaConnect does not crash

Environment (please complete the following information):

• Strimzi version: 2.6.0
• Installation method: helm3
• Kubernetes cluster: 1.21
• Infrastructure: Amazon EKS

YAML files and logs

KafkaConnect:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: msk-cluster-connect
  namespace: kafka-tools
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 2.6.0
  image: my-image
  replicas: 3
  bootstrapServers: my-kafka-url
  resources:
    requests:
      memory: 2048Mi
      cpu: 400m
    limits:
      memory: 2560Mi
      cpu: 1000m
  config:
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    key.converter: org.apache.kafka.connect.storage.StringConverter
    value.converter: io.confluent.connect.protobuf.ProtobufConverter
    value.converter.schema.registry.url: "http://schema-registry-sr:8081"
    key.converter.schemas.enable: false
    value.converter.schemas.enable: false
    offset.flush.interval.ms: 1000
  template:
    pod:
      metadata:
        annotations:
          prometheus.io/scrape: 'true'
          prometheus.io/port: '9404'
  metricsConfig:
    type: jmxPrometheusExporter
    valueFrom:
      configMapKeyRef:
        name: jmx-exporter-metrics-config
        key: metrics-config.yml

Metrics

apiVersion: v1
kind: ConfigMap
metadata:
  name: jmx-exporter-metrics-config
  namespace: kafka-tools
  labels:
    app: strimzi
data:
  metrics-config.yml: |
    # Inspired by kafka-connect rules
    # https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-connect.yml
    lowercaseOutputName: true
    lowercaseOutputLabelNames: true
    rules:
      #kafka.connect:type=app-info,client-id="{clientid}"
      #kafka.consumer:type=app-info,client-id="{clientid}"
      #kafka.producer:type=app-info,client-id="{clientid}"
      - pattern: 'kafka.(.+)<type=app-info, client-id=(.+)><>start-time-ms'
        name: kafka_$1_start_time_seconds
        labels:
          clientId: "$2"
        help: "Kafka $1 JMX metric start time seconds"
        type: GAUGE
        valueFactor: 0.001
      - pattern: 'kafka.(.+)<type=app-info, client-id=(.+)><>(commit-id|version): (.+)'
        name: kafka_$1_$3_info
        value: 1
        labels:
          clientId: "$2"
          $3: "$4"
        help: "Kafka $1 JMX metric info version and commit-id"
        type: GAUGE

      #kafka.producer:type=producer-topic-metrics,client-id="{clientid}",topic="{topic}"", partition="{partition}"
      #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}",topic="{topic}"", partition="{partition}"
      - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), topic=(.+), partition=(.+)><>(.+-total|compression-rate|.+-avg|.+-replica|.+-lag|.+-lead)
        name: kafka_$2_$6
        labels:
          clientId: "$3"
          topic: "$4"
          partition: "$5"
        help: "Kafka $1 JMX metric type $2"
        type: GAUGE

      #kafka.producer:type=producer-topic-metrics,client-id="{clientid}",topic="{topic}"
      #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}",topic="{topic}"", partition="{partition}"
      - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), topic=(.+)><>(.+-total|compression-rate|.+-avg)
        name: kafka_$2_$5
        labels:
          clientId: "$3"
          topic: "$4"
        help: "Kafka $1 JMX metric type $2"
        type: GAUGE

      #kafka.connect:type=connect-node-metrics,client-id="{clientid}",node-id="{nodeid}"
      #kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id="{nodeid}"
      - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), node-id=(.+)><>(.+-total|.+-avg)
        name: kafka_$2_$5
        labels:
          clientId: "$3"
          nodeId: "$4"
        help: "Kafka $1 JMX metric type $2"
        type: UNTYPED

      #kafka.connect:type=kafka-metrics-count,client-id="{clientid}"
      #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}"
      #kafka.consumer:type=consumer-coordinator-metrics,client-id="{clientid}"
      #kafka.consumer:type=consumer-metrics,client-id="{clientid}"
      - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.*)><>(.+-total|.+-avg|.+-bytes|.+-count|.+-ratio|.+-age|.+-flight|.+-threads|.+-connectors|.+-tasks|.+-ago)
        name: kafka_$2_$4
        labels:
          clientId: "$3"
        help: "Kafka $1 JMX metric type $2"
        type: GAUGE

      #kafka.connect:type=connector-task-metrics,connector="{connector}",task="{task}<> status"
      - pattern: 'kafka.connect<type=connector-task-metrics, connector=(.+), task=(.+)><>status: ([a-z-]+)'
        name: kafka_connect_connector_status
        value: 1
        labels:
          connector: "$1"
          task: "$2"
          status: "$3"
        help: "Kafka Connect JMX Connector status"
        type: GAUGE

      #kafka.connect:type=task-error-metrics,connector="{connector}",task="{task}"
      #kafka.connect:type=source-task-metrics,connector="{connector}",task="{task}"
      #kafka.connect:type=sink-task-metrics,connector="{connector}",task="{task}"
      #kafka.connect:type=connector-task-metrics,connector="{connector}",task="{task}"
      - pattern: kafka.connect<type=(.+)-metrics, connector=(.+), task=(.+)><>(.+-total|.+-count|.+-ms|.+-ratio|.+-avg|.+-failures|.+-requests|.+-timestamp|.+-logged|.+-errors|.+-retries|.+-skipped)
        name: kafka_connect_$1_$4
        labels:
          connector: "$2"
          task: "$3"
        help: "Kafka Connect JMX metric type $1"
        type: GAUGE

      #kafka.connect:type=connector-metrics,connector="{connector}"
      #kafka.connect:type=connect-worker-metrics,connector="{connector}"
      - pattern: kafka.connect<type=connect-worker-metrics, connector=(.+)><>([a-z-]+)
        name: kafka_connect_worker_$2
        labels:
          connector: "$1"
        help: "Kafka Connect JMX metric $1"
        type: GAUGE

      #kafka.connect:type=connect-worker-metrics
      - pattern: kafka.connect<type=connect-worker-metrics><>([a-z-]+)
        name: kafka_connect_worker_$1
        help: "Kafka Connect JMX metric worker"
        type: GAUGE

      #kafka.connect:type=connect-worker-rebalance-metrics
      - pattern: kafka.connect<type=connect-worker-rebalance-metrics><>([a-z-]+)
        name: kafka_connect_worker_rebalance_$1
        help: "Kafka Connect JMX metric rebalance information"
        type: GAUGE

[scholzj]

You need to revuild your Connect image with your connectors with the new base image corresponding to the new Strimzi version. That should fix this issue.

[lkoniecz]

@scholzj
thanks, that solved the issue
I kind of missed the solution proposed #5512