kafkaExporter container cannot start anymore in strimzi 0.26

[thibthibus]

Hi

The kafkaExporter container cannot start anymore with the latest Strimzi release (that relies v1.4.2 of https://github.com/danielqsj/kafka_exporter project)

Error is: kafka_exporter: error: strconv.ParseFloat: parsing "info": invalid syntax, try --help

The cause is this change:

strimzi-kafka-operator/docker-images/kafka/exporter-scripts/kafka_exporter_run.sh

Line 24 in 111d4ba

loglevel="--verbosity=${KAFKA_EXPORTER_LOGGING}"

I'm not sure if verbosity parameter really replaced the log.level parameter of kafka_exporter.
I think it's more an issue in the documentation of kafka_exporter project as log.level disappeared from README.md hence seeming to have been replaced by the new verbosity parameter.
I think that verbosity is more a new parameter that expects anyway an integer value.
log.level seams still to exist.

sh-4.4$ ./kafka_exporter --help
usage: kafka_exporter [<flags>]

Flags:
  -h, --help                     Show context-sensitive help (also try --help-long and --help-man).
      --web.listen-address=":9308"  
                                 Address to listen on for web interface and telemetry.
      --web.telemetry-path="/metrics"  
                                 Path under which to expose metrics.
      --topic.filter=".*"        Regex that determines which topics to collect.
      --group.filter=".*"        Regex that determines which consumer groups to collect.
      --log.enable-sarama        Turn on Sarama logging.
      --kafka.server=kafka:9092 ...  
                                 Address (host:port) of Kafka server.
      --sasl.enabled             Connect using SASL/PLAIN.
      --sasl.handshake           Only set this to false if using a non-Kafka SASL proxy.
      --sasl.username=""         SASL user name.
      --sasl.password=""         SASL user password.
      --sasl.mechanism=""        The SASL SCRAM SHA algorithm sha256 or sha512 or gssapi as mechanism
      --sasl.service-name=""     Service name when using kerberos Auth
      --sasl.kerberos-config-path=""  
                                 Kerberos config path
      --sasl.realm=""            Kerberos realm
      --sasl.kerberos-auth-type=""  
                                 Kerberos auth type. Either 'keytabAuth' or 'userAuth'
      --sasl.keytab-path=""      Kerberos keytab file path
      --tls.enabled              Connect to Kafka using TLS.
      --tls.server-name=""       Used to verify the hostname on the returned certificates unless tls.insecure-skip-tls-verify is given. The kafka server's name should be given.
      --tls.ca-file=""           The optional certificate authority file for Kafka TLS client authentication.
      --tls.cert-file=""         The optional certificate file for Kafka client authentication.
      --tls.key-file=""          The optional key file for Kafka client authentication.
      --server.tls.enabled       Enable TLS for web server.
      --server.tls.mutual-auth-enabled  
                                 Enable TLS client mutual authentication.
      --server.tls.ca-file=""    The certificate authority file for the web server.
      --server.tls.cert-file=""  The certificate file for the web server.
      --server.tls.key-file=""   The key file for the web server.
      --tls.insecure-skip-tls-verify  
                                 If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure.
      --kafka.version="2.0.0"    Kafka broker version
      --use.consumelag.zookeeper  
                                 if you need to use a group from zookeeper
      --zookeeper.server=localhost:2181 ...  
                                 Address (hosts) of zookeeper server.
      --kafka.labels=""          Kafka cluster name
      --refresh.metadata="30s"   Metadata refresh interval
      --offset.show-all          Whether show the offset/lag for all consumer group, otherwise, only show connected consumer groups
      --concurrent.enable        If true, all scrapes will trigger kafka operations otherwise, they will share results. WARN: This should be disabled on large clusters
      --topic.workers=100        Number of topic workers
      --verbosity=0              Verbosity log level
      --log.level=info           Only log messages with the given severity or above. One of: [debug, info, warn, error]
      --log.format=logfmt        Output format of log messages. One of: [logfmt, json]
      --version                  Show application version.

[sknot-rh]

The Strimzi operator does the mapping between literal and integer values: https://github.com/strimzi/strimzi-kafka-operator/blob/main/cluster-operator/src/main/java/io/strimzi/operator/cluster/model/KafkaExporter.java#L249
Older version of Kafka Exporter seems to ignore the log.level option (not found here https://github.com/danielqsj/kafka_exporter/blob/v1.3.1/kafka_exporter.go#L509)
Can you please share the logs of CO and the env vars of the KafkaExporter pod?

[thibthibus]

Hi @sknot-rh,

Thanks for your quick reply. Ok understood, so that explains everything then:

1. That log.level had no impact before
2. Why I'm having this error: We are running RH AMQ Streams 1.8 that is using Strimzi 0.24. Since I had issue on KafkaExporter I tried to use the latest image of kafkaExporter on quay.io to test. But I didn't upgrade strimzi-operator (and won't anyway as I have to wait for a newest release of AMQ Streams)

If I set logging variable to 3 in Kafka CR, the container is now starting.

I guess you can close this issue then.

[scholzj]

@thibthibus You cannot mix the containers between Strimzi versions. There might be many things which will not work. You should always use all containers from the same Strimzi version.