Kafka-exporter being halted everyday and not sending logs, we needed to restart kafka-exporter pod to start sending logs

[kvenu1458]

Hi Team,

We have strimzi kafka up and running without any issue, but kafka-exporter is beling halted with below error and stoping sending logs, if we restart kafka-exporter pod it's again sending logs, but we can't restart it daily, we need fix for this, please help us here

exec java -Dquarkus.http.host=0.0.0.0 -javaagent:/opt/agent-bond/agent-bond.jar=jmx_exporter{{9779:/opt/agent-bond/jmx_exporter_config.yml}} -XX:+ExitOnOutOfMemoryError -cp . -jar /deployments/quarkus-run.jar
Jun 27, 2022 11:16:59 AM org.jboss.threads.Version
INFO: JBoss Threads version 3.2.0.Final

---

--/ __ / / / / _ | / _ / /// / / / __/
-/ // / // / __ |/ , / ,< / // /\
--___// |//|//||_//
2022-06-27 11:17:00,119 INFO [io.str.kaf.ser.ver.VertxConfiguration] (main) Adding custom metrics paths '/metrics', scheduled collect 'true' ...
2022-06-27 11:17:00,147 INFO [io.str.kaf.ser.uti.KafkaUtil] (pool-7-thread-1) Replaced property 'ssl.keystore.key' with file's content ...
2022-06-27 11:17:00,205 INFO [io.str.kaf.ser.uti.KafkaUtil] (pool-7-thread-1) Replaced property 'ssl.keystore.certificate.chain' with file's content ...
2022-06-27 11:17:00,205 INFO [io.str.kaf.ser.uti.KafkaUtil] (pool-7-thread-1) Replaced property 'ssl.truststore.certificates' with file's content ...
2022-06-27 11:17:00,235 INFO [org.apa.kaf.cli.adm.AdminClientConfig] (pool-7-thread-1) AdminClientConfig values:
bootstrap.servers = [dev-demosight-kafka-bootstrap:9091]
client.dns.lookup = use_all_dns_ips
client.id =
connections.max.idle.ms = 300000
default.api.timeout.ms = 60000
metadata.max.age.ms = 60000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = SSL
security.providers = null
send.buffer.bytes = 131072
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = [hidden]
ssl.keystore.key = [hidden]
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = PEM
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = [hidden]
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = PEM

Jun 27, 2022 11:17:00 AM io.quarkus.bootstrap.runner.Timing printStartupTime
INFO: kafka-exporter-server 1.0.0-SNAPSHOT on JVM (powered by Quarkus 2.0.0.Alpha2) started in 2.483s. Listening on: http://0.0.0.0:9404
Jun 27, 2022 11:17:00 AM io.quarkus.bootstrap.runner.Timing printStartupTime
INFO: Profile strimzi activated.
Jun 27, 2022 11:17:00 AM io.quarkus.bootstrap.runner.Timing printStartupTime
INFO: Installed features: [cdi, micrometer, smallrye-context-propagation, vertx]
2022-06-27 11:17:01,631 INFO [org.apa.kaf.com.uti.AppInfoParser] (pool-7-thread-1) Kafka version: 2.8.0
2022-06-27 11:17:01,632 INFO [org.apa.kaf.com.uti.AppInfoParser] (pool-7-thread-1) Kafka commitId: ebb1d6e21cc92130
2022-06-27 11:17:01,632 INFO [org.apa.kaf.com.uti.AppInfoParser] (pool-7-thread-1) Kafka startTimeMs: 1656328621629
2022-06-27 23:22:34,793 INFO [org.apa.kaf.cli.adm.KafkaAdminClient] (kafka-admin-client-thread | adminclient-1) [AdminClient clientId=adminclient-1] Retrying to fetch metadata.
2022-06-27 23:22:34,800 INFO [org.apa.kaf.cli.adm.KafkaAdminClient] (kafka-admin-client-thread | adminclient-1) [AdminClient clientId=adminclient-1] Retrying to fetch metadata.
2022-06-27 23:22:34,887 INFO [org.apa.kaf.cli.adm.KafkaAdminClient] (kafka-admin-client-thread | adminclient-1) [AdminClient clientId=adminclient-1] Retrying to fetch metadata.
2022-06-27 23:22:34,892 INFO [org.apa.kaf.cli.adm.KafkaAdminClient] (kafka-admin-client-thread | adminclient-1) [AdminClient clientId=adminclient-1] Retrying to fetch metadata.
2022-06-27 23:22:35,203 INFO [org.apa.kaf.cli.adm.KafkaAdminClient] (kafka-admin-client-thread | adminclient-1) [AdminClient clientId=adminclient-1] Retrying to fetch metadata.
2022-06-27 23:22:35,378 INFO [org.apa.kaf.cli.adm.KafkaAdminClient] (kafka-admin-client-thread | adminclient-1) [AdminClient clientId=adminclient-1] Retrying to fetch metadata.
2022-06-27 23:22:35,575 INFO [org.apa.kaf.cli.adm.KafkaAdminClient] (kafka-admin-client-thread | adminclient-1) [AdminClient clientId=adminclient-1] Retrying to fetch metadata.
2022-06-27 23:22:35,816 WARN [io.str.kaf.ser.KafkaAsyncCollector] (pool-8-thread-2) Error while collecting metrics, not all could (probably) be collected.: java.util.concurrent.CompletionException: org.apache.kafka.common.errors.OffsetNotAvailableException: The leader high watermark has not caught up from a recent leader election so the offsets cannot be guaranteed to be monotonically increasing.
at java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367)
at java.base/java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:376)
at java.base/java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:1019)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
at io.strimzi.kafkaexporter.server.utils.KafkaUtil.lambda$toCF$0(KafkaUtil.java:27)
at org.apache.kafka.common.internals.KafkaFutureImpl$WhenCompleteBiConsumer.accept(KafkaFutureImpl.java:175)
at org.apache.kafka.common.internals.KafkaFutureImpl$WhenCompleteBiConsumer.accept(KafkaFutureImpl.java:162)
at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:238)
at org.apache.kafka.common.internals.KafkaFutureImpl$Applicant.accept(KafkaFutureImpl.java:61)
at org.apache.kafka.common.internals.KafkaFutureImpl$Applicant.accept(KafkaFutureImpl.java:49)
at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:238)
at org.apache.kafka.common.KafkaFuture$AllOfAdapter.accept(KafkaFuture.java:73)
at org.apache.kafka.common.KafkaFuture$AllOfAdapter.accept(KafkaFuture.java:57)
at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:238)
at org.apache.kafka.clients.admin.KafkaAdminClient$33.handleResponse(KafkaAdminClient.java:4083)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.handleResponses(KafkaAdminClient.java:1189)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1341)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.kafka.common.errors.OffsetNotAvailableException: The leader high watermark has not caught up from a recent leader election so the offsets cannot be guaranteed to be monotonically increasing.

[scholzj]

Kafka Exporter is tool which exposes some additional metrics such as consumer lag as a Prometheus metric. It has nothing to do with your Quarkus application - it has its own pod deployed by Strimzi.

[kvenu1458]

it's stopping collecting metrics, when below log appears inkafka- exporter pod log

2022-06-27 23:22:35,816 WARN [io.str.kaf.ser.KafkaAsyncCollector] (pool-8-thread-2) Error while collecting metrics, not all could (probably) be collected.: java.util.concurrent.CompletionException: org.apache.kafka.common.errors.OffsetNotAvailableException: The leader high watermark has not caught up from a recent leader election so the offsets cannot be guaranteed to be monotonically increasing.
at java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367)

[scholzj]

I'm sorry, but the Kafka Exporter Strimzi uses is not any Java based application. It is written in Golang: https://github.com/danielqsj/kafka_exporter ... so I really don't understand what are you talking about.