ERROR Shutdown broker because all log dirs in /var/lib/kafka/data-0/kafka-log0 have failed (kafka.log.LogManager) [LogDirFailureHandler] #4832

rjcastillejo-chile · 2021-04-27T22:59:44Z

rjcastillejo-chile
Apr 27, 2021

After a week of work this error always appears.
I have in yaml topic retention.bytes = 54857600 set and cluster log.retention.ms: -1

scholzj · 2021-04-27T23:01:58Z

scholzj
Apr 27, 2021
Maintainer

Can you share the full log? From one line in the title it is quite hard to tell. You should also make sure you use block storage (and not something like NFS) and have enough free disk space.

5 replies

rjcastillejo-chile Apr 28, 2021
Author

log.retention.check.interval.ms = 300000
log.retention.hours = 168
log.retention.minutes = null
log.retention.ms = null
log.roll.hours = 168
log.roll.jitter.hours = 0
log.roll.jitter.ms = null
log.roll.ms = null
log.segment.bytes = 1073741824
log.segment.delete.delay.ms = 60000
max.connections = 2147483647
max.connections.per.ip = 2147483647
max.connections.per.ip.overrides = 
max.incremental.fetch.session.cache.slots = 1000
message.max.bytes = 1048588
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
min.insync.replicas = 1
num.io.threads = 8
num.network.threads = 3
num.partitions = 1
num.recovery.threads.per.data.dir = 1
num.replica.alter.log.dirs.threads = null
num.replica.fetchers = 1
offset.metadata.max.bytes = 4096
offsets.commit.required.acks = -1
offsets.commit.timeout.ms = 5000
offsets.load.buffer.size = 5242880
offsets.retention.check.interval.ms = 600000
offsets.retention.minutes = 10080
offsets.topic.compression.codec = 0
offsets.topic.num.partitions = 50
offsets.topic.replication.factor = 1
offsets.topic.segment.bytes = 104857600
password.encoder.cipher.algorithm = AES/CBC/PKCS5Padding
password.encoder.iterations = 4096
password.encoder.key.length = 128
password.encoder.keyfactory.algorithm = null
password.encoder.old.secret = null
password.encoder.secret = null
port = 9092
principal.builder.class = null
producer.purgatory.purge.interval.requests = 1000
queued.max.request.bytes = -1
queued.max.requests = 500
quota.consumer.default = 9223372036854775807
quota.producer.default = 9223372036854775807
quota.window.num = 11
quota.window.size.seconds = 1
replica.fetch.backoff.ms = 1000
replica.fetch.max.bytes = 1048576
replica.fetch.min.bytes = 1
replica.fetch.response.max.bytes = 10485760
replica.fetch.wait.max.ms = 500
replica.high.watermark.checkpoint.interval.ms = 5000
replica.lag.time.max.ms = 30000
replica.selector.class = null
replica.socket.receive.buffer.bytes = 65536
replica.socket.timeout.ms = 30000
replication.quota.window.num = 11
replication.quota.window.size.seconds = 1
request.timeout.ms = 30000
reserved.broker.max.id = 1000
sasl.client.callback.handler.class = null
sasl.enabled.mechanisms = []
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.principal.to.local.rules = [DEFAULT]
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism.inter.broker.protocol = GSSAPI
sasl.server.callback.handler.class = null
security.inter.broker.protocol = PLAINTEXT
security.providers = null
socket.receive.buffer.bytes = 102400
socket.request.max.bytes = 104857600
socket.send.buffer.bytes = 102400
ssl.cipher.suites = []
ssl.client.auth = none
ssl.enabled.protocols = [TLSv1.2]
ssl.endpoint.identification.algorithm = HTTPS
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.principal.mapping.rules = DEFAULT
ssl.protocol = TLSv1.2
ssl.provider = null
ssl.secure.random.implementation = SHA1PRNG
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.abort.timed.out.transaction.cleanup.interval.ms = 10000
transaction.max.timeout.ms = 900000
transaction.remove.expired.transaction.cleanup.interval.ms = 3600000
transaction.state.log.load.buffer.size = 5242880
transaction.state.log.min.isr = 1
transaction.state.log.num.partitions = 50
transaction.state.log.replication.factor = 1
transaction.state.log.segment.bytes = 104857600
transactional.id.expiration.ms = 604800000
unclean.leader.election.enable = false
zookeeper.clientCnxnSocket = org.apache.zookeeper.ClientCnxnSocketNetty
zookeeper.connect = kafka-cluster-zookeeper-client:2181
zookeeper.connection.timeout.ms = null
zookeeper.max.in.flight.requests = 10
zookeeper.session.timeout.ms = 18000
zookeeper.set.acl = false
zookeeper.ssl.cipher.suites = null
zookeeper.ssl.client.enable = true
zookeeper.ssl.crl.enable = false
zookeeper.ssl.enabled.protocols = null
zookeeper.ssl.endpoint.identification.algorithm = HTTPS
zookeeper.ssl.keystore.location = /tmp/kafka/cluster.keystore.p12
zookeeper.ssl.keystore.password = [hidden]
zookeeper.ssl.keystore.type = PKCS12
zookeeper.ssl.ocsp.enable = false
zookeeper.ssl.protocol = TLSv1.2
zookeeper.ssl.truststore.location = /tmp/kafka/cluster.truststore.p12
zookeeper.ssl.truststore.password = [hidden]
zookeeper.ssl.truststore.type = PKCS12
zookeeper.sync.time.ms = 2000

(kafka.server.KafkaConfig) [main]
2021-04-28 16:32:59,416 INFO [ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [ThrottledChannelReaper-Produce]
2021-04-28 16:32:59,416 INFO [ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [ThrottledChannelReaper-Fetch]
2021-04-28 16:32:59,429 INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [ThrottledChannelReaper-Request]
2021-04-28 16:32:59,528 INFO Loading logs. (kafka.log.LogManager) [main]
2021-04-28 16:32:59,739 INFO [Log partition=interfaz-0, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:32:59,744 INFO [Log partition=interfaz-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:00,371 INFO [Log partition=interfaz-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:00,428 INFO [Log partition=interfaz-0, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 785 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:00,509 INFO [Log partition=ncaofa-0, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:00,509 INFO [Log partition=ncaofa-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:00,959 INFO [Log partition=ncaofa-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:01,022 INFO [Log partition=ncaofa-0, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 575 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:01,094 INFO [Log partition=ncastock-0, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:01,095 INFO [Log partition=ncastock-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:01,399 INFO [Log partition=ncastock-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:01,461 INFO [Log partition=ncastock-0, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 433 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:01,531 INFO [Log partition=null-0, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:01,531 INFO [Log partition=null-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:01,972 INFO [Log partition=null-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:02,028 INFO [Log partition=null-0, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 560 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:02,095 INFO [Log partition=test-0, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:02,096 INFO [Log partition=test-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:02,674 INFO [Log partition=test-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:02,733 INFO [Log partition=test-0, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 701 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:02,799 INFO [Log partition=tpc-actions-0, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 3377 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:02,799 INFO [Log partition=tpc-actions-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 3377 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:02,844 INFO [ProducerStateManager partition=tpc-actions-0] Writing producer snapshot at offset 3377 (kafka.log.ProducerStateManager) [pool-4-thread-1]
2021-04-28 16:33:03,293 INFO [Log partition=tpc-actions-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 3377 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:03,357 INFO [Log partition=tpc-actions-0, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 3377 and log end offset 3377 in 620 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:03,423 INFO [Log partition=tpc-events-0, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 12830 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:03,424 INFO [Log partition=tpc-events-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 12830 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:03,476 INFO [ProducerStateManager partition=tpc-events-0] Writing producer snapshot at offset 12830 (kafka.log.ProducerStateManager) [pool-4-thread-1]
2021-04-28 16:33:03,916 INFO [Log partition=tpc-events-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 12830 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:03,990 INFO [Log partition=tpc-events-0, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 12830 and log end offset 12830 in 628 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:04,058 INFO [Log partition=tpc-interfaz-0, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:04,058 INFO [Log partition=tpc-interfaz-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:04,519 INFO [Log partition=tpc-interfaz-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:04,575 INFO [Log partition=tpc-interfaz-0, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 578 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:04,648 INFO [Log partition=tpc-nca-ofa-0, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:04,649 INFO [Log partition=tpc-nca-ofa-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:05,022 INFO [Log partition=tpc-nca-ofa-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:05,079 INFO [Log partition=tpc-nca-ofa-0, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 500 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:05,143 INFO [Log partition=tpc-nca-pru-0, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:05,143 INFO [Log partition=tpc-nca-pru-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:05,749 INFO [Log partition=tpc-nca-pru-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:05,813 INFO [Log partition=tpc-nca-pru-0, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 730 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:05,880 INFO [Log partition=tpc-nca-stock-0, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:05,880 INFO [Log partition=tpc-nca-stock-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:06,350 INFO [Log partition=tpc-nca-stock-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:06,407 INFO [Log partition=tpc-nca-stock-0, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 590 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:06,472 INFO [Log partition=tpc-test-0, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:06,473 INFO [Log partition=tpc-test-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:06,883 INFO [Log partition=tpc-test-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:06,937 INFO [Log partition=tpc-test-0, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 527 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:07,001 INFO [Log partition=__consumer_offsets-0, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:07,002 INFO [Log partition=__consumer_offsets-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:07,494 INFO [Log partition=__consumer_offsets-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:07,549 INFO [Log partition=__consumer_offsets-0, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 609 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:07,615 INFO [Log partition=__consumer_offsets-1, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:07,616 INFO [Log partition=__consumer_offsets-1, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:08,150 INFO [Log partition=__consumer_offsets-1, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:08,212 INFO [Log partition=__consumer_offsets-1, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 660 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:08,277 INFO [Log partition=__consumer_offsets-10, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:08,278 INFO [Log partition=__consumer_offsets-10, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:08,816 INFO [Log partition=__consumer_offsets-10, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:08,872 INFO [Log partition=__consumer_offsets-10, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 657 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:08,940 INFO [Log partition=__consumer_offsets-11, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:08,940 INFO [Log partition=__consumer_offsets-11, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:09,223 INFO [Log partition=__consumer_offsets-11, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:09,286 INFO [Log partition=__consumer_offsets-11, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 411 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:09,356 INFO [Log partition=__consumer_offsets-12, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:09,357 INFO [Log partition=__consumer_offsets-12, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:09,609 INFO [Log partition=__consumer_offsets-12, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:09,672 INFO [Log partition=__consumer_offsets-12, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 383 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:09,744 INFO [Log partition=__consumer_offsets-13, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:09,744 INFO [Log partition=__consumer_offsets-13, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:09,997 INFO [Log partition=__consumer_offsets-13, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:10,067 INFO [Log partition=__consumer_offsets-13, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 393 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:10,137 INFO [Log partition=__consumer_offsets-14, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:10,137 INFO [Log partition=__consumer_offsets-14, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:10,452 INFO [Log partition=__consumer_offsets-14, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:10,516 INFO [Log partition=__consumer_offsets-14, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 446 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:10,587 INFO [Log partition=__consumer_offsets-15, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:10,588 INFO [Log partition=__consumer_offsets-15, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:10,910 INFO [Log partition=__consumer_offsets-15, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:10,978 INFO [Log partition=__consumer_offsets-15, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 455 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:11,043 INFO [Log partition=__consumer_offsets-16, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:11,043 INFO [Log partition=__consumer_offsets-16, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:11,371 INFO [Log partition=__consumer_offsets-16, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:11,452 INFO [Log partition=__consumer_offsets-16, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 472 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:11,807 INFO [Log partition=__consumer_offsets-17, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:11,807 INFO [Log partition=__consumer_offsets-17, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:12,109 INFO [Log partition=__consumer_offsets-17, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:12,172 INFO [Log partition=__consumer_offsets-17, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 716 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:12,237 INFO [Log partition=__consumer_offsets-18, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:12,237 INFO [Log partition=__consumer_offsets-18, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:12,559 INFO [Log partition=__consumer_offsets-18, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:12,616 INFO [Log partition=__consumer_offsets-18, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 442 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:12,687 INFO [Log partition=__consumer_offsets-19, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:12,687 INFO [Log partition=__consumer_offsets-19, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:13,341 INFO [Log partition=__consumer_offsets-19, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:13,399 INFO [Log partition=__consumer_offsets-19, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 780 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:13,473 INFO [Log partition=__consumer_offsets-2, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:13,474 INFO [Log partition=__consumer_offsets-2, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:13,912 INFO [Log partition=__consumer_offsets-2, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:13,974 INFO [Log partition=__consumer_offsets-2, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 572 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:14,041 INFO [Log partition=__consumer_offsets-20, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:14,042 INFO [Log partition=__consumer_offsets-20, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:14,453 INFO [Log partition=__consumer_offsets-20, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:14,513 INFO [Log partition=__consumer_offsets-20, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 537 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:14,577 INFO [Log partition=__consumer_offsets-21, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:14,578 INFO [Log partition=__consumer_offsets-21, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:15,037 INFO [Log partition=__consumer_offsets-21, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:15,099 INFO [Log partition=__consumer_offsets-21, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 584 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:15,175 INFO [Log partition=__consumer_offsets-22, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:15,176 INFO [Log partition=__consumer_offsets-22, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:15,539 INFO [Log partition=__consumer_offsets-22, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:15,599 INFO [Log partition=__consumer_offsets-22, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 498 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:15,669 INFO [Log partition=__consumer_offsets-23, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:15,669 INFO [Log partition=__consumer_offsets-23, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:16,092 INFO [Log partition=__consumer_offsets-23, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:16,155 INFO [Log partition=__consumer_offsets-23, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 554 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:16,223 INFO [Log partition=__consumer_offsets-24, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:16,223 INFO [Log partition=__consumer_offsets-24, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:16,528 INFO [Log partition=__consumer_offsets-24, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:16,795 INFO [Log partition=__consumer_offsets-24, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 636 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:16,859 INFO [Log partition=__consumer_offsets-25, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:16,860 INFO [Log partition=__consumer_offsets-25, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:18,034 INFO [Log partition=__consumer_offsets-25, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:18,091 INFO [Log partition=__consumer_offsets-25, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 1294 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:18,157 INFO [Log partition=__consumer_offsets-26, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:18,157 INFO [Log partition=__consumer_offsets-26, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:18,432 INFO [Log partition=__consumer_offsets-26, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:18,493 INFO [Log partition=__consumer_offsets-26, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 401 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:18,557 INFO [Log partition=__consumer_offsets-27, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:18,557 INFO [Log partition=__consumer_offsets-27, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:19,101 INFO [Log partition=__consumer_offsets-27, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:19,160 INFO [Log partition=__consumer_offsets-27, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 664 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:19,224 INFO [Log partition=__consumer_offsets-28, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:19,225 INFO [Log partition=__consumer_offsets-28, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:19,532 INFO [Log partition=__consumer_offsets-28, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:19,589 INFO [Log partition=__consumer_offsets-28, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 427 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:19,659 INFO [Log partition=__consumer_offsets-29, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:19,659 INFO [Log partition=__consumer_offsets-29, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:19,966 INFO [Log partition=__consumer_offsets-29, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:20,023 INFO [Log partition=__consumer_offsets-29, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 432 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:20,086 INFO [Log partition=__consumer_offsets-3, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:20,086 INFO [Log partition=__consumer_offsets-3, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:20,376 INFO [Log partition=__consumer_offsets-3, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:20,435 INFO [Log partition=__consumer_offsets-3, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 410 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:20,497 INFO [Log partition=__consumer_offsets-30, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:20,497 INFO [Log partition=__consumer_offsets-30, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:20,842 INFO [Log partition=__consumer_offsets-30, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:21,017 INFO [Log partition=__consumer_offsets-30, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 580 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:21,084 INFO [Log partition=__consumer_offsets-31, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:21,085 INFO [Log partition=__consumer_offsets-31, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:21,396 INFO [Log partition=__consumer_offsets-31, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:21,460 INFO [Log partition=__consumer_offsets-31, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 441 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:21,528 INFO [Log partition=__consumer_offsets-32, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:21,529 INFO [Log partition=__consumer_offsets-32, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:21,860 INFO [Log partition=__consumer_offsets-32, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:21,925 INFO [Log partition=__consumer_offsets-32, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 463 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:21,991 INFO [Log partition=__consumer_offsets-33, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:21,991 INFO [Log partition=__consumer_offsets-33, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:22,335 INFO [Log partition=__consumer_offsets-33, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:22,429 INFO [Log partition=__consumer_offsets-33, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 503 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:22,495 INFO [Log partition=__consumer_offsets-34, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:22,495 INFO [Log partition=__consumer_offsets-34, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:22,834 INFO [Log partition=__consumer_offsets-34, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:22,889 INFO [Log partition=__consumer_offsets-34, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 459 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:23,164 INFO [Log partition=__consumer_offsets-35, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:23,164 INFO [Log partition=__consumer_offsets-35, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:23,486 INFO [Log partition=__consumer_offsets-35, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:23,559 INFO [Log partition=__consumer_offsets-35, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 668 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:23,668 INFO [Log partition=__consumer_offsets-36, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:23,668 INFO [Log partition=__consumer_offsets-36, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:24,076 INFO [ProducerStateManager partition=__consumer_offsets-36] Writing producer snapshot at offset 2183 (kafka.log.ProducerStateManager) [pool-4-thread-1]
2021-04-28 16:33:24,097 INFO [Log partition=__consumer_offsets-36, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 2183 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:24,097 INFO [Log partition=__consumer_offsets-36, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 2183 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:24,149 INFO [ProducerStateManager partition=__consumer_offsets-36] Loading producer state from snapshot file '/var/lib/kafka/data-0/kafka-log0/__consumer_offsets-36/00000000000000002183.snapshot' (kafka.log.ProducerStateManager) [pool-4-thread-1]
2021-04-28 16:33:24,400 INFO [ProducerStateManager partition=__consumer_offsets-36] Writing producer snapshot at offset 2185 (kafka.log.ProducerStateManager) [pool-4-thread-1]
2021-04-28 16:33:24,718 INFO [Log partition=__consumer_offsets-36, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 2185 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:24,771 INFO [ProducerStateManager partition=__consumer_offsets-36] Loading producer state from snapshot file '/var/lib/kafka/data-0/kafka-log0/__consumer_offsets-36/00000000000000002185.snapshot' (kafka.log.ProducerStateManager) [pool-4-thread-1]
2021-04-28 16:33:24,787 INFO [Log partition=__consumer_offsets-36, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 2 segments, log start offset 0 and log end offset 2185 in 1226 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:24,850 INFO [Log partition=__consumer_offsets-37, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:24,850 INFO [Log partition=__consumer_offsets-37, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:25,318 INFO [ProducerStateManager partition=__consumer_offsets-37] Writing producer snapshot at offset 12855 (kafka.log.ProducerStateManager) [pool-4-thread-1]
2021-04-28 16:33:25,566 INFO [Log partition=__consumer_offsets-37, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 12855 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:25,623 INFO [ProducerStateManager partition=__consumer_offsets-37] Loading producer state from snapshot file '/var/lib/kafka/data-0/kafka-log0/__consumer_offsets-37/00000000000000012855.snapshot' (kafka.log.ProducerStateManager) [pool-4-thread-1]
2021-04-28 16:33:25,637 INFO [Log partition=__consumer_offsets-37, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 12855 in 849 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:25,701 INFO [Log partition=__consumer_offsets-38, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:25,701 INFO [Log partition=__consumer_offsets-38, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:26,009 INFO [Log partition=__consumer_offsets-38, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:26,065 INFO [Log partition=__consumer_offsets-38, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 426 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:26,142 INFO [Log partition=__consumer_offsets-39, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:26,142 INFO [Log partition=__consumer_offsets-39, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:26,494 INFO [Log partition=__consumer_offsets-39, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:26,551 INFO [Log partition=__consumer_offsets-39, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 482 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:26,619 INFO [Log partition=__consumer_offsets-4, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:26,619 INFO [Log partition=__consumer_offsets-4, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:26,919 INFO [Log partition=__consumer_offsets-4, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:26,976 INFO [Log partition=__consumer_offsets-4, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 422 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:27,049 INFO [Log partition=__consumer_offsets-40, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:27,051 INFO [Log partition=__consumer_offsets-40, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:27,443 INFO [Log partition=__consumer_offsets-40, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:27,507 INFO [Log partition=__consumer_offsets-40, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 530 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:27,573 INFO [Log partition=__consumer_offsets-41, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:27,573 INFO [Log partition=__consumer_offsets-41, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:27,866 INFO [Log partition=__consumer_offsets-41, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:27,931 INFO [Log partition=__consumer_offsets-41, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 421 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:27,997 INFO [Log partition=__consumer_offsets-42, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:27,997 INFO [Log partition=__consumer_offsets-42, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:28,335 INFO [Log partition=__consumer_offsets-42, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:28,391 INFO [Log partition=__consumer_offsets-42, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 458 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:28,454 INFO [Log partition=__consumer_offsets-43, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:28,454 INFO [Log partition=__consumer_offsets-43, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:28,734 INFO [Log partition=__consumer_offsets-43, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:28,793 INFO [Log partition=__consumer_offsets-43, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 400 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:28,856 INFO [Log partition=__consumer_offsets-44, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:28,856 INFO [Log partition=__consumer_offsets-44, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:29,201 INFO [Log partition=__consumer_offsets-44, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:29,267 INFO [Log partition=__consumer_offsets-44, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 472 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:29,333 INFO [Log partition=__consumer_offsets-45, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:29,333 INFO [Log partition=__consumer_offsets-45, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:29,603 INFO [Log partition=__consumer_offsets-45, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:29,668 INFO [Log partition=__consumer_offsets-45, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 400 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:29,955 INFO [Log partition=__consumer_offsets-46, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:29,955 INFO [Log partition=__consumer_offsets-46, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:30,320 INFO [Log partition=__consumer_offsets-46, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:30,379 INFO [Log partition=__consumer_offsets-46, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 709 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:30,461 INFO [Log partition=__consumer_offsets-47, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:30,461 INFO [Log partition=__consumer_offsets-47, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:30,806 INFO [Log partition=__consumer_offsets-47, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:30,864 INFO [Log partition=__consumer_offsets-47, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 484 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:30,937 INFO [Log partition=__consumer_offsets-48, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:30,937 INFO [Log partition=__consumer_offsets-48, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:31,224 INFO [Log partition=__consumer_offsets-48, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:31,285 INFO [Log partition=__consumer_offsets-48, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 420 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:31,359 INFO [Log partition=__consumer_offsets-49, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:31,359 INFO [Log partition=__consumer_offsets-49, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:31,679 INFO [Log partition=__consumer_offsets-49, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:31,738 INFO [Log partition=__consumer_offsets-49, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 451 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:31,816 INFO [Log partition=__consumer_offsets-5, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:31,816 INFO [Log partition=__consumer_offsets-5, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:32,238 INFO [Log partition=__consumer_offsets-5, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:32,303 INFO [Log partition=__consumer_offsets-5, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 563 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:32,383 INFO [Log partition=__consumer_offsets-6, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:32,383 INFO [Log partition=__consumer_offsets-6, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:32,661 INFO [Log partition=__consumer_offsets-6, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:32,731 INFO [Log partition=__consumer_offsets-6, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 426 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:32,807 INFO [Log partition=__consumer_offsets-7, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:32,808 INFO [Log partition=__consumer_offsets-7, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:33,076 INFO [Log partition=__consumer_offsets-7, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:33,134 INFO [Log partition=__consumer_offsets-7, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 402 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:33,204 INFO [Log partition=__consumer_offsets-8, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:33,205 INFO [Log partition=__consumer_offsets-8, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:33,621 INFO [Log partition=__consumer_offsets-8, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:33,689 INFO [Log partition=__consumer_offsets-8, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 554 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:33,781 INFO [Log partition=__consumer_offsets-9, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 0 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:33,782 INFO [Log partition=__consumer_offsets-9, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:34,105 INFO [Log partition=__consumer_offsets-9, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:34,159 INFO [Log partition=__consumer_offsets-9, dir=/var/lib/kafka/data-0/kafka-log0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 468 ms (kafka.log.Log) [pool-4-thread-1]
2021-04-28 16:33:34,165 INFO Logs loading complete in 34637 ms. (kafka.log.LogManager) [main]
2021-04-28 16:33:34,192 INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager) [main]
2021-04-28 16:33:34,193 INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager) [main]
2021-04-28 16:33:34,196 INFO Starting the log cleaner (kafka.log.LogCleaner) [main]
2021-04-28 16:33:34,255 INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:34,317 INFO Cleaner 0: Beginning cleaning of log __consumer_offsets-36. (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:34,317 INFO Cleaner 0: Building offset map for __consumer_offsets-36... (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:34,363 INFO Cleaner 0: Building offset map for log __consumer_offsets-36 for 1 segments in offset range [0, 2183). (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:34,498 INFO Cleaner 0: Offset map for log __consumer_offsets-36 complete. (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:34,533 INFO Cleaner 0: Cleaning log __consumer_offsets-36 (cleaning prior to Tue Apr 20 11:41:57 UTC 2021, discarding tombstones prior to Thu Jan 01 00:00:00 UTC 1970)... (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:34,568 INFO Cleaner 0: Cleaning LogSegment(baseOffset=0, size=268511, lastModifiedTime=1618918917846, largestTime=1618918887159) in log __consumer_offsets-36 into 0 with deletion horizon 0, retaining deletes. (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:34,855 INFO Awaiting socket connections on 0.0.0.0:9091. (kafka.network.Acceptor) [main]
2021-04-28 16:33:35,136 ERROR Failed to clean up log for __consumer_offsets-36 in dir /var/lib/kafka/data-0/kafka-log0 due to IOException (kafka.server.LogDirFailureChannel) [kafka-log-cleaner-thread-0]
java.nio.file.FileSystemException: /var/lib/kafka/data-0/kafka-log0/_consumer_offsets-36/00000000000000000000.log.cleaned: Operation not permitted
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.setTimes(UnixFileAttributeViews.java:125)
at java.base/java.nio.file.Files.setLastModifiedTime(Files.java:2354)
at kafka.log.LogSegment.lastModified$eq(LogSegment.scala:644)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:590)
at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:519)
at kafka.log.Cleaner.doClean(LogCleaner.scala:518)
at kafka.log.Cleaner.clean(LogCleaner.scala:492)
at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:361)
at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:334)
at kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:314)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:303)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
2021-04-28 16:33:35,342 INFO Cleaner 0: Beginning cleaning of log __consumer_offsets-36. (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:35,342 INFO Cleaner 0: Building offset map for __consumer_offsets-36... (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:35,375 INFO Cleaner 0: Building offset map for log __consumer_offsets-36 for 1 segments in offset range [0, 2183). (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:35,396 INFO Cleaner 0: Offset map for log __consumer_offsets-36 complete. (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:35,397 INFO Cleaner 0: Cleaning log __consumer_offsets-36 (cleaning prior to Tue Apr 20 11:41:57 UTC 2021, discarding tombstones prior to Thu Jan 01 00:00:00 UTC 1970)... (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:35,564 ERROR Failed to clean up log for __consumer_offsets-36 in dir /var/lib/kafka/data-0/kafka-log0 due to IOException (kafka.server.LogDirFailureChannel) [kafka-log-cleaner-thread-0]
java.nio.file.NoSuchFileException: /var/lib/kafka/data-0/kafka-log0/__consumer_offsets-36/00000000000000000000.log.cleaned
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182)
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:292)
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:345)
at org.apache.kafka.common.record.FileRecords.openChannel(FileRecords.java:448)
at org.apache.kafka.common.record.FileRecords.open(FileRecords.java:411)
at org.apache.kafka.common.record.FileRecords.open(FileRecords.java:420)
at kafka.log.LogSegment$.open(LogSegment.scala:657)
at kafka.log.LogCleaner$.createNewCleanedSegment(LogCleaner.scala:437)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:547)
at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:519)
at kafka.log.Cleaner.doClean(LogCleaner.scala:518)
at kafka.log.Cleaner.clean(LogCleaner.scala:492)
at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:361)
at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:334)
at kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:314)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:303)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
2021-04-28 16:33:35,757 INFO Cleaner 0: Beginning cleaning of log __consumer_offsets-36. (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:35,757 INFO Cleaner 0: Building offset map for __consumer_offsets-36... (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:35,824 INFO Cleaner 0: Building offset map for log __consumer_offsets-36 for 1 segments in offset range [0, 2183). (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:35,840 INFO Cleaner 0: Offset map for log __consumer_offsets-36 complete. (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:35,841 INFO Cleaner 0: Cleaning log __consumer_offsets-36 (cleaning prior to Tue Apr 20 11:41:57 UTC 2021, discarding tombstones prior to Thu Jan 01 00:00:00 UTC 1970)... (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:35,910 INFO [SocketServer brokerId=0] Created data-plane acceptor and processors for endpoint : EndPoint(0.0.0.0,9091,ListenerName(REPLICATION-9091),SSL) (kafka.network.SocketServer) [main]
2021-04-28 16:33:35,910 INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor) [main]
2021-04-28 16:33:35,944 INFO [SocketServer brokerId=0] Created data-plane acceptor and processors for endpoint : EndPoint(0.0.0.0,9092,ListenerName(PLAIN-9092),PLAINTEXT) (kafka.network.SocketServer) [main]
2021-04-28 16:33:35,944 INFO Awaiting socket connections on 0.0.0.0:9094. (kafka.network.Acceptor) [main]
2021-04-28 16:33:35,946 ERROR Failed to clean up log for __consumer_offsets-36 in dir /var/lib/kafka/data-0/kafka-log0 due to IOException (kafka.server.LogDirFailureChannel) [kafka-log-cleaner-thread-0]
java.nio.file.NoSuchFileException: /var/lib/kafka/data-0/kafka-log0/__consumer_offsets-36/00000000000000000000.log.cleaned
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182)
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:292)
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:345)
at org.apache.kafka.common.record.FileRecords.openChannel(FileRecords.java:448)
at org.apache.kafka.common.record.FileRecords.open(FileRecords.java:411)
at org.apache.kafka.common.record.FileRecords.open(FileRecords.java:420)
at kafka.log.LogSegment$.open(LogSegment.scala:657)
at kafka.log.LogCleaner$.createNewCleanedSegment(LogCleaner.scala:437)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:547)
at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:519)
at kafka.log.Cleaner.doClean(LogCleaner.scala:518)
at kafka.log.Cleaner.clean(LogCleaner.scala:492)
at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:361)
at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:334)
at kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:314)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:303)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
2021-04-28 16:33:35,992 INFO [SocketServer brokerId=0] Created data-plane acceptor and processors for endpoint : EndPoint(0.0.0.0,9094,ListenerName(EXTERNAL-9094),PLAINTEXT) (kafka.network.SocketServer) [main]
2021-04-28 16:33:35,994 INFO [SocketServer brokerId=0] Started 3 acceptor threads for data-plane (kafka.network.SocketServer) [main]
2021-04-28 16:33:36,056 INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [ExpirationReaper-0-Produce]
2021-04-28 16:33:36,057 INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [ExpirationReaper-0-Fetch]
2021-04-28 16:33:36,065 INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [ExpirationReaper-0-DeleteRecords]
2021-04-28 16:33:36,086 INFO [ExpirationReaper-0-ElectLeader]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [ExpirationReaper-0-ElectLeader]
2021-04-28 16:33:36,159 INFO [LogDirFailureHandler]: Starting (kafka.server.ReplicaManager$LogDirFailureHandler) [LogDirFailureHandler]
2021-04-28 16:33:36,160 WARN [ReplicaManager broker=0] Stopping serving replicas in dir /var/lib/kafka/data-0/kafka-log0 (kafka.server.ReplicaManager) [LogDirFailureHandler]
2021-04-28 16:33:36,179 INFO Cleaner 0: Beginning cleaning of log __consumer_offsets-36. (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:36,179 INFO Cleaner 0: Building offset map for __consumer_offsets-36... (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:36,194 WARN [ReplicaManager broker=0] Broker 0 stopped fetcher for partitions and stopped moving logs for partitions because they are in the failed log directory /var/lib/kafka/data-0/kafka-log0. (kafka.server.ReplicaManager) [LogDirFailureHandler]
2021-04-28 16:33:36,203 INFO Cleaner 0: Building offset map for log __consumer_offsets-36 for 1 segments in offset range [0, 2183). (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:36,206 WARN Stopping serving logs in dir /var/lib/kafka/data-0/kafka-log0 (kafka.log.LogManager) [LogDirFailureHandler]
2021-04-28 16:33:36,214 INFO Cleaner 0: Offset map for log __consumer_offsets-36 complete. (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:36,225 INFO Cleaner 0: Cleaning log __consumer_offsets-36 (cleaning prior to Tue Apr 20 11:41:57 UTC 2021, discarding tombstones prior to Thu Jan 01 00:00:00 UTC 1970)... (kafka.log.LogCleaner) [kafka-log-cleaner-thread-0]
2021-04-28 16:33:36,229 ERROR Shutdown broker because all log dirs in /var/lib/kafka/data-0/kafka-log0 have failed (kafka.log.LogManager) [LogDirFailureHandler]

rjcastillejo-chile Apr 28, 2021
Author

I have two kafka strinzi clusters with different clients and they are giving me problems with this

rjcastillejo-chile Apr 28, 2021
Author

the pvc has 100gb, and I don't occupy 1gb

scholzj Apr 28, 2021
Maintainer

I do not think Azure File storage is a block storage. I think you need to use Azure Disk storage. That is normally causing this
kind of issues. It happens after one week because that is when often new log segment files are rolled out and old ones are deleted.

rjcastillejo-chile Apr 28, 2021
Author

ok

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

ERROR Shutdown broker because all log dirs in /var/lib/kafka/data-0/kafka-log0 have failed (kafka.log.LogManager) [LogDirFailureHandler] #4832

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 5 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Strimzi

ERROR Shutdown broker because all log dirs in /var/lib/kafka/data-0/kafka-log0 have failed (kafka.log.LogManager) [LogDirFailureHandler] #4832

Uh oh!

rjcastillejo-chile Apr 27, 2021

Replies: 1 comment · 5 replies

Uh oh!

scholzj Apr 27, 2021 Maintainer

Uh oh!

rjcastillejo-chile Apr 28, 2021 Author

Uh oh!

rjcastillejo-chile Apr 28, 2021 Author

Uh oh!

rjcastillejo-chile Apr 28, 2021 Author

Uh oh!

scholzj Apr 28, 2021 Maintainer

Uh oh!

rjcastillejo-chile Apr 28, 2021 Author

rjcastillejo-chile
Apr 27, 2021

Replies: 1 comment 5 replies

scholzj
Apr 27, 2021
Maintainer

rjcastillejo-chile Apr 28, 2021
Author

rjcastillejo-chile Apr 28, 2021
Author

rjcastillejo-chile Apr 28, 2021
Author

scholzj Apr 28, 2021
Maintainer

rjcastillejo-chile Apr 28, 2021
Author