Restarting POD taking lot of time to complete reading logs

[bj9306]

Pods are taking close to 40mins after restarting as it is reading existing logfiles. is there anyway to speed this up. below log is after 10mins into pod restart. is there anyway we can speed this up?

2021-09-27 17:29:19,002 INFO [Log partition=aiopx_avertackmetrics_famli_enriched-12, dir=/var/lib/kafka/data-0/kafka-log3] Loading producer state till offset 205302204 with message format version 2 (kafka.log.Log) [pool-6-thread-1]
2021-09-27 17:29:19,002 INFO [ProducerStateManager partition=aiopx_avertackmetrics_famli_enriched-12] Loading producer state from snapshot file 'SnapshotFile(/var/lib/kafka/data-0/kafka-log3/aiopx_avertackmetrics_famli_enriched-12/00000000000205302204.snapshot,205302204)' (kafka.log.ProducerStateManager) [pool-6-thread-1]
2021-09-27 17:29:19,005 INFO Completed load of Log(dir=/var/lib/kafka/data-0/kafka-log3/aiopx_avertackmetrics_famli_enriched-12, topic=aiopx_avertackmetrics_famli_enriched, partition=12, highWatermark=200491352, lastStableOffset=200491352, logStartOffset=200491352, logEndOffset=205302204) with 3 segments in 809ms (161/433 loaded in /var/lib/kafka/data-0/kafka-log3) (kafka.log.LogManager) [pool-6-thread-1]
2021-09-27 17:29:19,010 INFO [Log partition=lkl-stats-to-aiopxaggregator-enriched-12, dir=/var/lib/kafka/data-0/kafka-log3] Recovering unflushed segment 4774123 (kafka.log.Log) [pool-6-thread-1]
2021-09-27 17:29:19,010 INFO [Log partition=lkl-stats-to-aiopxaggregator-enriched-12, dir=/var/lib/kafka/data-0/kafka-log3] Loading producer state till offset 4774123 with message format version 2 (kafka.log.Log) [pool-6-thread-1]
2021-09-27 17:29:19,010 INFO [ProducerStateManager partition=lkl-stats-to-aiopxaggregator-enriched-12] Loading producer state from snapshot file 'SnapshotFile(/var/lib/kafka/data-0/kafka-log3/lkl-stats-to-aiopxaggregator-enriched-12/00000000000004774123.snapshot,4774123)' (kafka.log.ProducerStateManager) [pool-6-thread-1]

[scholzj]

That is hard to say without full logs. It seems to be loading the data from disk. You should check if your brokers terminated cleanly before this, maybe there is some storage damage. If they are not terminating cleanly during regular restarts, you might need to increase the termination grace period. But there could be also other reasons why they might not be terminating cleanly.

[bj9306]

i restarted pod using sts annotation and it is happening with that case too.

kubectl annotate statefulset watchtower-kafka strimzi.io/manual-rolling-update=true

[scholzj]

Then as I said, check if the termination grace period is long enough to allow the broker to shutdown cleanly.

[bj9306]

I have increased graceperiod. will let you know if this issue happens again.

[bj9306]

Thanks @scholzj , setting graceperiod fixed the issue