You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
we have a lot of broker restart and timeout connection with zookeeper. 2023-01-13 07:49:17,750 WARN Client session timed out, have not heard from server in 12400ms for session id 0x300bfb45a470457 (org.apache.zookeeper.ClientCnxn) [main-SendThread(midlayer-zookeeper-client:2181)] 2023-01-13 07:49:17,751 WARN Session 0x300bfb45a470457 for sever midlayer-zookeeper-client/10.111.15.14:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn) [main-SendThread(midlayer-zookeeper-client:2181)] org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 12400ms for session id 0x300bfb45a470457 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1258) 2023-01-13 07:49:17,760 INFO channel is disconnected: [id: 0x53b01562, L:/192.168.145.139:46308 ! R:midlayer-zookeeper-client/10.111.15.14:2181] (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1] 2023-01-13 07:49:17,760 INFO channel is told closing (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1] 2023-01-13 07:49:18,061 INFO [LogLoader partition=ngxp-cveshv-events-26, dir=/var/lib/kafka/data-0/kafka-log10] Producer state recovery took 9918ms for snapshot load and 0ms for segment recovery from offset 31261129 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:18,064 INFO Completed load of Log(dir=/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-26, topicId=OC0nTuN_R4eETDFDErH7jA, topic=ngxp-cveshv-events, partition=26, highWatermark=26031812, lastStableOffset=26031812, logStartOffset=26031812, logEndOffset=31261129) with 5 segments in 14757ms (50/55 loaded in /var/lib/kafka/data-0/kafka-log10) (kafka.log.LogManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:18,066 INFO [LogLoader partition=ngxp-cveshv-events-45, dir=/var/lib/kafka/data-0/kafka-log10] Recovering unflushed segment 31214242 (kafka.log.LogLoader) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:18,067 INFO [LogLoader partition=ngxp-cveshv-events-45, dir=/var/lib/kafka/data-0/kafka-log10] Loading producer state till offset 31214242 with message format version 2 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:18,067 INFO [LogLoader partition=ngxp-cveshv-events-45, dir=/var/lib/kafka/data-0/kafka-log10] Reloading from producer snapshot and rebuilding producer state from offset 31214242 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:18,067 INFO Deleted producer state snapshot /var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-45/00000000000032390217.snapshot (kafka.log.SnapshotFile) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:18,067 INFO [ProducerStateManager partition=ngxp-cveshv-events-45] Loading producer state from snapshot file 'SnapshotFile(/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-45/00000000000031214242.snapshot,31214242)' (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:18,676 INFO [LogLoader partition=ngxp-cveshv-events-45, dir=/var/lib/kafka/data-0/kafka-log10] Producer state recovery took 609ms for snapshot load and 0ms for segment recovery from offset 31214242 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:19,217 INFO Opening socket connection to server midlayer-zookeeper-client/10.111.15.14:2181. (org.apache.zookeeper.ClientCnxn) [main-SendThread(midlayer-zookeeper-client:2181)] 2023-01-13 07:49:19,367 INFO SSL handler added for channel: [id: 0x81aa767f] (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1] 2023-01-13 07:49:19,368 INFO Socket connection established, initiating session, client: /192.168.145.139:49788, server: midlayer-zookeeper-client/10.111.15.14:2181 (org.apache.zookeeper.ClientCnxn) [nioEventLoopGroup-2-1] 2023-01-13 07:49:19,370 INFO channel is connected: [id: 0x81aa767f, L:/192.168.145.139:49788 - R:midlayer-zookeeper-client/10.111.15.14:2181] (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1] 2023-01-13 07:49:19,667 INFO Session establishment complete on server midlayer-zookeeper-client/10.111.15.14:2181, session id = 0x300bfb45a470457, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn) [nioEventLoopGroup-2-1] 2023-01-13 07:49:23,614 INFO [ProducerStateManager partition=ngxp-cveshv-events-45] Wrote producer snapshot at offset 32390697 with 1201416 producer ids in 993 ms. (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:23,615 INFO [LogLoader partition=ngxp-cveshv-events-45, dir=/var/lib/kafka/data-0/kafka-log10] Loading producer state till offset 32390697 with message format version 2 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:23,615 INFO [LogLoader partition=ngxp-cveshv-events-45, dir=/var/lib/kafka/data-0/kafka-log10] Reloading from producer snapshot and rebuilding producer state from offset 32390697 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:23,616 INFO [ProducerStateManager partition=ngxp-cveshv-events-45] Loading producer state from snapshot file 'SnapshotFile(/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-45/00000000000032390697.snapshot,32390697)' (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:24,128 INFO [LogLoader partition=ngxp-cveshv-events-45, dir=/var/lib/kafka/data-0/kafka-log10] Producer state recovery took 513ms for snapshot load and 0ms for segment recovery from offset 32390697 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:24,130 INFO Completed load of Log(dir=/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-45, topicId=OC0nTuN_R4eETDFDErH7jA, topic=ngxp-cveshv-events, partition=45, highWatermark=26878353, lastStableOffset=26878353, logStartOffset=26878353, logEndOffset=32390697) with 5 segments in 6066ms (51/55 loaded in /var/lib/kafka/data-0/kafka-log10) (kafka.log.LogManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:24,134 INFO [LogLoader partition=ngxp-cveshv-events-135, dir=/var/lib/kafka/data-0/kafka-log10] Recovering unflushed segment 37123540 (kafka.log.LogLoader) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:24,134 INFO [LogLoader partition=ngxp-cveshv-events-135, dir=/var/lib/kafka/data-0/kafka-log10] Loading producer state till offset 37123540 with message format version 2 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:24,134 INFO [LogLoader partition=ngxp-cveshv-events-135, dir=/var/lib/kafka/data-0/kafka-log10] Reloading from producer snapshot and rebuilding producer state from offset 37123540 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:24,134 INFO [ProducerStateManager partition=ngxp-cveshv-events-135] Loading producer state from snapshot file 'SnapshotFile(/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-135/00000000000037123540.snapshot,37123540)' (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:24,986 INFO [LogLoader partition=ngxp-cveshv-events-135, dir=/var/lib/kafka/data-0/kafka-log10] Producer state recovery took 852ms for snapshot load and 0ms for segment recovery from offset 37123540 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:26,508 INFO [ProducerStateManager partition=ngxp-cveshv-events-135] Wrote producer snapshot at offset 37750795 with 1201657 producer ids in 795 ms. (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:26,510 INFO [LogLoader partition=ngxp-cveshv-events-135, dir=/var/lib/kafka/data-0/kafka-log10] Loading producer state till offset 37750795 with message format version 2 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:26,510 INFO [LogLoader partition=ngxp-cveshv-events-135, dir=/var/lib/kafka/data-0/kafka-log10] Reloading from producer snapshot and rebuilding producer state from offset 37750795 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:26,510 INFO [ProducerStateManager partition=ngxp-cveshv-events-135] Loading producer state from snapshot file 'SnapshotFile(/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-135/00000000000037750795.snapshot,37750795)' (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:36,502 INFO [LogLoader partition=ngxp-cveshv-events-135, dir=/var/lib/kafka/data-0/kafka-log10] Producer state recovery took 9991ms for snapshot load and 1ms for segment recovery from offset 37750795 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:36,504 INFO Completed load of Log(dir=/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-135, topicId=OC0nTuN_R4eETDFDErH7jA, topic=ngxp-cveshv-events, partition=135, highWatermark=32261570, lastStableOffset=32261570, logStartOffset=32261570, logEndOffset=37750795) with 5 segments in 12374ms (52/55 loaded in /var/lib/kafka/data-0/kafka-log10) (kafka.log.LogManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:36,507 INFO [LogLoader partition=ngxp-cveshv-events-97, dir=/var/lib/kafka/data-0/kafka-log10] Recovering unflushed segment 33289905 (kafka.log.LogLoader) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:36,507 INFO [LogLoader partition=ngxp-cveshv-events-97, dir=/var/lib/kafka/data-0/kafka-log10] Loading producer state till offset 33289905 with message format version 2 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:36,507 INFO [LogLoader partition=ngxp-cveshv-events-97, dir=/var/lib/kafka/data-0/kafka-log10] Reloading from producer snapshot and rebuilding producer state from offset 33289905 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:36,507 INFO Deleted producer state snapshot /var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-97/00000000000033773174.snapshot (kafka.log.SnapshotFile) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:36,507 INFO [ProducerStateManager partition=ngxp-cveshv-events-97] Loading producer state from snapshot file 'SnapshotFile(/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-97/00000000000033289905.snapshot,33289905)' (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:37,108 INFO [LogLoader partition=ngxp-cveshv-events-97, dir=/var/lib/kafka/data-0/kafka-log10] Producer state recovery took 601ms for snapshot load and 0ms for segment recovery from offset 33289905 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:39,812 INFO [ProducerStateManager partition=ngxp-cveshv-events-97] Wrote producer snapshot at offset 33777331 with 1202199 producer ids in 799 ms. (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:39,813 INFO [LogLoader partition=ngxp-cveshv-events-97, dir=/var/lib/kafka/data-0/kafka-log10] Loading producer state till offset 33777331 with message format version 2 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:39,813 INFO [LogLoader partition=ngxp-cveshv-events-97, dir=/var/lib/kafka/data-0/kafka-log10] Reloading from producer snapshot and rebuilding producer state from offset 33777331 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:39,813 INFO [ProducerStateManager partition=ngxp-cveshv-events-97] Loading producer state from snapshot file 'SnapshotFile(/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-97/00000000000033777331.snapshot,33777331)' (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:41,019 INFO [LogLoader partition=ngxp-cveshv-events-97, dir=/var/lib/kafka/data-0/kafka-log10] Producer state recovery took 1206ms for snapshot load and 0ms for segment recovery from offset 33777331 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:41,021 INFO Completed load of Log(dir=/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-97, topicId=OC0nTuN_R4eETDFDErH7jA, topic=ngxp-cveshv-events, partition=97, highWatermark=28845558, lastStableOffset=28845558, logStartOffset=28845558, logEndOffset=33777331) with 5 segments in 4518ms (53/55 loaded in /var/lib/kafka/data-0/kafka-log10) (kafka.log.LogManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:41,024 INFO [LogLoader partition=ngxp-cveshv-events-126, dir=/var/lib/kafka/data-0/kafka-log10] Recovering unflushed segment 40294969 (kafka.log.LogLoader) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:41,024 INFO [LogLoader partition=ngxp-cveshv-events-126, dir=/var/lib/kafka/data-0/kafka-log10] Loading producer state till offset 40294969 with message format version 2 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:41,024 INFO [LogLoader partition=ngxp-cveshv-events-126, dir=/var/lib/kafka/data-0/kafka-log10] Reloading from producer snapshot and rebuilding producer state from offset 40294969 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:41,024 INFO Deleted producer state snapshot /var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-126/00000000000041370665.snapshot (kafka.log.SnapshotFile) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:41,024 INFO [ProducerStateManager partition=ngxp-cveshv-events-126] Loading producer state from snapshot file 'SnapshotFile(/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-126/00000000000040294969.snapshot,40294969)' (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:50,314 WARN Client session timed out, have not heard from server in 14163ms for session id 0x300bfb45a470457 (org.apache.zookeeper.ClientCnxn) [main-SendThread(midlayer-zookeeper-client:2181)] 2023-01-13 07:49:50,314 WARN Session 0x300bfb45a470457 for sever midlayer-zookeeper-client/10.111.15.14:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn) [main-SendThread(midlayer-zookeeper-client:2181)] org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 14163ms for session id 0x300bfb45a470457 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1258) 2023-01-13 07:49:50,315 INFO channel is disconnected: [id: 0x81aa767f, L:/192.168.145.139:49788 ! R:midlayer-zookeeper-client/10.111.15.14:2181] (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1] 2023-01-13 07:49:50,315 INFO channel is told closing (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1] 2023-01-13 07:49:50,893 INFO [LogLoader partition=ngxp-cveshv-events-126, dir=/var/lib/kafka/data-0/kafka-log10] Producer state recovery took 9869ms for snapshot load and 0ms for segment recovery from offset 40294969 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:51,890 INFO Opening socket connection to server midlayer-zookeeper-client/10.111.15.14:2181. (org.apache.zookeeper.ClientCnxn) [main-SendThread(midlayer-zookeeper-client:2181)] 2023-01-13 07:49:51,990 INFO SSL handler added for channel: [id: 0xea6bff81] (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1] 2023-01-13 07:49:51,991 INFO Socket connection established, initiating session, client: /192.168.145.139:50294, server: midlayer-zookeeper-client/10.111.15.14:2181 (org.apache.zookeeper.ClientCnxn) [nioEventLoopGroup-2-1] 2023-01-13 07:49:51,991 INFO channel is connected: [id: 0xea6bff81, L:/192.168.145.139:50294 - R:midlayer-zookeeper-client/10.111.15.14:2181] (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1] 2023-01-13 07:49:52,063 INFO Session establishment complete on server midlayer-zookeeper-client/10.111.15.14:2181, session id = 0x300bfb45a470457, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn) [nioEventLoopGroup-2-1] 2023-01-13 07:49:55,362 INFO [ProducerStateManager partition=ngxp-cveshv-events-126] Wrote producer snapshot at offset 41371135 with 1201958 producer ids in 788 ms. (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:55,363 INFO [LogLoader partition=ngxp-cveshv-events-126, dir=/var/lib/kafka/data-0/kafka-log10] Loading producer state till offset 41371135 with message format version 2 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:55,363 INFO [LogLoader partition=ngxp-cveshv-events-126, dir=/var/lib/kafka/data-0/kafka-log10] Reloading from producer snapshot and rebuilding producer state from offset 41371135 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:55,363 INFO [ProducerStateManager partition=ngxp-cveshv-events-126] Loading producer state from snapshot file 'SnapshotFile(/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-126/00000000000041371135.snapshot,41371135)' (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:50:06,148 WARN Client session timed out, have not heard from server in 14085ms for session id 0x300bfb45a470457 (org.apache.zookeeper.ClientCnxn) [main-SendThread(midlayer-zookeeper-client:2181)] 2023-01-13 07:50:06,149 WARN Session 0x300bfb45a470457 for sever midlayer-zookeeper-client/10.111.15.14:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn) [main-SendThread(midlayer-zookeeper-client:2181)] org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 14085ms for session id 0x300bfb45a470457 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1258)
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Strimzi
Uh oh!
There was an error while loading. Please reload this page.
-
2023-01-13 07:49:17,750 WARN Client session timed out, have not heard from server in 12400ms for session id 0x300bfb45a470457 (org.apache.zookeeper.ClientCnxn) [main-SendThread(midlayer-zookeeper-client:2181)] 2023-01-13 07:49:17,751 WARN Session 0x300bfb45a470457 for sever midlayer-zookeeper-client/10.111.15.14:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn) [main-SendThread(midlayer-zookeeper-client:2181)] org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 12400ms for session id 0x300bfb45a470457 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1258) 2023-01-13 07:49:17,760 INFO channel is disconnected: [id: 0x53b01562, L:/192.168.145.139:46308 ! R:midlayer-zookeeper-client/10.111.15.14:2181] (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1] 2023-01-13 07:49:17,760 INFO channel is told closing (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1] 2023-01-13 07:49:18,061 INFO [LogLoader partition=ngxp-cveshv-events-26, dir=/var/lib/kafka/data-0/kafka-log10] Producer state recovery took 9918ms for snapshot load and 0ms for segment recovery from offset 31261129 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:18,064 INFO Completed load of Log(dir=/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-26, topicId=OC0nTuN_R4eETDFDErH7jA, topic=ngxp-cveshv-events, partition=26, highWatermark=26031812, lastStableOffset=26031812, logStartOffset=26031812, logEndOffset=31261129) with 5 segments in 14757ms (50/55 loaded in /var/lib/kafka/data-0/kafka-log10) (kafka.log.LogManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:18,066 INFO [LogLoader partition=ngxp-cveshv-events-45, dir=/var/lib/kafka/data-0/kafka-log10] Recovering unflushed segment 31214242 (kafka.log.LogLoader) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:18,067 INFO [LogLoader partition=ngxp-cveshv-events-45, dir=/var/lib/kafka/data-0/kafka-log10] Loading producer state till offset 31214242 with message format version 2 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:18,067 INFO [LogLoader partition=ngxp-cveshv-events-45, dir=/var/lib/kafka/data-0/kafka-log10] Reloading from producer snapshot and rebuilding producer state from offset 31214242 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:18,067 INFO Deleted producer state snapshot /var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-45/00000000000032390217.snapshot (kafka.log.SnapshotFile) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:18,067 INFO [ProducerStateManager partition=ngxp-cveshv-events-45] Loading producer state from snapshot file 'SnapshotFile(/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-45/00000000000031214242.snapshot,31214242)' (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:18,676 INFO [LogLoader partition=ngxp-cveshv-events-45, dir=/var/lib/kafka/data-0/kafka-log10] Producer state recovery took 609ms for snapshot load and 0ms for segment recovery from offset 31214242 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:19,217 INFO Opening socket connection to server midlayer-zookeeper-client/10.111.15.14:2181. (org.apache.zookeeper.ClientCnxn) [main-SendThread(midlayer-zookeeper-client:2181)] 2023-01-13 07:49:19,367 INFO SSL handler added for channel: [id: 0x81aa767f] (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1] 2023-01-13 07:49:19,368 INFO Socket connection established, initiating session, client: /192.168.145.139:49788, server: midlayer-zookeeper-client/10.111.15.14:2181 (org.apache.zookeeper.ClientCnxn) [nioEventLoopGroup-2-1] 2023-01-13 07:49:19,370 INFO channel is connected: [id: 0x81aa767f, L:/192.168.145.139:49788 - R:midlayer-zookeeper-client/10.111.15.14:2181] (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1] 2023-01-13 07:49:19,667 INFO Session establishment complete on server midlayer-zookeeper-client/10.111.15.14:2181, session id = 0x300bfb45a470457, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn) [nioEventLoopGroup-2-1] 2023-01-13 07:49:23,614 INFO [ProducerStateManager partition=ngxp-cveshv-events-45] Wrote producer snapshot at offset 32390697 with 1201416 producer ids in 993 ms. (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:23,615 INFO [LogLoader partition=ngxp-cveshv-events-45, dir=/var/lib/kafka/data-0/kafka-log10] Loading producer state till offset 32390697 with message format version 2 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:23,615 INFO [LogLoader partition=ngxp-cveshv-events-45, dir=/var/lib/kafka/data-0/kafka-log10] Reloading from producer snapshot and rebuilding producer state from offset 32390697 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:23,616 INFO [ProducerStateManager partition=ngxp-cveshv-events-45] Loading producer state from snapshot file 'SnapshotFile(/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-45/00000000000032390697.snapshot,32390697)' (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:24,128 INFO [LogLoader partition=ngxp-cveshv-events-45, dir=/var/lib/kafka/data-0/kafka-log10] Producer state recovery took 513ms for snapshot load and 0ms for segment recovery from offset 32390697 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:24,130 INFO Completed load of Log(dir=/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-45, topicId=OC0nTuN_R4eETDFDErH7jA, topic=ngxp-cveshv-events, partition=45, highWatermark=26878353, lastStableOffset=26878353, logStartOffset=26878353, logEndOffset=32390697) with 5 segments in 6066ms (51/55 loaded in /var/lib/kafka/data-0/kafka-log10) (kafka.log.LogManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:24,134 INFO [LogLoader partition=ngxp-cveshv-events-135, dir=/var/lib/kafka/data-0/kafka-log10] Recovering unflushed segment 37123540 (kafka.log.LogLoader) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:24,134 INFO [LogLoader partition=ngxp-cveshv-events-135, dir=/var/lib/kafka/data-0/kafka-log10] Loading producer state till offset 37123540 with message format version 2 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:24,134 INFO [LogLoader partition=ngxp-cveshv-events-135, dir=/var/lib/kafka/data-0/kafka-log10] Reloading from producer snapshot and rebuilding producer state from offset 37123540 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:24,134 INFO [ProducerStateManager partition=ngxp-cveshv-events-135] Loading producer state from snapshot file 'SnapshotFile(/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-135/00000000000037123540.snapshot,37123540)' (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:24,986 INFO [LogLoader partition=ngxp-cveshv-events-135, dir=/var/lib/kafka/data-0/kafka-log10] Producer state recovery took 852ms for snapshot load and 0ms for segment recovery from offset 37123540 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:26,508 INFO [ProducerStateManager partition=ngxp-cveshv-events-135] Wrote producer snapshot at offset 37750795 with 1201657 producer ids in 795 ms. (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:26,510 INFO [LogLoader partition=ngxp-cveshv-events-135, dir=/var/lib/kafka/data-0/kafka-log10] Loading producer state till offset 37750795 with message format version 2 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:26,510 INFO [LogLoader partition=ngxp-cveshv-events-135, dir=/var/lib/kafka/data-0/kafka-log10] Reloading from producer snapshot and rebuilding producer state from offset 37750795 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:26,510 INFO [ProducerStateManager partition=ngxp-cveshv-events-135] Loading producer state from snapshot file 'SnapshotFile(/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-135/00000000000037750795.snapshot,37750795)' (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:36,502 INFO [LogLoader partition=ngxp-cveshv-events-135, dir=/var/lib/kafka/data-0/kafka-log10] Producer state recovery took 9991ms for snapshot load and 1ms for segment recovery from offset 37750795 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:36,504 INFO Completed load of Log(dir=/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-135, topicId=OC0nTuN_R4eETDFDErH7jA, topic=ngxp-cveshv-events, partition=135, highWatermark=32261570, lastStableOffset=32261570, logStartOffset=32261570, logEndOffset=37750795) with 5 segments in 12374ms (52/55 loaded in /var/lib/kafka/data-0/kafka-log10) (kafka.log.LogManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:36,507 INFO [LogLoader partition=ngxp-cveshv-events-97, dir=/var/lib/kafka/data-0/kafka-log10] Recovering unflushed segment 33289905 (kafka.log.LogLoader) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:36,507 INFO [LogLoader partition=ngxp-cveshv-events-97, dir=/var/lib/kafka/data-0/kafka-log10] Loading producer state till offset 33289905 with message format version 2 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:36,507 INFO [LogLoader partition=ngxp-cveshv-events-97, dir=/var/lib/kafka/data-0/kafka-log10] Reloading from producer snapshot and rebuilding producer state from offset 33289905 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:36,507 INFO Deleted producer state snapshot /var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-97/00000000000033773174.snapshot (kafka.log.SnapshotFile) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:36,507 INFO [ProducerStateManager partition=ngxp-cveshv-events-97] Loading producer state from snapshot file 'SnapshotFile(/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-97/00000000000033289905.snapshot,33289905)' (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:37,108 INFO [LogLoader partition=ngxp-cveshv-events-97, dir=/var/lib/kafka/data-0/kafka-log10] Producer state recovery took 601ms for snapshot load and 0ms for segment recovery from offset 33289905 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:39,812 INFO [ProducerStateManager partition=ngxp-cveshv-events-97] Wrote producer snapshot at offset 33777331 with 1202199 producer ids in 799 ms. (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:39,813 INFO [LogLoader partition=ngxp-cveshv-events-97, dir=/var/lib/kafka/data-0/kafka-log10] Loading producer state till offset 33777331 with message format version 2 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:39,813 INFO [LogLoader partition=ngxp-cveshv-events-97, dir=/var/lib/kafka/data-0/kafka-log10] Reloading from producer snapshot and rebuilding producer state from offset 33777331 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:39,813 INFO [ProducerStateManager partition=ngxp-cveshv-events-97] Loading producer state from snapshot file 'SnapshotFile(/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-97/00000000000033777331.snapshot,33777331)' (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:41,019 INFO [LogLoader partition=ngxp-cveshv-events-97, dir=/var/lib/kafka/data-0/kafka-log10] Producer state recovery took 1206ms for snapshot load and 0ms for segment recovery from offset 33777331 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:41,021 INFO Completed load of Log(dir=/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-97, topicId=OC0nTuN_R4eETDFDErH7jA, topic=ngxp-cveshv-events, partition=97, highWatermark=28845558, lastStableOffset=28845558, logStartOffset=28845558, logEndOffset=33777331) with 5 segments in 4518ms (53/55 loaded in /var/lib/kafka/data-0/kafka-log10) (kafka.log.LogManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:41,024 INFO [LogLoader partition=ngxp-cveshv-events-126, dir=/var/lib/kafka/data-0/kafka-log10] Recovering unflushed segment 40294969 (kafka.log.LogLoader) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:41,024 INFO [LogLoader partition=ngxp-cveshv-events-126, dir=/var/lib/kafka/data-0/kafka-log10] Loading producer state till offset 40294969 with message format version 2 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:41,024 INFO [LogLoader partition=ngxp-cveshv-events-126, dir=/var/lib/kafka/data-0/kafka-log10] Reloading from producer snapshot and rebuilding producer state from offset 40294969 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:41,024 INFO Deleted producer state snapshot /var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-126/00000000000041370665.snapshot (kafka.log.SnapshotFile) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:41,024 INFO [ProducerStateManager partition=ngxp-cveshv-events-126] Loading producer state from snapshot file 'SnapshotFile(/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-126/00000000000040294969.snapshot,40294969)' (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:50,314 WARN Client session timed out, have not heard from server in 14163ms for session id 0x300bfb45a470457 (org.apache.zookeeper.ClientCnxn) [main-SendThread(midlayer-zookeeper-client:2181)] 2023-01-13 07:49:50,314 WARN Session 0x300bfb45a470457 for sever midlayer-zookeeper-client/10.111.15.14:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn) [main-SendThread(midlayer-zookeeper-client:2181)] org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 14163ms for session id 0x300bfb45a470457 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1258) 2023-01-13 07:49:50,315 INFO channel is disconnected: [id: 0x81aa767f, L:/192.168.145.139:49788 ! R:midlayer-zookeeper-client/10.111.15.14:2181] (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1] 2023-01-13 07:49:50,315 INFO channel is told closing (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1] 2023-01-13 07:49:50,893 INFO [LogLoader partition=ngxp-cveshv-events-126, dir=/var/lib/kafka/data-0/kafka-log10] Producer state recovery took 9869ms for snapshot load and 0ms for segment recovery from offset 40294969 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:51,890 INFO Opening socket connection to server midlayer-zookeeper-client/10.111.15.14:2181. (org.apache.zookeeper.ClientCnxn) [main-SendThread(midlayer-zookeeper-client:2181)] 2023-01-13 07:49:51,990 INFO SSL handler added for channel: [id: 0xea6bff81] (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1] 2023-01-13 07:49:51,991 INFO Socket connection established, initiating session, client: /192.168.145.139:50294, server: midlayer-zookeeper-client/10.111.15.14:2181 (org.apache.zookeeper.ClientCnxn) [nioEventLoopGroup-2-1] 2023-01-13 07:49:51,991 INFO channel is connected: [id: 0xea6bff81, L:/192.168.145.139:50294 - R:midlayer-zookeeper-client/10.111.15.14:2181] (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1] 2023-01-13 07:49:52,063 INFO Session establishment complete on server midlayer-zookeeper-client/10.111.15.14:2181, session id = 0x300bfb45a470457, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn) [nioEventLoopGroup-2-1] 2023-01-13 07:49:55,362 INFO [ProducerStateManager partition=ngxp-cveshv-events-126] Wrote producer snapshot at offset 41371135 with 1201958 producer ids in 788 ms. (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:55,363 INFO [LogLoader partition=ngxp-cveshv-events-126, dir=/var/lib/kafka/data-0/kafka-log10] Loading producer state till offset 41371135 with message format version 2 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:55,363 INFO [LogLoader partition=ngxp-cveshv-events-126, dir=/var/lib/kafka/data-0/kafka-log10] Reloading from producer snapshot and rebuilding producer state from offset 41371135 (kafka.log.UnifiedLog$) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:49:55,363 INFO [ProducerStateManager partition=ngxp-cveshv-events-126] Loading producer state from snapshot file 'SnapshotFile(/var/lib/kafka/data-0/kafka-log10/ngxp-cveshv-events-126/00000000000041371135.snapshot,41371135)' (kafka.log.ProducerStateManager) [log-recovery-/var/lib/kafka/data-0/kafka-log10] 2023-01-13 07:50:06,148 WARN Client session timed out, have not heard from server in 14085ms for session id 0x300bfb45a470457 (org.apache.zookeeper.ClientCnxn) [main-SendThread(midlayer-zookeeper-client:2181)] 2023-01-13 07:50:06,149 WARN Session 0x300bfb45a470457 for sever midlayer-zookeeper-client/10.111.15.14:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn) [main-SendThread(midlayer-zookeeper-client:2181)] org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 14085ms for session id 0x300bfb45a470457 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1258)Beta Was this translation helpful? Give feedback.
All reactions