Topic operator in CrashLoopBack after cert rotation

[SSgarima]

After Cert rotation Strimzi topic operator goes in CrashLoopBackOff state after cert rotation
We are using our custom ssl certs

Zookeeper logs:

2023-04-17 07:39:53,595 INFO Processing ruok command from /127.0.0.1:37364 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2023-04-17 07:39:53,625 INFO Processing ruok command from /127.0.0.1:37374 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2023-04-17 07:39:54,723 WARN Got zxid 0x600000001 expected 0x1 (org.apache.zookeeper.server.quorum.Learner) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
2023-04-17 07:39:54,723 INFO Creating new log file: log.600000001 (org.apache.zookeeper.server.persistence.FileTxnLog) [SyncThread:1]
2023-04-17 07:39:55,197 INFO Submitting global closeSession request for session 0x200386abbef0002 (org.apache.zookeeper.server.ZooKeeperServer) [RequestThrottler]
2023-04-17 07:39:58,294 ERROR Unsuccessful handshake with session 0x0 (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-4]
2023-04-17 07:39:58,294 WARN Exception caught (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-4]
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: decrypt_error
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: decrypt_error
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:358)
        at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:204)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
        at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736)
        at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)
        at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1343)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
        ... 17 more
2023-04-17 07:40:03,583 INFO Processing ruok command from /127.0.0.1:35982 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-3]
2023-04-17 07:40:03,613 INFO Processing ruok command from /127.0.0.1:35998 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]
2023-04-17 07:40:09,154 INFO Authenticated Id 'CN=aks-ddp-dev-foster-mural--cluster-kafka,O=io.strimzi' for Scheme 'x509' (org.apache.zookeeper.server.auth.X509AuthenticationProvider) [nioEventLoopGroup-7-1]
2023-04-17 07:40:09,162 INFO Committing global session 0x10038727cab0000 (org.apache.zookeeper.server.quorum.LearnerSessionTracker) [CommitProcessor:1]
2023-04-17 07:40:12,555 INFO Committing global session 0x2003872779b0000 (org.apache.zookeeper.server.quorum.LearnerSessionTracker) [CommitProcessor:1]
2023-04-17 07:40:13,570 INFO Processing ruok command from /127.0.0.1:60080 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2023-04-17 07:40:13,599 INFO Processing ruok command from /127.0.0.1:60092 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2023-04-17 07:40:23,583 INFO Processing ruok command from /127.0.0.1:49952 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-3]
2023-04-17 07:40:23,613 INFO Processing ruok command from /127.0.0.1:49960 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]
2023-04-17 07:40:25,022 INFO Submitting global closeSession request for session 0x100386a8cb80000 (org.apache.zookeeper.server.ZooKeeperServer) [RequestThrottler]
2023-04-17 07:40:33,566 INFO Processing ruok command from /127.0.0.1:56262 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2023-04-17 07:40:33,595 INFO Processing ruok command from /127.0.0.1:56264 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2023-04-17 07:40:40,021 INFO Authenticated Id 'CN=aks-ddp-dev-foster-mural--cluster-kafka,O=io.strimzi' for Scheme 'x509' (org.apache.zookeeper.server.auth.X509AuthenticationProvider) [nioEventLoopGroup-7-2]
2023-04-17 07:40:40,029 INFO Committing global session 0x10038727cab0001 (org.apache.zookeeper.server.quorum.LearnerSessionTracker) [CommitProcessor:1]
2023-04-17 07:40:43,582 INFO Processing ruok command from /127.0.0.1:49212 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-3]
2023-04-17 07:40:43,612 INFO Processing ruok command from /127.0.0.1:49224 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]
2023-04-17 07:40:43,676 INFO Authenticated Id 'CN=aks-ddp-dev-foster-mural--cluster-kafka,O=io.strimzi' for Scheme 'x509' (org.apache.zookeeper.server.auth.X509AuthenticationProvider) [nioEventLoopGroup-7-3]
2023-04-17 07:40:43,683 INFO Committing global session 0x10038727cab0002 (org.apache.zookeeper.server.quorum.LearnerSessionTracker) [CommitProcessor:1]
2023-04-17 07:40:53,578 INFO Processing ruok command from /127.0.0.1:60144 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2023-04-17 07:40:53,608 INFO Processing ruok command from /127.0.0.1:60160 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2023-04-17 07:41:03,566 INFO Processing ruok command from /127.0.0.1:46516 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-3]
2023-04-17 07:41:03,605 INFO Processing ruok command from /127.0.0.1:46518 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]
2023-04-17 07:41:10,807 INFO Committing global session 0x2003872779b0001 (org.apache.zookeeper.server.quorum.LearnerSessionTracker) [CommitProcessor:1]
2023-04-17 07:41:13,586 INFO Processing ruok command from /127.0.0.1:54240 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2023-04-17 07:41:13,617 INFO Processing ruok command from /127.0.0.1:54250 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2023-04-17 07:41:14,382 INFO Authenticated Id 'CN=aks-ddp-dev-foster-mural--cluster-kafka,O=io.strimzi' for Scheme 'x509' (org.apache.zookeeper.server.auth.X509AuthenticationProvider) [nioEventLoopGroup-7-4]
2023-04-17 07:41:14,388 INFO Committing global session 0x10038727cab0003 (org.apache.zookeeper.server.quorum.LearnerSessionTracker) [CommitProcessor:1]
2023-04-17 07:41:23,580 INFO Processing ruok command from /127.0.0.1:39714 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-3]
2023-04-17 07:41:23,608 INFO Processing ruok command from /127.0.0.1:39720 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]
2023-04-17 07:41:33,573 INFO Processing ruok command from /127.0.0.1:41494 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2023-04-17 07:41:33,601 INFO Processing ruok command from /127.0.0.1:41506 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2023-04-17 07:41:43,574 INFO Processing ruok command from /127.0.0.1:49102 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-3]
2023-04-17 07:41:43,603 INFO Processing ruok command from /127.0.0.1:49118 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]
2023-04-17 07:41:48,984 ERROR Unsuccessful handshake with session 0x0 (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-1]
2023-04-17 07:41:48,985 WARN Exception caught (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-1]
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: decrypt_error
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: decrypt_error
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:358)
        at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:204)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
        at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736)
        at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)
        at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1343)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
        ... 17 more
2023-04-17 07:41:53,571 INFO Processing ruok command from /127.0.0.1:43866 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2023-04-17 07:41:53,600 INFO Processing ruok command from /127.0.0.1:43874 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2023-04-17 07:42:03,574 INFO Processing ruok command from /127.0.0.1:40888 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-3]
2023-04-17 07:42:03,603 INFO Processing ruok command from /127.0.0.1:40904 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]
2023-04-17 07:42:13,578 INFO Processing ruok command from /127.0.0.1:48996 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2023-04-17 07:42:13,607 INFO Processing ruok command from /127.0.0.1:49010 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2023-04-17 07:42:23,566 INFO Processing ruok command from /127.0.0.1:35158 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-3]
2023-04-17 07:42:23,594 INFO Processing ruok command from /127.0.0.1:35160 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]
2023-04-17 07:42:33,585 INFO Processing ruok command from /127.0.0.1:46432 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2023-04-17 07:42:33,614 INFO Processing ruok command from /127.0.0.1:46440 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2023-04-17 07:42:43,570 INFO Processing ruok command from /127.0.0.1:59098 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-3]
2023-04-17 07:42:43,599 INFO Processing ruok command from /127.0.0.1:59100 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]
2023-04-17 07:42:53,115 ERROR Unsuccessful handshake with session 0x0 (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-2]
2023-04-17 07:42:53,115 WARN Exception caught (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-2]
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: decrypt_error
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: decrypt_error
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:358)
        at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:204)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
        at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736)
        at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)
        at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1343)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
        ... 17 more
2023-04-17 07:42:53,583 INFO Processing ruok command from /127.0.0.1:39130 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2023-04-17 07:42:53,611 INFO Processing ruok command from /127.0.0.1:39144 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2023-04-17 07:42:56,446 ERROR Unsuccessful handshake with session 0x0 (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-3]
2023-04-17 07:42:56,446 WARN Exception caught (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-3]
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: decrypt_error
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: decrypt_error
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:358)
        at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:204)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
        at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736)
        at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)
        at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1343)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
        ... 17 more
2023-04-17 07:43:03,574 INFO Processing ruok command from /127.0.0.1:48842 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-3]
2023-04-17 07:43:03,604 INFO Processing ruok command from /127.0.0.1:48850 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]
2023-04-17 07:43:13,558 INFO Processing ruok command from /127.0.0.1:36186 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2023-04-17 07:43:13,586 INFO Processing ruok command from /127.0.0.1:36188 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2023-04-17 07:43:23,573 INFO Processing ruok command from /127.0.0.1:57726 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-3]
2023-04-17 07:43:23,605 INFO Processing ruok command from /127.0.0.1:57740 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]
2023-04-17 07:43:33,593 INFO Processing ruok command from /127.0.0.1:56210 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2023-04-17 07:43:33,624 INFO Processing ruok command from /127.0.0.1:56220 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]

Topic Operator logs:

removed directory \'/tmp/hsperfdata_strimzi'
removed '/tmp/topic-operator/replication.keystore.p12'
removed '/tmp/topic-operator/replication.truststore.p12'
removed directory '/tmp/topic-operator'
Preparing trust store certificates for internal communication
Adding /etc/tls-sidecar/cluster-ca-certs/ca-2026-04-12T18-21-13Z.crt to truststore /tmp/topic-operator/replication.truststore.p12 with alias ca-2026-04-12T18-21-13Z
Certificate was added to keystore
Adding /etc/tls-sidecar/cluster-ca-certs/ca.crt to truststore /tmp/topic-operator/replication.truststore.p12 with alias ca
Certificate was added to keystore
Preparing trust store certificates for internal communication is completed
Preparing key store certificates for internal communication
Preparing key store certificates for internal communication is completed`
`+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/topic-operator/custom-config/log4j2.properties -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=15 -XX:MaxRAMPercentage=20 -classpath lib/io.strimzi.topic-operator-0.33.2.jar:lib/io.strimzi.api-0.33.2.jar:lib/io.fabric8.kubernetes-model-networking-6.3.1.jar:lib/io.fabric8.kubernetes-model-common-6.3.1.jar:lib/io.fabric8.kubernetes-model-apiextensions-6.3.1.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.14.1.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.14.1.jar:lib/io.fabric8.kubernetes-client-6.3.1.jar:lib/io.fabric8.kubernetes-httpclient-okhttp-6.3.1.jar:lib/com.squareup.okhttp3.okhttp-3.12.12.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/io.fabric8.kubernetes-client-api-6.3.1.jar:lib/io.fabric8.kubernetes-model-gatewayapi-6.3.1.jar:lib/io.fabric8.kubernetes-model-rbac-6.3.1.jar:lib/io.fabric8.kubernetes-model-admissionregistration-6.3.1.jar:lib/io.fabric8.kubernetes-model-apps-6.3.1.jar:lib/io.fabric8.kubernetes-model-autoscaling-6.3.1.jar:lib/io.fabric8.kubernetes-model-batch-6.3.1.jar:lib/io.fabric8.kubernetes-model-certificates-6.3.1.jar:lib/io.fabric8.kubernetes-model-coordination-6.3.1.jar:lib/io.fabric8.kubernetes-model-discovery-6.3.1.jar:lib/io.fabric8.kubernetes-model-events-6.3.1.jar:lib/io.fabric8.kubernetes-model-extensions-6.3.1.jar:lib/io.fabric8.kubernetes-model-flowcontrol-6.3.1.jar:lib/io.fabric8.kubernetes-model-metrics-6.3.1.jar:lib/io.fabric8.kubernetes-model-policy-6.3.1.jar:lib/io.fabric8.kubernetes-model-scheduling-6.3.1.jar:lib/io.fabric8.kubernetes-model-storageclass-6.3.1.jar:lib/io.fabric8.kubernetes-model-node-6.3.1.jar:lib/org.yaml.snakeyaml-1.33.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.14.1.jar:lib/io.fabric8.kubernetes-model-core-6.3.1.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/org.apache.logging.log4j.log4j-jul-2.17.2.jar:lib/org.apache.kafka.kafka-clients-3.4.0.jar:lib/com.github.luben.zstd-jni-1.5.2-1.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.4.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/com.101tec.zkclient-0.11.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.netty.netty-handler-4.1.86.Final.jar:lib/io.netty.netty-transport-native-unix-common-4.1.86.Final.jar:lib/io.netty.netty-codec-4.1.86.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.86.Final.jar:lib/io.netty.netty-transport-classes-epoll-4.1.86.Final.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/com.fasterxml.jackson.core.jackson-core-2.14.1.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.14.1.jar:lib/io.vertx.vertx-core-4.3.7.jar:lib/io.netty.netty-common-4.1.86.Final.jar:lib/io.netty.netty-buffer-4.1.86.Final.jar:lib/io.netty.netty-transport-4.1.86.Final.jar:lib/io.netty.netty-handler-proxy-4.1.86.Final.jar:lib/io.netty.netty-codec-socks-4.1.86.Final.jar:lib/io.netty.netty-codec-http-4.1.86.Final.jar:lib/io.netty.netty-codec-http2-4.1.86.Final.jar:lib/io.netty.netty-resolver-4.1.86.Final.jar:lib/io.netty.netty-resolver-dns-4.1.86.Final.jar:lib/io.netty.netty-codec-dns-4.1.86.Final.jar:lib/io.apicurio.apicurio-registry-utils-streams-1.3.2.Final.jar:lib/jakarta.annotation.jakarta.annotation-api-1.3.5.jar:lib/io.apicurio.apicurio-registry-utils-kafka-1.3.2.Final.jar:lib/org.eclipse.microprofile.config.microprofile-config-api-1.4.jar:lib/org.apache.kafka.kafka-streams-3.4.0.jar:lib/org.rocksdb.rocksdbjni-7.1.2.jar:lib/io.fabric8.openshift-client-api-6.3.1.jar:lib/io.fabric8.openshift-model-clusterautoscaling-6.3.1.jar:lib/io.fabric8.openshift-model-operator-6.3.1.jar:lib/io.fabric8.openshift-model-operatorhub-6.3.1.jar:lib/io.fabric8.openshift-model-machine-6.3.1.jar:lib/io.fabric8.openshift-model-whereabouts-6.3.1.jar:lib/io.fabric8.openshift-model-monitoring-6.3.1.jar:lib/io.fabric8.openshift-model-storageversionmigrator-6.3.1.jar:lib/io.fabric8.openshift-model-tuned-6.3.1.jar:lib/io.fabric8.openshift-model-console-6.3.1.jar:lib/io.fabric8.openshift-model-config-6.3.1.jar:lib/io.fabric8.openshift-model-machineconfig-6.3.1.jar:lib/io.fabric8.openshift-model-miscellaneous-6.3.1.jar:lib/io.fabric8.openshift-model-hive-6.3.1.jar:lib/io.fabric8.openshift-model-installer-6.3.1.jar:lib/io.strimzi.operator-common-0.33.2.jar:lib/io.strimzi.crd-annotations-0.33.2.jar:lib/io.strimzi.certificate-manager-0.33.2.jar:lib/io.fabric8.openshift-model-6.3.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/io.micrometer.micrometer-core-1.9.5.jar:lib/org.hdrhistogram.HdrHistogram-2.1.12.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.micrometer.micrometer-registry-prometheus-1.9.5.jar:lib/io.prometheus.simpleclient_common-0.15.0.jar:lib/io.prometheus.simpleclient-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_common-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel_agent-0.15.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.3.7.jar io.strimzi.operator.topic.Main
2023-04-17 07:51:31,93600 INFO  [main] Main:38 - TopicOperator 0.33.2 is starting
2023-04-17 07:51:35,43588 INFO  [main] Session:121 - Using config:
        STRIMZI_TRUSTSTORE_LOCATION: /tmp/topic-operator/replication.truststore.p12
        STRIMZI_RESOURCE_LABELS: strimzi.io/cluster=aks-ddp-dev-foster-mural--cluster
        STRIMZI_SASL_MECHANISM:
        STRIMZI_SASL_USERNAME:
        STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
        STRIMZI_CLIENT_ID: strimzi-topic-operator-a172a993-d996-4358-9f75-d579a63e4e14
        STRIMZI_SECURITY_PROTOCOL: SSL
        STRIMZI_STALE_RESULT_TIMEOUT_MS: 5000
        STRIMZI_TOPIC_METADATA_MAX_ATTEMPTS: 6
        STRIMZI_KEYSTORE_LOCATION: /tmp/topic-operator/replication.keystore.p12
        STRIMZI_REASSIGN_THROTTLE: 9223372036854775807
        STRIMZI_USE_ZOOKEEPER_TOPIC_STORE: false
        STRIMZI_SASL_PASSWORD: ********
        STRIMZI_KAFKA_BOOTSTRAP_SERVERS: aks-ddp-dev-foster-mural--cluster-kafka-bootstrap:9091
        STRIMZI_SASL_ENABLED: false
        STRIMZI_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: HTTPS
        STRIMZI_NAMESPACE: ddp-foundation-services
        STRIMZI_APPLICATION_ID: __strimzi-topic-operator-kstreams
        STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS: 60000
        STRIMZI_TOPICS_PATH: /strimzi/topics
        STRIMZI_ZOOKEEPER_CONNECT: localhost:2181
        STRIMZI_TLS_ENABLED: true
        STRIMZI_KEYSTORE_PASSWORD: ********
        STRIMZI_STORE_NAME: topic-store
        STRIMZI_REASSIGN_VERIFY_INTERVAL_MS: 120000
        TC_ZK_CONNECTION_TIMEOUT_MS: 18000
        STRIMZI_TRUSTSTORE_PASSWORD: ********
        STRIMZI_STORE_TOPIC: __strimzi_store_topic
2023-04-17 07:51:35,52355 INFO  [vert.x-eventloop-thread-0] Session:209 - Starting
2023-04-17 07:51:36,62533 INFO  [vert.x-eventloop-thread-0] AppInfoParser:119 - Kafka version: 3.4.0
2023-04-17 07:51:36,62556 INFO  [vert.x-eventloop-thread-0] AppInfoParser:120 - Kafka commitId: 2e1947d240607d53
2023-04-17 07:51:36,62569 INFO  [vert.x-eventloop-thread-0] AppInfoParser:121 - Kafka startTimeMs: 1681717896623
2023-04-17 07:51:37,42028 INFO  [ZkClient-EventThread-23-localhost:2181] ZkEventThread:65 - Starting ZkClient event thread.
2023-04-17 07:51:37,42448 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT
2023-04-17 07:51:37,42488 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:host.name=aks-ddp-dev-foster-mural--cluster-entity-operator-57bbd5c5t7xpf
2023-04-17 07:51:37,42500 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.version=17.0.6
2023-04-17 07:51:37,42511 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.vendor=Red Hat, Inc.
2023-04-17 07:51:37,42523 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.home=/usr/lib/jvm/java-17-openjdk-17.0.6.0.10-3.el8_7.x86_64
2023-04-17 07:51:37,42536 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.class.path=lib/io.strimzi.topic-operator-0.33.2.jar:lib/io.strimzi.api-0.33.2.jar:lib/io.fabric8.kubernetes-model-networking-6.3.1.jar:lib/io.fabric8.kubernetes-model-common-6.3.1.jar:lib/io.fabric8.kubernetes-model-apiextensions-6.3.1.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.14.1.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.14.1.jar:lib/io.fabric8.kubernetes-client-6.3.1.jar:lib/io.fabric8.kubernetes-httpclient-okhttp-6.3.1.jar:lib/com.squareup.okhttp3.okhttp-3.12.12.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/io.fabric8.kubernetes-client-api-6.3.1.jar:lib/io.fabric8.kubernetes-model-gatewayapi-6.3.1.jar:lib/io.fabric8.kubernetes-model-rbac-6.3.1.jar:lib/io.fabric8.kubernetes-model-admissionregistration-6.3.1.jar:lib/io.fabric8.kubernetes-model-apps-6.3.1.jar:lib/io.fabric8.kubernetes-model-autoscaling-6.3.1.jar:lib/io.fabric8.kubernetes-model-batch-6.3.1.jar:lib/io.fabric8.kubernetes-model-certificates-6.3.1.jar:lib/io.fabric8.kubernetes-model-coordination-6.3.1.jar:lib/io.fabric8.kubernetes-model-discovery-6.3.1.jar:lib/io.fabric8.kubernetes-model-events-6.3.1.jar:lib/io.fabric8.kubernetes-model-extensions-6.3.1.jar:lib/io.fabric8.kubernetes-model-flowcontrol-6.3.1.jar:lib/io.fabric8.kubernetes-model-metrics-6.3.1.jar:lib/io.fabric8.kubernetes-model-policy-6.3.1.jar:lib/io.fabric8.kubernetes-model-scheduling-6.3.1.jar:lib/io.fabric8.kubernetes-model-storageclass-6.3.1.jar:lib/io.fabric8.kubernetes-model-node-6.3.1.jar:lib/org.yaml.snakeyaml-1.33.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.14.1.jar:lib/io.fabric8.kubernetes-model-core-6.3.1.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/org.apache.logging.log4j.log4j-jul-2.17.2.jar:lib/org.apache.kafka.kafka-clients-3.4.0.jar:lib/com.github.luben.zstd-jni-1.5.2-1.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.4.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/com.101tec.zkclient-0.11.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.netty.netty-handler-4.1.86.Final.jar:lib/io.netty.netty-transport-native-unix-common-4.1.86.Final.jar:lib/io.netty.netty-codec-4.1.86.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.86.Final.jar:lib/io.netty.netty-transport-classes-epoll-4.1.86.Final.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/com.fasterxml.jackson.core.jackson-core-2.14.1.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.14.1.jar:lib/io.vertx.vertx-core-4.3.7.jar:lib/io.netty.netty-common-4.1.86.Final.jar:lib/io.netty.netty-buffer-4.1.86.Final.jar:lib/io.netty.netty-transport-4.1.86.Final.jar:lib/io.netty.netty-handler-proxy-4.1.86.Final.jar:lib/io.netty.netty-codec-socks-4.1.86.Final.jar:lib/io.netty.netty-codec-http-4.1.86.Final.jar:lib/io.netty.netty-codec-http2-4.1.86.Final.jar:lib/io.netty.netty-resolver-4.1.86.Final.jar:lib/io.netty.netty-resolver-dns-4.1.86.Final.jar:lib/io.netty.netty-codec-dns-4.1.86.Final.jar:lib/io.apicurio.apicurio-registry-utils-streams-1.3.2.Final.jar:lib/jakarta.annotation.jakarta.annotation-api-1.3.5.jar:lib/io.apicurio.apicurio-registry-utils-kafka-1.3.2.Final.jar:lib/org.eclipse.microprofile.config.microprofile-config-api-1.4.jar:lib/org.apache.kafka.kafka-streams-3.4.0.jar:lib/org.rocksdb.rocksdbjni-7.1.2.jar:lib/io.fabric8.openshift-client-api-6.3.1.jar:lib/io.fabric8.openshift-model-clusterautoscaling-6.3.1.jar:lib/io.fabric8.openshift-model-operator-6.3.1.jar:lib/io.fabric8.openshift-model-operatorhub-6.3.1.jar:lib/io.fabric8.openshift-model-machine-6.3.1.jar:lib/io.fabric8.openshift-model-whereabouts-6.3.1.jar:lib/io.fabric8.openshift-model-monitoring-6.3.1.jar:lib/io.fabric8.openshift-model-storageversionmigrator-6.3.1.jar:lib/io.fabric8.openshift-model-tuned-6.3.1.jar:lib/io.fabric8.openshift-model-console-6.3.1.jar:lib/io.fabric8.openshift-model-config-6.3.1.jar:lib/io.fabric8.openshift-model-machineconfig-6.3.1.jar:lib/io.fabric8.openshift-model-miscellaneous-6.3.1.jar:lib/io.fabric8.openshift-model-hive-6.3.1.jar:lib/io.fabric8.openshift-model-installer-6.3.1.jar:lib/io.strimzi.operator-common-0.33.2.jar:lib/io.strimzi.crd-annotations-0.33.2.jar:lib/io.strimzi.certificate-manager-0.33.2.jar:lib/io.fabric8.openshift-model-6.3.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/io.micrometer.micrometer-core-1.9.5.jar:lib/org.hdrhistogram.HdrHistogram-2.1.12.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.micrometer.micrometer-registry-prometheus-1.9.5.jar:lib/io.prometheus.simpleclient_common-0.15.0.jar:lib/io.prometheus.simpleclient-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_common-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel_agent-0.15.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.3.7.jar
2023-04-17 07:51:37,42564 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
2023-04-17 07:51:37,42576 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.io.tmpdir=/tmp
2023-04-17 07:51:37,43085 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.compiler=<NA>
2023-04-17 07:51:37,43109 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:os.name=Linux
2023-04-17 07:51:37,43126 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:os.arch=amd64
2023-04-17 07:51:37,43138 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:os.version=5.4.0-1104-azure
2023-04-17 07:51:37,43148 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:user.name=strimzi
2023-04-17 07:51:37,43158 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:user.home=/home/strimzi
2023-04-17 07:51:37,43168 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:user.dir=/opt/strimzi
2023-04-17 07:51:37,43178 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:os.memory.free=9MB
2023-04-17 07:51:37,43188 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:os.memory.max=193MB
2023-04-17 07:51:37,43204 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:os.memory.total=34MB
2023-04-17 07:51:37,43402 INFO  [vert.x-worker-thread-0] ZooKeeper:1006 - Initiating client connection, connectString=localhost:2181 sessionTimeout=60000 watcher=org.I0Itec.zkclient.ZkClient@e9f0e78
2023-04-17 07:51:37,51862 INFO  [vert.x-worker-thread-0] X509Util:77 - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2023-04-17 07:51:37,52194 INFO  [vert.x-worker-thread-0] ClientCnxnSocket:239 - jute.maxbuffer value is 1048575 Bytes
2023-04-17 07:51:37,52688 INFO  [vert.x-worker-thread-0] ClientCnxn:1736 - zookeeper.request.timeout value is 0. feature enabled=false
2023-04-17 07:51:37,52751 INFO  [vert.x-worker-thread-0] ZkClient:936 - Waiting for keeper state SyncConnected
2023-04-17 07:51:37,61670 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/127.0.0.1:2181.
2023-04-17 07:51:37,61724 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-04-17 07:51:37,61791 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1013 - Socket connection established, initiating session, client: /127.0.0.1:51458, server: localhost/127.0.0.1:2181
2023-04-17 07:51:37,64781 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.SocketException: Connection reset
        at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:74) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-04-17 07:51:39,31953 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/127.0.0.1:2181.
2023-04-17 07:51:39,31980 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-04-17 07:51:39,32040 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1013 - Socket connection established, initiating session, client: /127.0.0.1:51470, server: localhost/127.0.0.1:2181
2023-04-17 07:51:39,33453 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.SocketException: Connection reset
        at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:74) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-04-17 07:51:40,96993 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/127.0.0.1:2181.
2023-04-17 07:51:40,97020 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-04-17 07:51:40,97068 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1013 - Socket connection established, initiating session, client: /127.0.0.1:58158, server: localhost/127.0.0.1:2181
2023-04-17 07:51:40,98301 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.SocketException: Connection reset
        at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:74) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-04-17 07:51:42,25564 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181.
2023-04-17 07:51:42,25594 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-04-17 07:51:42,25676 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/[0:0:0:0:0:0:0:1]:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-04-17 07:51:44,35712 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/127.0.0.1:2181.
2023-04-17 07:51:44,35739 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-04-17 07:51:44,35788 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1013 - Socket connection established, initiating session, client: /127.0.0.1:58160, server: localhost/127.0.0.1:2181
2023-04-17 07:51:44,37096 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.SocketException: Connection reset
        at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:74) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-04-17 07:51:45,62829 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/127.0.0.1:2181.
2023-04-17 07:51:45,62855 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-04-17 07:51:45,62901 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1013 - Socket connection established, initiating session, client: /127.0.0.1:58166, server: localhost/127.0.0.1:2181
2023-04-17 07:51:45,64064 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.SocketException: Connection reset
        at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:74) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-04-17 07:51:47,55605 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/127.0.0.1:2181.
2023-04-17 07:51:47,55633 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-04-17 07:51:47,55689 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1013 - Socket connection established, initiating session, client: /127.0.0.1:58180, server: localhost/127.0.0.1:2181
2023-04-17 07:51:47,56962 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.SocketException: Connection reset
        at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:74) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-04-17 07:51:49,09798 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181.
2023-04-17 07:51:49,09824 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-04-17 07:51:49,09879 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/[0:0:0:0:0:0:0:1]:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-04-17 07:51:50,57207 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181.
2023-04-17 07:51:50,57240 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-04-17 07:51:50,57294 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/[0:0:0:0:0:0:0:1]:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-04-17 07:51:52,26919 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181.
2023-04-17 07:51:52,26949 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-04-17 07:51:52,26999 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/[0:0:0:0:0:0:0:1]:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-04-17 07:51:53,99932 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181.
2023-04-17 07:51:53,99959 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-04-17 07:51:54,00012 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/[0:0:0:0:0:0:0:1]:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-04-17 07:51:55,52553 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/127.0.0.1:2181.
2023-04-17 07:51:55,52576 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-04-17 07:51:55,52634 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1013 - Socket connection established, initiating session, client: /127.0.0.1:54168, server: localhost/127.0.0.1:2181
2023-04-17 07:51:55,52757 INFO  [ZkClient-EventThread-23-localhost:2181] ZkEventThread:83 - Terminate ZkClient event thread.
2023-04-17 07:51:55,53823 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1294 - An exception was thrown while closing send thread for session 0x0.
java.net.SocketException: Connection reset
        at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:74) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-04-17 07:51:55,64235 INFO  [vert.x-worker-thread-0] ZooKeeper:1619 - Session: 0x0 closed
2023-04-17 07:51:55,64287 INFO  [vert.x-worker-thread-0-EventThread] ClientCnxn:578 - EventThread shut down for session: 0x0
2023-04-17 07:51:55,64342 ERROR [vert.x-eventloop-thread-0] Session:278 - Failed to start ZK client
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server 'localhost:2181' with timeout of 18000 ms
        at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1233) ~[com.101tec.zkclient-0.11.jar:?]
        at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157) ~[com.101tec.zkclient-0.11.jar:?]
        at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131) ~[com.101tec.zkclient-0.11.jar:?]
        at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98) ~[com.101tec.zkclient-0.11.jar:?]
        at io.strimzi.operator.topic.zk.Zk.createSync(Zk.java:53) ~[io.strimzi.topic-operator-0.33.2.jar:0.33.2]
        at io.strimzi.operator.topic.zk.Zk.lambda$create$0(Zk.java:36) ~[io.strimzi.topic-operator-0.33.2.jar:0.33.2]
        at io.vertx.core.impl.ContextBase.lambda$null$0(ContextBase.java:137) ~[io.vertx.vertx-core-4.3.7.jar:4.3.7]
        at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264) ~[io.vertx.vertx-core-4.3.7.jar:4.3.7]
        at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:135) ~[io.vertx.vertx-core-4.3.7.jar:4.3.7]
        at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.3.7.jar:4.3.7]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.86.Final.jar:4.1.86.Final]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-04-17 07:51:55,65189 ERROR [vert.x-eventloop-thread-0] Session:246 - Topic operator start up failed, cause was
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server 'localhost:2181' with timeout of 18000 ms
        at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1233) ~[com.101tec.zkclient-0.11.jar:?]
        at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157) ~[com.101tec.zkclient-0.11.jar:?]
        at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131) ~[com.101tec.zkclient-0.11.jar:?]
        at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98) ~[com.101tec.zkclient-0.11.jar:?]
        at io.strimzi.operator.topic.zk.Zk.createSync(Zk.java:53) ~[io.strimzi.topic-operator-0.33.2.jar:0.33.2]
        at io.strimzi.operator.topic.zk.Zk.lambda$create$0(Zk.java:36) ~[io.strimzi.topic-operator-0.33.2.jar:0.33.2]
        at io.vertx.core.impl.ContextBase.lambda$null$0(ContextBase.java:137) ~[io.vertx.vertx-core-4.3.7.jar:4.3.7]
        at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264) ~[io.vertx.vertx-core-4.3.7.jar:4.3.7]
        at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:135) ~[io.vertx.vertx-core-4.3.7.jar:4.3.7]
        at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.3.7.jar:4.3.7]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.86.Final.jar:4.1.86.Final]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-04-17 07:51:55,65216 ERROR [vert.x-eventloop-thread-1] Main:70 - Error deploying Session
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server 'localhost:2181' with timeout of 18000 ms
        at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1233) ~[com.101tec.zkclient-0.11.jar:?]
        at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157) ~[com.101tec.zkclient-0.11.jar:?]
        at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131) ~[com.101tec.zkclient-0.11.jar:?]
        at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98) ~[com.101tec.zkclient-0.11.jar:?]
        at io.strimzi.operator.topic.zk.Zk.createSync(Zk.java:53) ~[io.strimzi.topic-operator-0.33.2.jar:0.33.2]
        at io.strimzi.operator.topic.zk.Zk.lambda$create$0(Zk.java:36) ~[io.strimzi.topic-operator-0.33.2.jar:0.33.2]
        at io.vertx.core.impl.ContextBase.lambda$null$0(ContextBase.java:137) ~[io.vertx.vertx-core-4.3.7.jar:4.3.7]
        at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264) ~[io.vertx.vertx-core-4.3.7.jar:4.3.7]
        at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:135) ~[io.vertx.vertx-core-4.3.7.jar:4.3.7]
        at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.3.7.jar:4.3.7]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.86.Final.jar:4.1.86.Final]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]

[scholzj]

Can you please properly format the logs to make it readable?

[SSgarima]

Sure, Done

[scholzj]

So, you use your own custom Cluster CA and you renewed it by following this procedure: https://strimzi.io/docs/operators/latest/full/deploying.html#renewing-your-own-ca-certificates-str?

Or did you replaced the private key and followed this procedure: https://strimzi.io/docs/operators/latest/full/deploying.html#proc-replacing-your-own-private-keys-str?

Also, in which phase of the renewal did it failed? And did you used a new CN for the new CA certificate?

[SSgarima]

I am following procedure mentioned here: https://strimzi.io/docs/operators/latest/full/deploying.html#proc-replacing-your-own-private-keys-str
Using Same CN
Kafka , Zookeeper both are working, its failing only failing for topic operator
Because topic operator keeps failing Strimzi operator keeps restarting all pods after some 8mins

[scholzj]

Well, as the documentation says: Before going through the following steps, make sure that the CN (Common Name) of the new CA certificate is different from the current one.. So that is likely causing the issue in your case.

Maybe to work around it, you can try to disable the Topic Operator for the time being and enable it again only after the renewal is complete. (But I never tried this -> so it is just an untested idea)