Kafka not creating service brokers #10797

brunooliveiramac · 2024-11-04T14:53:09Z

brunooliveiramac
Nov 4, 2024

I am facing some weird issue with Kafka in GKE 1.30/1.31

Im not see my dev environment creating the service brokers.. The same code works in the other envs.

I see errors to solve Zookeeper host, from the operator, but the zookeepers pods seems helth.

java.net.UnknownHostException: asgard-cluster-zookeeper-1.asgard-cluster-zookeeper-nodes.kafka-operator.svc: Name or service not known
	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
	at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934) ~[?:?]
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543) ~[?:?]
	at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852) ~[?:?]
	at java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[?:?]
	at java.net.InetAddress.getAllByName(InetAddress.java:1385) ~[?:?]
	at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?]
	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88) ~[org.apache.zookeeper.zookeeper-3.8.4.jar:3.8.4]
	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141) ~[org.apache.zookeeper.zookeeper-3.8.4.jar:3.8.4]
	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368) ~[org.apache.zookeeper.zookeeper-3.8.4.jar:3.8.4]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204) ~[org.apache.zookeeper.zookeeper-3.8.4.jar:3.8.4]
2024-11-04 14:50:08 WARN  ClientCnxnSocketNetty:151 - future isn't success.
java.net.UnknownHostException: asgard-cluster-zookeeper-1.asgard-cluster-zookeeper-nodes.kafka-operator.svc
	at java.net.InetAddress$CachedAddresses.get(InetAddress.java:801) ~[?:?]
	at java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[?:?]
	at java.net.InetAddress.getAllByName(InetAddress.java:1385) ~[?:?]
	at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?]
	at java.net.InetAddress.getByName(InetAddress.java:1256) ~[?:?]
	at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:156) ~[io.netty.netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:153) ~[io.netty.netty-common-4.1.111.Final.jar:4.1.111.Final]
	at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
	at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:153) ~[io.netty.netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:41) ~[io.netty.netty-resolver-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:61) ~[io.netty.netty-resolver-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:53) ~[io.netty.netty-resolver-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:55) ~[io.netty.netty-resolver-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:31) ~[io.netty.netty-resolver-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:106) ~[io.netty.netty-resolver-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:220) ~[io.netty.netty-transport-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:47) ~[io.netty.netty-transport-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:189) ~[io.netty.netty-transport-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:175) ~[io.netty.netty-transport-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty.netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[io.netty.netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty.netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty.netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) ~[io.netty.netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) ~[io.netty.netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[io.netty.netty-transport-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:988) ~[io.netty.netty-transport-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:515) ~[io.netty.netty-transport-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:428) ~[io.netty.netty-transport-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:485) ~[io.netty.netty-transport-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[io.netty.netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[io.netty.netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[io.netty.netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405) ~[io.netty.netty-transport-classes-epoll-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) ~[io.netty.netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.111.Final.jar:4.1.111.Final]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]

and some logs from a zookeper node:

2024-11-04 14:28:08,063 INFO Learner received UPTODATE message (org.apache.zookeeper.server.quorum.Learner) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)]
2024-11-04 14:28:08,064 INFO Peer state changed: following - broadcast (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)]
2024-11-04 14:28:13,994 INFO Processing ruok command from /127.0.0.1:44112 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2024-11-04 14:28:13,994 INFO Processing ruok command from /127.0.0.1:44126 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2024-11-04 14:28:15,813 INFO Notification: my state:FOLLOWING; n.sid:3, n.state:LOOKING, n.leader:3, n.round:0xf, n.peerEpoch:0x5, n.zxid:0x500000000, message format version:0x2, n.config version:0x300000000 (org.apache.zookeeper.server.quorum.FastLeaderElection) [WorkerReceiver[myid=1]]
2024-11-04 14:28:18,148 WARN Connection broken for id 3, my id = 1 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [RecvWorker:3]
java.net.SocketException: Socket is closed
	at java.base/java.net.Socket.shutdownInput(Socket.java:1601)
	at java.base/sun.security.ssl.BaseSSLSocketImpl.shutdownInput(BaseSSLSocketImpl.java:219)
	at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:853)
	at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:825)
	at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1733)
	at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1467)
	at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1069)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.read(UnifiedServerSocket.java:693)
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:263)
	at java.base/java.io.DataInputStream.readInt(DataInputStream.java:381)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1390)
2024-11-04 14:28:18,149 WARN Interrupting SendWorker thread from RecvWorker. sid: 3. myId: 1 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [RecvWorker:3]
2024-11-04 14:28:18,149 WARN Interrupted while waiting for message on queue (org.apache.zookeeper.server.quorum.QuorumCnxManager) [SendWorker:3]
java.lang.InterruptedException
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1686)
	at org.apache.zookeeper.util.CircularBlockingQueue.poll(CircularBlockingQueue.java:105)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1453)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$900(QuorumCnxManager.java:99)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1277)
2024-11-04 14:28:18,149 WARN Send worker leaving thread id 3 my id = 1 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [SendWorker:3]
2024-11-04 14:28:23,990 INFO Processing ruok command from /127.0.0.1:55490 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2024-11-04 14:28:23,991 INFO Processing ruok command from /127.0.0.1:55504 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2024-11-04 14:28:33,728 INFO Received connection request from /10.221.11.8:51712 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-asgard-cluster-zookeeper-0.asgard-cluster-zookeeper-nodes.kafka-operator.svc/10.221.8.8:3888]
2024-11-04 14:28:34,073 INFO Processing ruok command from /127.0.0.1:35496 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2024-11-04 14:28:34,076 INFO Processing ruok command from /127.0.0.1:35512 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2024-11-04 14:28:34,482 INFO Accepted TLS connection from /10.221.11.8:51712 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (org.apache.zookeeper.server.quorum.UnifiedServerSocket) [ListenerHandler-asgard-cluster-zookeeper-0.asgard-cluster-zookeeper-nodes.kafka-operator.svc/10.221.8.8:3888]
2024-11-04 14:28:34,498 INFO Notification: my state:FOLLOWING; n.sid:3, n.state:LOOKING, n.leader:3, n.round:0x1, n.peerEpoch:0x6, n.zxid:0x600000000, message format version:0x2, n.config version:0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection) [WorkerReceiver[myid=1]]
2024-11-04 14:28:43,994 INFO Processing ruok command from /127.0.0.1:59338 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2024-11-04 14:28:43,994 INFO Processing ruok command from /127.0.0.1:59348 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2024-11-04 14:28:54,000 INFO Processing ruok command from /127.0.0.1:46764 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2024-11-04 14:28:54,047 INFO Processing ruok command from /127.0.0.1:46778 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2024-11-04 14:29:03,993 INFO Processing ruok command from /127.0.0.1:57818 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2024-11-04 14:29:03,995 INFO Processing ruok command from /127.0.0.1:57822 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2024-11-04 14:29:13,996 INFO Processing ruok command from /127.0.0.1:42900 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2024-11-04 14:29:14,055 INFO Processing ruok command from /127.0.0.1:42908 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2024-11-04 14:29:23,989 INFO Processing ruok command from /127.0.0.1:52304 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2024-11-04 14:29:23,994 INFO Processing ruok command from /127.0.0.1:52308 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2024-11-04 14:29:33,990 INFO Processing ruok command from /127.0.0.1:45442 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2024-11-04 14:29:33,992 INFO Processing ruok command from /127.0.0.1:45444 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2024-11-04 14:29:43,994 INFO Processing ruok command from /127.0.0.1:46908 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2024-11-04 14:29:44,046 INFO Processing ruok command from /127.0.0.1:46920 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]

Has any of you faced any similar issues?

scholzj · 2024-11-05T03:49:09Z

scholzj
Nov 5, 2024
Maintainer

UnknownHostException suggests some DNS issues in your cluster.

1 reply

brunooliveiramac Nov 5, 2024
Author

yes, that was some intermittence

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

Kafka not creating service brokers #10797

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Strimzi

Kafka not creating service brokers #10797

Uh oh!

Uh oh!

brunooliveiramac Nov 4, 2024

Replies: 1 comment · 1 reply

Uh oh!

scholzj Nov 5, 2024 Maintainer

Uh oh!

brunooliveiramac Nov 5, 2024 Author

brunooliveiramac
Nov 4, 2024

Replies: 1 comment 1 reply

scholzj
Nov 5, 2024
Maintainer

brunooliveiramac Nov 5, 2024
Author