Topic Operator and User Operator Fail with 1 replica but work with 3 replicas

[eshwaribhide]

Description/How to Reproduce

I started up a Kafka cluster with 3 replicas each of Kafka and Zookeeper. Everything worked fine. I then deleted this cluster and then recreated it with 1 replica each of Kafka and Zookeeper. I then got errors in Topic Operator. I kept trying to restart the cluster but kept getting the same errors. Then I recreated the 3-replica cluster, and everything worked. Tried 1-replica again right after, it failed. Recreated 3-replica cluster, it worked. During the whole period of trying this out, I uninstalled and reinstalled Strimzi twice (according to Quickstart) but this did not change anything for the 1-replica cluster. After the second reinstall, I tried the 1-replica cluster and proceeded to get both the Topic Operator error and an error in User Operator. (I have already seen 1030 and 1050).

I also deleted all kafka-log* directories to clear things but it did not help. I am wondering if there is anywhere else I should look in case there is anything stale I have to delete, if that might be the issue?

Errors
Topic Operator:
2021-07-16 21:58:56,33405 WARN [vertx-blocked-thread-checker] BlockedThreadChecker: - Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 5600 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at jdk.internal.misc.Unsafe.park(Native Method) ~[?:?] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) ~[?:?] at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796) ~[?:?] at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) ~[?:?] at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823) ~[?:?] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998) ~[?:?] at io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:35) ~[io.apicurio.apicurio-registry-common-1.3.0.Final.jar:?] at io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27) ~[io.apicurio.apicurio-registry-common-1.3.0.Final.jar:?] at io.apicurio.registry.utils.ConcurrentUtil.result(ConcurrentUtil.java:54) ~[io.apicurio.apicurio-registry-common-1.3.0.Final.jar:?] at io.strimzi.operator.topic.Session.lambda$start$9(Session.java:202) ~[io.strimzi.topic-operator-0.24.0.jar:0.24.0] at io.strimzi.operator.topic.Session$$Lambda$232/0x000000084025c440.handle(Unknown Source) ~[?:?] at io.vertx.core.impl.future.FutureImpl$3.onSuccess(FutureImpl.java:124) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0] at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54) ~[io.vertx.vertx-core-4.1.0.jar:4.1.0] at io.vertx.core.impl.future.FutureBase$$Lambda$254/0x00000008402bcc40.run(Unknown Source) ~[?:?] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[io.netty.netty-common-4.1.65.Final.jar:4.1.65.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[io.netty.netty-common-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) ~[io.netty.netty-transport-4.1.65.Final.jar:4.1.65.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[io.netty.netty-common-4.1.65.Final.jar:4.1.65.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.65.Final.jar:4.1.65.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.65.Final.jar:4.1.65.Final] at java.lang.Thread.run(Thread.java:829) ~[?:?]

User Operator:
2021-07-16 23:14:10 ERROR Main:68 - Unable to start operator io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get] for kind: [Secret] with name: [kafka-cluster-cluster-ca-cert] in namespace: [kafka-cluster-deployment] failed. Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[?:?] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[?:?] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[?:?] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:?] at java.net.Socket.connect(Socket.java:609) ~[?:?]

Environment:

• Strimzi version: 0.24.0
• Installation method: Quickstart
• Kubernetes cluster: 1.21.2
• Infrastructure: Bare metal

[scholzj]

It is hard to say from some short log snippets. The User Operator error means that it cannot connect to the Kubernetes API server. Not sure about the Topic Operator - that message on its own doesn't mean much.

[eshwaribhide]

Ultimately, I found that it was some error with the node that the Topic Operator and User Operator pods were running on. If I run everything Strimzi-related on different nodes, then I do not have issues. Actually, I had noticed that no Kafka or Zookeeper pods could run on this node, so in my 3-replica yaml file, I had constrained Kafka and Zookeeper pods, along with Topic Operator and User Operator pods, to run on a different node. In my 1-replica yaml file, I was missing this constraint for Topic Operator and User Operator pods.

[eshwaribhide]

Actually, though I know that node is defective, upon restarting the cluster, and having the current node configurations (for all Strimzi related things to not run on the "defective" nodes, essentially changing no configuration) I am actually getting that error again. These are the full logs.

Could it be that there is some stale configuration somewhere? I have been deleting all kafka-log* directories. Or is this the result of deploying too soon after taking the cluster down? I did a restart within minutes.

kafka-logs.txt

[scholzj]

Well, that seems to be only the log from the Topic Operator which I do not know very well. But IMHO it cannot connect to Kafka. I'm not sure what exactly have you deleted in Kafka, so it is hard to understand if you could have broken something. The right approach would be to delete the cluster + all KAfka and Zoo PVCs + possibly all KafkaTopic custom resources (to avoid topics being recreated with the old settings).

[eshwaribhide]

Along with everything in kafka-log*, I have tried deleting the cluster and PVCs and even redeployed the CRDs and RBAC resources completely and kept running into that error.
As a workaround for now, I tried just running on yet another node (that I know is not defective) for the timebeing and the issue went away.

Also, when trying to redeploy CRDs and RBAC resources earlier today, I was getting errors regarding connecting to the Kubernetes API server in the cluster-operator pod, similar to those I was getting in the User Operator. None of my app pods (for the app I am using Kubernetes to manage) had this error. Ultimately, I ended up completely restarting my Kubernetes cluster itself and those errors went away. I am not sure if the cluster operator was running on that "defective" node, or if something else was happening (manually restarting api server, kubelet, dns didn't work).