Topic Operator fails to start with VertxException: Thread blocked

[sksahu-244]

Hello,

When kafka replica is set to 1 or 2, Topic Operator fails to start with below VertxException: Thread blocked. The issue is not reproducible with 3 replicas of kafka.

Below is the exception stacktrace:

Thread Thread[blocking-startup-ops-0,5,main] has been blocked for 412528 ms, time limit is 60000 ms io.vertx.core.VertxException: Thread blocked|\tat [email protected]/jdk.internal.misc.Unsafe.park(Native Method)|\tat [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)|\tat [email protected]/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796)|\tat [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128)|\tat [email protected]/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823)|\tat [email protected]/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998)|\tat app//io.strimzi.operator.topic.Session.createTopicStore(Session.java:324)|\tat app//io.strimzi.operator.topic.Session$$Lambda$234/0x000000010038e840.apply(Unknown Source)|\tat app//io.strimzi.operator.topic.Session.lambda$createTopicStoreAsync$15(Session.java:254)|\tat app//io.strimzi.operator.topic.Session$$Lambda$448/0x000000010051f440.handle(Unknown Source)|\tat app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)|\tat app//io.vertx.core.impl.ContextImpl$$Lambda$314/0x0000000100469040.handle(Unknown Source)|\tat app//io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)|\tat app//io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)|\tat app//io.vertx.core.impl.ContextImpl$$Lambda$307/0x000000010044a840.run(Unknown Source)|\tat app//io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)|\tat app//io.vertx.core.impl.TaskQueue$$Lambda$238/0x000000010038d840.run(Unknown Source)|\tat [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)|\tat [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)|\tat app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)|\tat [email protected]/java.lang.Thread.run(Thread.java:829)

I checked an older discussion #6046 with same issue.
The workaround suggested in this discussion to use the environment variable USE_ZOOKEEPER_TOPIC_STORE and set it to true works for me fine. I also checked the PR #6645, it seems the fix was delivered in 0.29.0. I am currently using 0.31.0. I am still facing this issue. Could you please take a look and let me know if I am missing something here or is it a Bug?

Thanks,
Santosh

[scholzj]

You would probably need to share all the logs and not just one exception which tells nothing about the actual issue. You should not use USE_ZOOKEEPER_TOPIC_STORE. It is not really covered by any real tests.

[sksahu-244]

Hi, Would the below POD manifest work for you and would help you in reproducing this issue?

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: backup-restore-with-stanza
spec:
  kafka:
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
    storage:
      type: ephemeral
  zookeeper:   
    replicas: 1
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}

Thanks,
Santosh

[scholzj]

It's not about what works for me. I know how to get it working. The problem is that you say that you have some problem - but the resource you are supposed to have it with is clearly invalid and does not work. So clearly, trying to reproduce it with it would not really be possible. You should not share what would work for me but what exactly are you doing and how exactly does your resource look like.

[sksahu-244]

Hi, Sorry for the late response. I can confirm I am testing with below Kafka manifest and now I am testing with latest strimzi version 0.36.1 helm chart and I could reproduce the issue in latest strimzi version 0.36.1. Please find attached the topic-operator log with latest strimzi version 0.36.1.
latest_strimzi_topic_operator_log.txt

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: backup-restore-with-stanza
spec:
  kafka:
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
    storage:
      type: ephemeral
  zookeeper:
    replicas: 1
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}

When I changed the kafka replicas to 3 in the above Kafka manifest, the vertx exception is no more seen in the topic-operator log and topic-operator container becomes up and running.

Thanks,
Santosh

[scholzj]

This is because you have it misconfigured. You have one Kafka replica only, but your configuration use the Kafka defaults which is 3 replicas for every topic. You should follow the Strimzi examples which configure it properly: https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/kafka/kafka-ephemeral-single.yaml

[sksahu-244]

Thanks a lot for your guidance. I added the recommended kafka config parameters (from https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/kafka/kafka-ephemeral-single.yaml) in my POD manifest file and it is working fine now. topic-operator container is coming up and running with single replica of Kafka.