Can't list kafkatopic

[wyangsun]

Please use this to only for bug reports. For questions or when you need help, you can use the GitHub Discussions, our #strimzi Slack channel or out user mailing list.

Describe the bug
I have two Kafka clusters in same namespace. I got nothing when I list kafkatopic. I can get the error logs in both topic-operator.
Seems the both topic-operator always create consumer-offsets---84e7a678d08f4bd226872e5cdd4eb527fadc1c6a

To Reproduce
The error keeps coming

Expected behavior
Is there any way to fix this?

Environment (please complete the following information):

• Strimzi version: 0.27.0
• Installation method: Helm chart
• Kubernetes cluster: 1.21
• Infrastructure: Amazon EKS

YAML files and logs

2022-02-16 05:56:19,60037 ERROR [vert.x-eventloop-thread-1] TopicOperator:1490 - Reconciliation #11038(periodic -__consumer_offsets) KafkaTopic(ops/consumer-offsets---84e7a678d08f4bd226872e5cdd4eb527fadc1c6a): Error reconciling KafkaTopic null io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.20.0.1/apis/kafka.strimzi.io/v1beta2/namespaces/ops/kafkatopics. Message: kafkatopics.kafka.strimzi.io "consumer-offsets---84e7a678d08f4bd226872e5cdd4eb527fadc1c6a" already exists. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=kafka.strimzi.io, kind=kafkatopics, name=consumer-offsets---84e7a678d08f4bd226872e5cdd4eb527fadc1c6a, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=kafkatopics.kafka.strimzi.io "consumer-offsets---84e7a678d08f4bd226872e5cdd4eb527fadc1c6a" already exists, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=AlreadyExists, status=Failure, additionalProperties={}). at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:697) ~[io.fabric8.kubernetes-client-5.10.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:676) ~[io.fabric8.kubernetes-client-5.10.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:629) ~[io.fabric8.kubernetes-client-5.10.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:566) ~[io.fabric8.kubernetes-client-5.10.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:527) ~[io.fabric8.kubernetes-client-5.10.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:315) ~[io.fabric8.kubernetes-client-5.10.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:651) ~[io.fabric8.kubernetes-client-5.10.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:91) ~[io.fabric8.kubernetes-client-5.10.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61) ~[io.fabric8.kubernetes-client-5.10.2.jar:?] at io.strimzi.operator.topic.K8sImpl.lambda$createResource$0(K8sImpl.java:51) ~[io.strimzi.topic-operator-0.27.1.jar:0.27.1] at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1] at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1] at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1] at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final] at java.lang.Thread.run(Thread.java:829) [?:?] 2022-02-16 05:56:19,61419 INFO [vert.x-eventloop-thread-1] TopicOperator:576 - Reconciliation #11043(periodic -strimzi.cruisecontrol.metrics) KafkaTopic(ops/strimzi.cruisecontrol.metrics): Reconciling topic strimzi.cruisecontrol.metrics, k8sTopic:null, kafkaTopic:nonnull, privateTopic:null 2022-02-16 05:56:19,62248 ERROR [vert.x-eventloop-thread-1] TopicOperator:1490 - Reconciliation #11048(periodic -strimzi.cruisecontrol.metrics) KafkaTopic(ops/strimzi.cruisecontrol.metrics): Error reconciling KafkaTopic null io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.20.0.1/apis/kafka.strimzi.io/v1beta2/namespaces/ops/kafkatopics. Message: kafkatopics.kafka.strimzi.io "strimzi.cruisecontrol.metrics" already exists. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=kafka.strimzi.io, kind=kafkatopics, name=strimzi.cruisecontrol.metrics, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=kafkatopics.kafka.strimzi.io "strimzi.cruisecontrol.metrics" already exists, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=AlreadyExists, status=Failure, additionalProperties={}). at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:697) ~[io.fabric8.kubernetes-client-5.10.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:676) ~[io.fabric8.kubernetes-client-5.10.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:629) ~[io.fabric8.kubernetes-client-5.10.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:566) ~[io.fabric8.kubernetes-client-5.10.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:527) ~[io.fabric8.kubernetes-client-5.10.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:315) ~[io.fabric8.kubernetes-client-5.10.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:651) ~[io.fabric8.kubernetes-client-5.10.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:91) ~[io.fabric8.kubernetes-client-5.10.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61) ~[io.fabric8.kubernetes-client-5.10.2.jar:?] at io.strimzi.operator.topic.K8sImpl.lambda$createResource$0(K8sImpl.java:51) ~[io.strimzi.topic-operator-0.27.1.jar:0.27.1] at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1] at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1] at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1] at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final] at java.lang.Thread.run(Thread.java:829) [?:?]

To easily collect all YAMLs and logs, you can use our report script which will automatically collect all files and prepare a ZIP archive which can be easily attached to this issue.
The usage of this script is:
./report.sh [--namespace <string>] [--cluster <string>]

Additional context
Add any other context about the problem here.

[scholzj]

If you want to have multiple Kafka clusters in the same namespace, only one of them can have the Topic Operator enabled and it should be disabled in the others. Or you should use different namespaces for each cluster.

[wyangsun]

Thank you @scholzj
My bad, I found the root cause is in kafkatopic CRD. There is other operator update the kafkatopic CRD, I just recreate it, and it works.