UNKNOWN_TOPIC_OR_PARTITION

[nareshm25099]

Hi Team,

We are seeing this error after e created the topic. The topic was created but we are seeing the below issue in our logstash.

Error while fetching metadata with correlation id 169961 : {Topic=UNKNOWN_TOPIC_OR_PARTITION}

Can you help us with this issue. We tried restarting the zookeeper and kafka brokers but no success.

Thanks,
Naresh.

[nareshm25099]

Can anyone suggest why is this happening....

And in the strimzi cluster operator pod logs we are seeing this ERROR. Is this related to the above issue?

2022-02-13 00:51:46 ERROR AbstractOperator:437 - Watcher closed with exception in namespace strimzi
io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 158879072 (176612724)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:258) [io.fabric8.kubernetes-client-4.12.0.jar:?]
at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
2022-02-13 00:51:46 INFO OperatorWatcher:40 - Reconciliation #29042(watch) Kafka(strimzi/uslogs): Kafka uslogs in namespace strimzi was ADDED
2022-02-13 00:51:48 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace strimzi...
2022-02-13 00:51:56 WARN AbstractOperator:377 - Reconciliation #29042(watch) Kafka(strimzi/uslogs): Failed to acquire lock lock::strimzi::Kafka::uslogs within 10000ms.
2022-02-13 00:51:58 WARN AbstractOperator:377 - Reconciliation #29043(timer) Kafka(strimzi/uslogs): Failed to acquire lock lock::strimzi::Kafka::uslogs within 10000ms.
2022-02-13 00:53:48 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace strimzi...
2022-02-13 00:53:58 WARN AbstractOperator:377 - Reconciliation #29044(timer) Kafka(strimzi/uslogs): Failed to acquire lock lock::strimzi::Kafka::uslogs within 10000ms.
2022-02-13 00:55:48 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace strimzi...
2022-02-13 00:55:58 WARN AbstractOperator:377 - Reconciliation #29045(timer) Kafka(strimzi/uslogs): Failed to acquire lock lock::strimzi::Kafka::uslogs within 10000ms.
2022-02-13 00:57:48 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace strimzi...
2022-02-13 00:57:58 WARN AbstractOperator:377 - Reconciliation #29046(timer) Kafka(strimzi/uslogs): Failed to acquire lock lock::strimzi::Kafka::uslogs within 10000ms.
2022-02-13 00:59:48 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace strimzi...
2022-02-13 00:59:58 WARN AbstractOperator:377 - Reconciliation #29047(timer) Kafka(strimzi/uslogs): Failed to acquire lock lock::strimzi::Kafka::uslogs within 10000ms.
2022-02-13 01:01:48 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace strimzi...

Thanks,
Naresh.

[scholzj]

The watch error is more or less nromal - it is related to how Kubernetes watches work. Nothing to be worried about.

[nareshm25099]

But we are facing this issue "Error while fetching metadata with correlation id 169961 : {Topic=UNKNOWN_TOPIC_OR_PARTITION}". Topic was created but we are seeing this issue in logstash logs.

[scholzj]

Well, I replied to the second error you reported. I don't know abotu the first one -> apart from the obvious issues like topic does nto exist or is not online.

[nareshm25099]

Yeah but the major issue was the first one. We tried to create topic the topic the topic was created but we are seeing the issue in logstash. Any idea we tried restarting kafka brokers and zookeeper as well.

[nareshm25099]

The issue is happening for us on prod cluster and the topic was created but getting that Warning and logstash not pickup the topic. We have checked strmizi-cluster operator logs we are only seeing the watcher closed exception. If you can give us some advice on how to debug this issue that would be great.