TimeoutException describeConfigs

[DmytroDrachov]

Hey there, we do have timeout issue, it's probably related to count of topic, but I have not found any config to increase timeout.

2022-04-20 19:16:35 INFO  KafkaRoller:507 - Reconciliation #2(watch) Kafka(kafka/kafka-cluster): Pod 0 needs to be restarted. Reason: [Pod has old generation]
2022-04-20 19:16:35 DEBUG KafkaRoller:700 - Reconciliation #2(watch) Kafka(kafka/kafka-cluster): Creating AdminClient for kafka-cluster-kafka-0.kafka-cluster-kafka-brokers.kafka.svc.cluster.local:9091
2022-04-20 19:16:35 DEBUG KafkaRoller:753 - Reconciliation #2(watch) Kafka(kafka/kafka-cluster): Controller is 2
2022-04-20 19:16:35 DEBUG KafkaAvailability:62 - Reconciliation #2(watch) Kafka(kafka/kafka-cluster): Determining whether broker 0 can be rolled
2022-04-20 19:16:35 DEBUG KafkaAvailability:216 - Reconciliation #2(watch) Kafka(kafka/kafka-cluster): Got 8789 topic names
2022-04-20 19:16:35 DEBUG KafkaAvailability:51 - Reconciliation #2(watch) Kafka(kafka/kafka-cluster): Got 8789 topic names
2022-04-20 19:16:37 DEBUG KafkaAvailability:202 - Reconciliation #2(watch) Kafka(kafka/kafka-cluster): Got topic descriptions for 8789 topics
2022-04-20 19:16:37 DEBUG KafkaAvailability:69 - Reconciliation #2(watch) Kafka(kafka/kafka-cluster): Got 8789 topic descriptions
2022-04-20 19:16:37 DEBUG KafkaAvailability:161 - Reconciliation #2(watch) Kafka(kafka/kafka-cluster): Getting topic configs for 8774 topics

2022-04-20 19:17:17 WARN  KafkaAvailability:90 - Reconciliation #2(watch) Kafka(kafka/kafka-cluster): Error determining whether it is safe to restart pod 0
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConfigs, deadlineMs=1650482237891, tries=2, nextAllowedTryMs=1650482238011) timed out at 1650482237911 after 2 attempt(s)
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
	at org.apache.kafka.common.internals.KafkaCompletableFuture.kafkaCompleteExceptionally(KafkaCompletableFuture.java:49) ~[org.apache.kafka.kafka-clients-3.1.0.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:130) ~[org.apache.kafka.kafka-clients-3.1.0.jar:?]
	at org.apache.kafka.common.KafkaFuture.lambda$allOf$2(KafkaFuture.java:93) ~[org.apache.kafka.kafka-clients-3.1.0.jar:?]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
	at org.apache.kafka.common.internals.KafkaCompletableFuture.kafkaCompleteExceptionally(KafkaCompletableFuture.java:49) ~[org.apache.kafka.kafka-clients-3.1.0.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:130) ~[org.apache.kafka.kafka-clients-3.1.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.lambda$completeAllExceptionally$1(KafkaAdminClient.java:406) ~[org.apache.kafka.kafka-clients-3.1.0.jar:?]
	at java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1693) [?:?]
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) [?:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.completeAllExceptionally(KafkaAdminClient.java:406) [org.apache.kafka.kafka-clients-3.1.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.completeAllExceptionally(KafkaAdminClient.java:395) [org.apache.kafka.kafka-clients-3.1.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.access$3100(KafkaAdminClient.java:289) [org.apache.kafka.kafka-clients-3.1.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$11.handleFailure(KafkaAdminClient.java:2399) [org.apache.kafka.kafka-clients-3.1.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$Call.handleTimeoutFailure(KafkaAdminClient.java:840) [org.apache.kafka.kafka-clients-3.1.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:804) [org.apache.kafka.kafka-clients-3.1.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.handleResponses(KafkaAdminClient.java:1249) [org.apache.kafka.kafka-clients-3.1.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1408) [org.apache.kafka.kafka-clients-3.1.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1331) [org.apache.kafka.kafka-clients-3.1.0.jar:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConfigs, deadlineMs=1650482237891, tries=2, nextAllowedTryMs=1650482238011) timed out at 1650482237911 after 2 attempt(s)
Caused by: org.apache.kafka.common.errors.DisconnectException: Cancelled describeConfigs request with correlation id 9 due to node 2 being disconnected
2022-04-20 19:17:17 INFO  KafkaRoller:297 - Reconciliation #2(watch) Kafka(kafka/kafka-cluster): Could not roll pod 0 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: An error while trying to determine rollability, retrying after at least 250ms

[scholzj]

@tombentley Any ideas?

[DmytroDrachov]

@scholzj @tombentley Probably that could be related.

Props hardcoded:

strimzi-kafka-operator/operator-common/src/main/java/io/strimzi/operator/common/DefaultAdminClientProvider.java

Line 43 in 434ae04

public Admin createAdminClient(String bootstrapHostnames, Secret clusterCaCertSecret, Secret keyCertSecret, String keyCertName) {

Describe all topics in one request:

strimzi-kafka-operator/cluster-operator/src/main/java/io/strimzi/operator/cluster/operator/resource/KafkaAvailability.java

Line 197 in bf4fa3f

ac.describeTopics(names).allTopicNames()