KakfkaUserOperator crash. Timeout waiting for a node assignment

[MicahDevOps]

Operator version: 0.40.0

Hi,
I installed strimzi topic operator and user operator on my K8s cluster. I am using these 2 operators to manage my AWS MSK topics and ACLs. TopicOperator is working well and it indeed helps me manage my topics. But the user operator keeps crashing with those log:

2024-04-29 09:44:35 INFO Server:415 - Started @8646ms
2024-04-29 09:44:35 INFO AbstractCache:117 - Starting QuotasCache
2024-04-29 09:44:37 INFO NetworkClient:997 - [AdminClient clientId=adminclient-1] Node -1 disconnected.
2024-04-29 09:44:37 INFO NetworkClient:353 - [AdminClient clientId=adminclient-1] Cancelled in-flight API_VERSIONS request with correlation id 0 due to node -1 being disconnected (elapsed time since creation: 3009ms, elapsed time since send: 3009ms, request timeout: 3600000ms)
2024-04-29 09:44:37 INFO NetworkClient:997 - [AdminClient clientId=adminclient-1] Node -2 disconnected.
2024-04-29 09:44:37 INFO NetworkClient:353 - [AdminClient clientId=adminclient-1] Cancelled in-flight API_VERSIONS request with correlation id 1 due to node -2 being disconnected (elapsed time since creation: 203ms, elapsed time since send: 203ms, request timeout: 3600000ms)
2024-04-29 09:44:37 INFO NetworkClient:997 - [AdminClient clientId=adminclient-1] Node -3 disconnected.
2024-04-29 09:44:37 INFO NetworkClient:353 - [AdminClient clientId=adminclient-1] Cancelled in-flight API_VERSIONS request with correlation id 2 due to node -3 being disconnected (elapsed time since creation: 211ms, elapsed time since send: 211ms, request timeout: 3600000ms)
2024-04-29 09:44:44 INFO AdminMetadataManager:267 - [AdminClient clientId=adminclient-1] Metadata update failed
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
2024-04-29 09:45:16 WARN QuotasCache:65 - Failed to load Quotas
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeClientQuotas
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096) ~[?:?]
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180) ~[org.apache.kafka.kafka-clients-3.7.0.jar:?]
at io.strimzi.operator.user.operator.cache.QuotasCache.loadCache(QuotasCache.java:51) ~[io.strimzi.user-operator-0.40.0.jar:0.40.0]
at io.strimzi.operator.user.operator.cache.AbstractCache.updateCache(AbstractCache.java:146) ~[io.strimzi.user-operator-0.40.0.jar:0.40.0]
at io.strimzi.operator.user.operator.cache.AbstractCache.initialize(AbstractCache.java:128) ~[io.strimzi.user-operator-0.40.0.jar:0.40.0]
at io.strimzi.operator.user.operator.cache.AbstractCache.start(AbstractCache.java:119) ~[io.strimzi.user-operator-0.40.0.jar:0.40.0]
at io.strimzi.operator.user.operator.QuotasOperator.start(QuotasOperator.java:96) ~[io.strimzi.user-operator-0.40.0.jar:0.40.0]
at io.strimzi.operator.user.operator.KafkaUserOperator.start(KafkaUserOperator.java:95) ~[io.strimzi.user-operator-0.40.0.jar:0.40.0]
at io.strimzi.operator.user.Main.main(Main.java:109) ~[io.strimzi.user-operator-0.40.0.jar:0.40.0]
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeClientQuotas
2024-04-29 09:45:16 ERROR AbstractCache:149 - QuotasCache failed to update
java.lang.RuntimeException: Failed to load Quotas
at io.strimzi.operator.user.operator.cache.QuotasCache.loadCache(QuotasCache.java:66) ~[io.strimzi.user-operator-0.40.0.jar:0.40.0]
at io.strimzi.operator.user.operator.cache.AbstractCache.updateCache(AbstractCache.java:146) ~[io.strimzi.user-operator-0.40.0.jar:0.40.0]
at io.strimzi.operator.user.operator.cache.AbstractCache.initialize(AbstractCache.java:128) ~[io.strimzi.user-operator-0.40.0.jar:0.40.0]
at io.strimzi.operator.user.operator.cache.AbstractCache.start(AbstractCache.java:119) ~[io.strimzi.user-operator-0.40.0.jar:0.40.0]
at io.strimzi.operator.user.operator.QuotasOperator.start(QuotasOperator.java:96) ~[io.strimzi.user-operator-0.40.0.jar:0.40.0]
at io.strimzi.operator.user.operator.KafkaUserOperator.start(KafkaUserOperator.java:95) ~[io.strimzi.user-operator-0.40.0.jar:0.40.0]
at io.strimzi.operator.user.Main.main(Main.java:109) ~[io.strimzi.user-operator-0.40.0.jar:0.40.0]
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeClientQuotas
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096) ~[?:?]
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180) ~[org.apache.kafka.kafka-clients-3.7.0.jar:?]
at io.strimzi.operator.user.operator.cache.QuotasCache.loadCache(QuotasCache.java:51) ~[io.strimzi.user-operator-0.40.0.jar:0.40.0]
... 6 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeClientQuotas

---

My UserOperator env:
env:
- name: STRIMZI_RESOURCE_LABELS
value: placeholder
- name: STRIMZI_KAFKA_BOOTSTRAP_SERVERS
value: placeholder
- name: STRIMZI_FULL_RECONCILIATION_INTERVAL_MS
value: "120000"
- name: STRIMZI_LOG_LEVEL
value: INFO
- name: STRIMZI_TLS_ENABLED
value: "true"
- name: STRIMZI_PUBLIC_CA
value: "true"
- name: STRIMZI_TLS_AUTH_ENABLED
value: "false"
- name: STRIMZI_SASL_ENABLED
value: "true"
- name: STRIMZI_SECURITY_PROTOCOL
value: SASL_SSL
- name: STRIMZI_SASL_MECHANISM
value: "scram-sha-512"
- name: STRIMZI_SASL_USERNAME
value: placeholder
- name: STRIMZI_SASL_PASSWORD
value: placeholder
- name: STRIMZI_CA_CERT_NAME
value: dummy-ca
- name: STRIMZI_CA_KEY_NAME
value: dummy-key
- name: STRIMZI_ACLS_ADMIN_API_SUPPORTED
value: "true"
- name: STRIMZI_GC_LOG_ENABLED
value: "false"
- name: STRIMZI_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: JAVA_OPTS
value: "-Xmx800m"

I have no ideas about the error messages:

• Timed out waiting to send the call. Call: fetchMetadata
• Failed to load Quotas
• Timed out waiting for a node assignment. Call: describeClientQuotas

I don't have 'Quotas' settings. Can any one suggest this crash ?
Many thanks

[scholzj]

I guess your Kafka cluster does not work or you failed to configure the user operator properly. I suspect the second might be the case as it seems that you just made up the configuration options? I think you have to use the STRIMZI_KAFKA_ADMIN_CLIENT_CONFIGURATION option to pass any custom options to configure the Admin client.