kafka external access loadbalancer tls | ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.

[tirelibirefe]

Hello,
I've installed Kafka, KafkaConnect and JDBC connector as kafkaconnector.
I configured TLS authentication & authorization as described in documentation.
I configured AWS NLB as described in @scholzj 's blog posts.

...
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
      - name: external
        port: 9094
        type: loadbalancer
        tls: true
        authentication:
          type: tls
        configuration:
          bootstrap:
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kuap-kafka-external-bootstrap.s1.mydomain.com.
              service.beta.kubernetes.io/aws-load-balancer-name: "kuap-kafka-external-bootstrap"
              service.beta.kubernetes.io/aws-load-balancer-type: external
              service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
              external-dns.alpha.kubernetes.io/ttl: "60"
            alternativeNames: 
            - kuap-kafka-external-bootstrap.s1.mydomain.com
          brokers:
          - broker: 0
            advertisedHost: kafka-broker-0.s1.mydomain.com
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-broker-0.s1.mydomain.com.
              service.beta.kubernetes.io/aws-load-balancer-name: "kafka-broker-0"
              service.beta.kubernetes.io/aws-load-balancer-type: external
              service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
              external-dns.alpha.kubernetes.io/ttl: "60"
          - broker: 1
            advertisedHost: kafka-broker-1.s1.mydomain.com
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-broker-1.s1.mydomain.com.
              service.beta.kubernetes.io/aws-load-balancer-name: "kafka-broker-1"
              service.beta.kubernetes.io/aws-load-balancer-type: external
              service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
              external-dns.alpha.kubernetes.io/ttl: "60"
          - broker: 2
            advertisedHost: kafka-broker-2.s1.mydomain.com
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-broker-2.s1.mydomain.com.
              service.beta.kubernetes.io/aws-load-balancer-name: "kafka-broker-2"
              service.beta.kubernetes.io/aws-load-balancer-type: external
              service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
              external-dns.alpha.kubernetes.io/ttl: "60"
...

Things work perfectly at port 9093.

...but things arent't working at port 9094 properly. External access is not stable.

In order to observe the situation I prepared a client pod as described in documentation, copied there certs and trying connectivity:

[kafka@strimzi-kafka-client kafka]$ for i in {1..10}; do bin/kafka-topics.sh --bootstrap-server kuap-kafka-external-bootstrap.s1.mycompany.com:9094 --command-config /tmp/config.properties --list; done
__consumer_offsets
__strimzi-topic-operator-kstreams-topic-store-changelog
__strimzi_store_topic
kuap-kafka-connect-configs
kuap-kafka-connect-offsets
kuap-kafka-connect-status
kukugateentrancedata
kupersonahealthdata
kupersonahealthdata-dlq
strimzi.cruisecontrol.metrics
strimzi.cruisecontrol.modeltrainingsamples
strimzi.cruisecontrol.partitionmetricsamples
__consumer_offsets
__strimzi-topic-operator-kstreams-topic-store-changelog
__strimzi_store_topic
kuap-kafka-connect-configs
kuap-kafka-connect-offsets
kuap-kafka-connect-status
kukugateentrancedata
kupersonahealthdata
kupersonahealthdata-dlq
strimzi.cruisecontrol.metrics
strimzi.cruisecontrol.modeltrainingsamples
strimzi.cruisecontrol.partitionmetricsamples
__consumer_offsets
__strimzi-topic-operator-kstreams-topic-store-changelog
__strimzi_store_topic
kuap-kafka-connect-configs
kuap-kafka-connect-offsets
kuap-kafka-connect-status
kukugateentrancedata
kupersonahealthdata
kupersonahealthdata-dlq
strimzi.cruisecontrol.metrics
strimzi.cruisecontrol.modeltrainingsamples
strimzi.cruisecontrol.partitionmetricsamples
__consumer_offsets
__strimzi-topic-operator-kstreams-topic-store-changelog
__strimzi_store_topic
kuap-kafka-connect-configs
kuap-kafka-connect-offsets
kuap-kafka-connect-status
kukugateentrancedata
kupersonahealthdata
kupersonahealthdata-dlq
strimzi.cruisecontrol.metrics
strimzi.cruisecontrol.modeltrainingsamples
strimzi.cruisecontrol.partitionmetricsamples
__consumer_offsets
__strimzi-topic-operator-kstreams-topic-store-changelog
__strimzi_store_topic
kuap-kafka-connect-configs
kuap-kafka-connect-offsets
kuap-kafka-connect-status
kukugateentrancedata
kupersonahealthdata
kupersonahealthdata-dlq
strimzi.cruisecontrol.metrics
strimzi.cruisecontrol.modeltrainingsamples
strimzi.cruisecontrol.partitionmetricsamples
Error while executing topic command : Timed out waiting for a node assignment. Call: listTopics
[2022-01-19 15:17:57,325] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
 (kafka.admin.TopicCommand$)
__consumer_offsets
__strimzi-topic-operator-kstreams-topic-store-changelog
__strimzi_store_topic
kuap-kafka-connect-configs
kuap-kafka-connect-offsets
kuap-kafka-connect-status
kukugateentrancedata
kupersonahealthdata
kupersonahealthdata-dlq
strimzi.cruisecontrol.metrics
strimzi.cruisecontrol.modeltrainingsamples
strimzi.cruisecontrol.partitionmetricsamples
__consumer_offsets
__strimzi-topic-operator-kstreams-topic-store-changelog
__strimzi_store_topic
kuap-kafka-connect-configs
kuap-kafka-connect-offsets
kuap-kafka-connect-status
kukugateentrancedata
kupersonahealthdata
kupersonahealthdata-dlq
strimzi.cruisecontrol.metrics
strimzi.cruisecontrol.modeltrainingsamples
strimzi.cruisecontrol.partitionmetricsamples
Error while executing topic command : Timed out waiting for a node assignment. Call: listTopics
[2022-01-19 15:19:28,194] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
 (kafka.admin.TopicCommand$)
command terminated with exit code 137

As you can see above, external access is not stable.

...but if I use 9093 port (internal access) , there is no problem, things work smoothly as expected.

you can find kafka, kafkaconnect configs and also their logs as attached.

It looks like I do something wrongly or miss something. Could you please help me figure out what I miss?

Thanks & Regards

[scholzj]

TBH, it is a long time since the blog post, but I do not remember any issues. From how you describe it, it sounds liek some issue with the connectivity between the client, NLB and Kubernetes. So I guess you should check there => whether the LB and the nodes are correct etc.