Nodeport type listeners do not observe the broker id/Host ordering

[sebady]

We have recently noticed that the broker IDs that are specified in the listeners of nodeport type do not match. Not sure if this expected behavior with nodeport type or possibly a bug? This has been seen with Strimzi version 0.30/Kafka3.1.1 (as well as 0.19.0)

For e.g. if we have 3 listeners configured like so

  listeners:
    - name: external
      port: 9094
      type: nodeport
      tls: false
      configuration:
        bootstrap:
          nodePort: 30700
        brokers:
           - advertisedHost: kafka-broker-1.ap-northeast-1
            broker: 0
            nodePort: 31110
          - advertisedHost: kafka-broker-2.ap-northeast-1
            broker: 1
            nodePort: 30111
          - advertisedHost: kafka-broker-3.ap-northeast-1
            broker: 2
            nodePort: 30101

We have noticed broker 0 was scheduled on host kafka broker 3.

We expected the above brokers list to be observed. Note that we have many Kafka clusters all on the same k8s cluster and clients appear to be connecting sometimes to "wrong" brokers when using just using nodeport IPs/Ports.

[scholzj]

I'm afraid your expectation is wrong in this case. This has nothing to do with the scheduling of the pods. This simply overrides the hostnames the broker advertises to the clients - so all it affects is the broker configuration and nothing more.

What exactly are you trying to achieve? If you want to make sure the brokers are advertising the node they run on, just remove the advertisedHost fields from your config and the operator will automatically make sure it advertises the node address.

[sebady]

We are trying to ensure that the specific brokers are scheduled to specific host just so that external clients have a reliable bootstrap. In this example kafka-broker-1.ap-northeast-1:31110,kafka-broker-2.ap-northeast-1:30111,kafka-broker-3.ap-northeast-1:30101
We clearly didn't understand advertisedHost's role exactly in defining the listener configuration. Thanks for clarifying

[scholzj]

Well, the bootstrap address is separate from the advertised addresses OR pod scheduling. You can use pretty much anything that routes to the brokers. For example with node ports, they normally listen on all worker nodes in your Kube cluster and Kube will route the traffic to the pods. So you can just create some DNS record routing to all your worker nodes and use that as the bootstrap configuration. The bootstrap connection is used only for routing the metadata to bootstrap the client, so it is not something that needs to point directly to the brokers - so you should leverage this node port feature.

If you want to control the scheduling, you can configure it through affinity or for example through node-specific storage classes. But I would normally argue against trying to pin the brokers to a particular worker node when it is not needed for anything as that will cause problems when the node is gone.