How to expose the external access with multiple kafka pods

[dumplingdumpling]

Hi, I'm a strimzi noob here. I'm trying to establish a cluster with 3 kafka replicas with external accessible service. I used the config from https://youtu.be/1qO2qGuJNQI and tweaked it a little bit according to the blog https://strimzi.io/blog/2019/04/23/accessing-kafka-part-2/.

However, when I tried to apply the configuration, only 3 zookeeper pods were initialized. The kafka pod doesn't show up by typing "kubectl get pods -n kafka". I don't know if I tweaked the config in the right way. I really appreciate If someone could provide some guidance about this.

Here is the context:

The environment is on a ubuntu PC with a docker desktop and k8s. I used kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka to deploy the strimzi operator. The config is:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 3.6.1
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9094
        type: nodeport
        tls: false
        configuration:
          # bootstrap:
          #   nodePort: 32100
          brokers:
          - broker: 0
            advertisedHost: localhost
            advertisedPort: 32000
          - broker: 1
            advertisedHost: localhost
            advertisedPort: 32001
          - broker: 2
            advertisedHost: localhost
            advertisedPort: 32002
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.6"
      message.max.bytes: 100000000
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 50Gi
        deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 50Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

[scholzj]

For a start, you have to format the YAML to make it readable. You should also check the logs to see what they say, explain what exactly does not work etc.

[dumplingdumpling]

just like this:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 3.6.1
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9094
        type: nodeport
        tls: false
        configuration:
          # bootstrap:
          #   nodePort: 32100
          brokers:
          - broker: 0
            advertisedHost: localhost
            advertisedPort: 31770
          - broker: 1
            advertisedHost: localhost
            advertisedPort: 31499
          - broker: 2
            advertisedHost: localhost
            advertisedPort: 30815
    config:
      zookeeper.connect: my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.6"
      message.max.bytes: 100000000
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 50Gi
        deleteClaim: false
  zookeeper:
    replicas: 1
    storage:
      type: persistent-claim
      size: 50Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

[scholzj]

That does not do anything. You cannot configure that option. It also doesn't do anything about your DNS.

[dumplingdumpling]

Yeah, but with this configuration, Kafka pods can be initialized. Still, I don't know why this configuration could solve the DNS problem.

[scholzj]

That is just a coincidence. Likely your DNS started to work.

[dumplingdumpling]

Yeah, maybe, but thank you for your precious time for answering these questions.