Quickstart Kafka setup fails

[LDTips]

Hello.

I am running a local microk8s instance. I was looking to use the quickstart example posted here https://strimzi.io/quickstarts/

I install the strimzi operator and the pod gets created - everything ok in logs.
But after I try to apply the configuration from https://strimzi.io/examples/latest/kafka/kafka-persistent-single.yaml with the only difference in reducing the sizes from 100Gi to 2Gi, I am unable to get a Kafka pod running. The zookeeper pod gets created, but no Kafka pod.

Interestingly, the default ephemeral configuration works.
To be sure I did not change something by accident, here is the applied yaml file:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 3.6.0
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.6"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 2Gi
        deleteClaim: false
  zookeeper:
    replicas: 1
    storage:
      type: persistent-claim
      size: 2Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

And attached are the logs from the zookeeper and the strimzi operator pods.

strimzi-cluster-op_new.log
zookeeper_new.log

[ppatierno]

Latest ZooKeeper version brought by last Kafka version is more sensible to DNS errors, so if your environment has a DNS not ready at right time, this is what happens. Quite often it works eventually. I have never used microk8s so I don't know if it's something you can tune DNS somehow.

[LDTips]

Do you know what options to tune this DNS I would need to use? The machine I am running on is using shared cores and can get slow at times, so I don't know - maybe that is the reason DNS sometimes isn't ready and this happens. But is there any way to avoid this?
Edit: I may also added that I tried this configuration several times and I did not manage to get this working once, so unfortunately there was not a single time I managed to get this working

[LDTips]

I did not find a granular fix, but increasing the number of zookeeper replicas (e.g. to 3) fixes this issue. At the start zookeepers restart and the correct DNS resolution for the service is obtained. I have no idea why zookeeper does only try to resolve the address when starting. This causes the issue that "DNS is not ready at right time" and zookeeper sends a DNS request to not-yet-ready coredns pod, resulting in failed address resolution. This is shown in the zookeeper logs with address being .

In case the logs in zookeeper look ok and no DNS issues are found there, it's most likely hostpath being wonky somewhere. In such case it's more of a cluster issue. Increate the allocations for pvcs or reduce data amounts created by pods in pvc (e.g. reduce kafka retention time)

[zekicaneksi]

Thank You @LDTips

I myself was following the quickstart step by step without changing anything and deploying to Minikube.

Zookeeper was failing to connect, when i checked the logs of the zookeper pod;

ERROR Couldn't bind to my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc/<unresolved>:2888 (org.apache.zookeeper.server.quorum.Leader) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)]
java.net.SocketException: Unresolved address

So, looking at your suggestion i increased the zookeper replicas to 3 and it worked fine. 2 replica amount doesn't work either btw.

I guess quickstart file needs to be updated since it is a Zookeper bug i guess.

[LDTips]

Yes, I forgot to mention that in zookeeper documentation there is a requirement for an odd number of replicas to be present. Indeed the persistent single example file here does have a single zookeeper replica. However, ephemeral example file here is correct and has 3 replicas defined