[resolved] 0.36.0/3.5.0 Zookeeper can't find other zookeepers

[maartengo]

On a fresh deployment of the strimzi-kafka-operator and kafka the zookeerper pods log the following.

ERROR StaticHostProvider:148 - Unable to resolve address: dev-dev2-zookeeper-1.dev-dev2-zookeeper-nodes.kafka.svc/:2181

This seems to block the rest of the kafka deployment, only zookeeper pods were created.

We've deployed to the kafka namespace, and the service has the name dev-dev2-zookeeper-nodes.

Our deployment steps;
0. Make sure no existing kafka configuration is present, delete kafka namespace

1. Deploy version 0.36.0 of the operator chart to kafka namespace
2. Wait for deployment to complete
3. Deploy version 3.5.0 of kafka
4. Wait for deployment to complete
5. See that only zookeeper pods are created
6. Open zookeeper logs
7. See something similar to zookeeper-logs.txt

Operator Helm chart; 0.36.0
Kubernetes; 1.26

---

Do we need to supply different values/configuration for the deployment of 3.5.0? What could be going wrong here?

[scholzj]

This looks like it might be related to the new ZooKeeper version used by KAfka 3.5 which seems to be a lot more sensitive to the quality of your Kubernetes DNS.

CC @ppatierno

[maartengo]

Ah right, I deleted the environment too many times and now we can't request a new certificate 🙈
I'll deploy it on another environment and will report back with the results

[maartengo]

Just tested on a new environment; the quay.io/ppatierno/kafka:0.36.0-kafka-3.5.0-all-binding-gt-2 image does indeed work when running 2 replicas of ZK

[ppatierno]

@maartengo thanks for the feedback. The fix should be included in the next 0.37.0 which won't be too soon (we just released 0.36.0) so my advice for you is to run at least 3 ZK because this private image I provided you for testing will soon go away as well and it's anyway not "official".

[maartengo]

Is it possible to include the fix in a patch release? It would be a shame if we'd have to wait for the next release just so we can run a certain amount of replicas. I can also image that this issue occurs for more users of the operator, who would all have to find out what exactly is going wrong before knowing to change their ZK replica count.

[ppatierno]

Is it possible to include the fix in a patch release?

@scholzj we are planning a 0.36.1, do you think we can include this fix?

I can also image that this issue occurs for more users of the operator

Actually ZK with 2 nodes is a really bad a not so used configuration. We totally discourage it and you can also see a warning in the Kafka custom resource status about it :-). I am not sure how many users really use this.

[showuon]

I only showed the first few logs since after that it keeps repeating, but I'll try to post more logs in the future as well. The logs after ~9 minutes for both replicas: zookeeper-custom-image-rep1.txt zookeeper-custom-image-rep2.txt

For the record, I've opened a JIRA issue in Zookeeper's project. I believe there is a bug there, but need zookeeper's expert to identify it.