Zookeeper crashes with 3 replicas but not with 1

[ralphflat]

We configured the Strimzi operator to watch two namespaces - oper and devel. The Strimzi operation is installed into the kafka-operator namespace.

We apply the Kafka kind definition file (below) and see zookeeper start up, but no kafka instance.

more kafka-operational.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: operational
spec:
  kafka:
    version:
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.2"
    storage:
      type: ephemeral
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}

We only see zookeeper pods (no kafka pods) in the watched namespace (the 'oper' namespace)

When we look at the logs of one of the zookeeper services, we see the following. Specifically, the warning of not being able to open channel followed by exception (identified as operational-zookeeper-1.operational-zookeeper-nodes.oper.svc/10.42.0.34:3888). While we see operational-zookeeper-nodes is attached to port 3888, we do not see anything with the ClusterIP of 10.42.0.34. ClusterIP for the operational-zookeeper-nodes is set to None - is that correct?

When we change replicas on zookeeper (with the Kafka kind configuration) to 1, Kafka start running, as shown below:

The following is the non-replicated YAML that currently works

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: operational
spec:
  kafka:
    version:
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 31234
        type: loadbalancer
        tls: false
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.2"
    storage:
      type: ephemeral
  zookeeper:
    replicas: 1
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}

What is wrong with our replicated configuration?

[scholzj]

Thanks for moving this to discussions. However, you will still need to share the full logs and not just some screenshots.

[davesargrad]

@scholzj ty for helping us with this. we will place the full logs into the discussion!!

[ralphflat]

Attached are the logs from the operator and one of the zookeeper instances

strimzi-operator.log
pod.development-zookeeper-0.log

[scholzj]

Ok, so, the problem is in your DNs settings and in the weird way how ZooKeeper does TLS hostname verification. ZooKeeper uses reverse DNS lookup to verify the hostname. And in your case, for the Pod IP addresses, your DNS seems to return the hostname <_gateway> which is obviously not in the ZooKeeper certificate SANs.

So in general you have two options:

• Fix your DNS to return some more reasonable hostname matching the SANs from the certificate (for example development-zookeeper-1.development-zookeeper-nodes.devel.svc which is where it connects)
• Disable the hostname verification in ZooKeeper

As I said, there are plenty of older issues discussing this:

• Kafka PODs are not appearing, only zookeeper pods are running. #3155
• [Question] Zookeeper failed to verify host address after upgraded to 0.18.0 #3099
• etc.