Few Brokers not coming up during cluster initialisation.

[Sai-Charan-Madhvaraj]

We're running a Kafka Cluster on version 3.0.0 on AKS. The configuration for the same is below -

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: cdt-north-1-kafka-cluster
  namespace: kafka
spec:
  kafka:
    version: 3.0.0
    replicas: 3
    resources:
      requests:
        memory: 500Mi
        cpu: "100m"
      limits:
        memory: 600Mi
        cpu: "1"
    listeners:
      - name: external
        port: 9094
        type: ingress
        tls: true
        authentication:
          type: scram-sha-512
        configuration:
          bootstrap:
            host: cdt-north-1-kafka-cluster-1-bootstrap.net
          brokers:
          - broker: 0
            host: cdt-north-1-kafka-cluster-1-kafka-0.net
          - broker: 1
            host: cdt-north-1-kafka-cluster-1-kafka-1.net
          - broker: 2
            host: cdt-north-1-kafka-cluster-1-kafka-2.net
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    authorization:
      type: simple
      superUsers:
        - admin
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "3.0"
      inter.broker.protocol.version: "3.0"
      default.replication.factor: 3
      min.insync.replicas: 3
      num.partitions: 1
      log.segment.bytes: 1073741824
      log.retention.check.interval.ms: 300000
      num.network.threads: 3
      num.io.threads: 8
      num.recovery.threads.per.data.dir: 1
      socket.send.buffer.bytes: 102400
      socket.receive.buffer.bytes: 102400
      socket.request.max.bytes: 104857600
      group.initial.rebalance.delay.ms: 0
      replica.fetch.max.bytes: 1048576
      auto.create.topics.enable: false
      delete.topic.enable: true
      offsets.topic.num.partitions: 50
      offsets.commit.required.acks: -1
      queued.max.requests: 500
      replica.socket.receive.buffer.bytes: 65536
      log.roll.ms: 604800000
      log.retention.bytes: 1073741824
      log.retention.ms: 1680000
      log.segment.delete.delay.ms: 60000
      log.cleanup.policy: compact,delete
      log.cleaner.backoff.ms: 15000
      log.cleaner.delete.retention.ms: 86400000
      log.cleaner.dedupe.buffer.size: 134217728
      log.cleaner.io.buffer.load.factor: 0.9
      log.cleaner.threads: 8
      log.cleaner.io.max.bytes.per.second: 1.7976931348623157E308
      message.max.bytes: 1048576
      compression.type: producer
      log.flush.scheduler.interval.ms: 2000
      log.flush.interval.ms: 50000
      log.flush.interval.messages: 100000
      replica.lag.time.max.ms: 500
      auto.leader.rebalance.enable: true
      leader.imbalance.check.interval.seconds: 300
      leader.imbalance.per.broker.percentage: 10
      unclean.leader.election.enable: false
      group.initial.rebalance.delay.ms: 3000
      zookeeper.connection.timeout.ms: 6000
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 20Gi
        deleteClaim: false
        class: kafka-storage
      - id: 1
        type: persistent-claim
        size: 20Gi
        deleteClaim: false
        class: kafka-storage
    template:
      pod:
        tolerations:
        - key: "kubernetes.azure.com/scalesetpriority"
          operator: "Equal"
          value: "spot"
          effect: "NoSchedule"
        topologySpreadConstraints:
        - maxSkew: 2
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: cdt-north-1-kafka-cluster-kafka
  zookeeper:
    replicas: 3
    resources:
      requests:
        memory: 200Mi
        cpu: "100m"
      limits:
        memory: 400Mi
        cpu: "200m"
    storage:
      type: persistent-claim
      size: 20Gi
      deleteClaim: true
      class: zookeeper-storage
    template:
      pod:
        tolerations:
        - key: "kubernetes.azure.com/scalesetpriority"
          operator: "Equal"
          value: "spot"
          effect: "NoSchedule"
        topologySpreadConstraints:
        - maxSkew: 2
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: cdt-north-1-kafka-cluster-zookeeper
  entityOperator:
    tlsSidecar:
      resources:
        requests:
          cpu: 100m
          memory: 64Mi
        limits:
          cpu: 200m
          memory: 128Mi
    topicOperator:
      resources:
        requests:
          memory: 512Mi
          cpu: "100m"
        limits:
          memory: 512Mi
          cpu: "200m"
    userOperator:
      resources:
        requests:
          memory: 512Mi
          cpu: "100m"
        limits:
          memory: 512Mi
          cpu: "200m"
  cruiseControl:
    tlsSidecar:
      resources:
        requests:
          memory: 512Mi
          cpu: "1"
        limits:
          memory: 2Gi
          cpu: "2"
    resources:
      requests:
        memory: 512Mi
        cpu: "1"
      limits:
        memory: 2Gi
        cpu: "2"
  kafkaExporter:
    resources:
      requests:
        cpu: 100m
        memory: 64Mi
      limits:
        cpu: 200m
        memory: 128Mi
    topicRegex: ".*"
    groupRegex: ".*"

The logs can I observed from the failed brokers were -

**2023-01-19 11:53:58,791 WARN Couldn't resolve server cdt-north-1-kafka-cluster-kafka-brokers:9091 from bootstrap.servers as DNS resolution failed for pp-kafka-kafka-brokers (org.apache.kafka.clients.ClientUtils) [main]**
2023-01-19 11:53:58,791 INFO [Producer clientId=CruiseControlMetricsReporter] Closing the Kafka producer with timeoutMillis = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer) [main]
2023-01-19 11:53:58,791 INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics) [main]
2023-01-19 11:53:58,791 INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics) [main]
2023-01-19 11:53:58,791 INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics) [main]
2023-01-19 11:53:58,795 INFO App info kafka.producer for CruiseControlMetricsReporter unregistered (org.apache.kafka.common.utils.AppInfoParser) [main]
2023-01-19 11:53:58,799 **ERROR [KafkaServer id=4] Fatal error during KafkaServer startup.** Prepare to shutdown (kafka.server.KafkaServer) [main]
org.apache.kafka.common.KafkaException: Failed to construct kafka producer
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:439)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:290)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:317)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:302)
	at com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter.configure(CruiseControlMetricsReporter.java:170)
	at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:405)
	at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstances(AbstractConfig.java:478)
	at kafka.server.DynamicMetricsReporters.createReporters(DynamicBrokerConfig.scala:800)
	at kafka.server.DynamicMetricsReporters.<init>(DynamicBrokerConfig.scala:749)
	at kafka.server.DynamicBrokerConfig.addReconfigurables(DynamicBrokerConfig.scala:248)
	at kafka.server.KafkaServer.startup(KafkaServer.scala:408)
	at kafka.Kafka$.main(Kafka.scala:109)
	at kafka.Kafka.main(Kafka.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:88)
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:47)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:413)
	... 12 more

We also observed that when the new broker is trying to spin up, the DNS resolution is not working for the headless service as highlighted in the error log.
What can be the resolution for this issue?

[scholzj]

You should probably share more information ... operator logs, full broker logs, Kubernetes events, and the YAML of the pods to make it clear what state they are in. The CPU and memory you give to Kafka and Zoo is super small. But otherwise, the config looks normal.

[Sai-Charan-Madhvaraj]

.

[scholzj]

The only error in your broker log seems to be about some DNS issue. So that is probably something to look into - not sure if that is the reason why it does not start.

[Sai-Charan-Madhvaraj]

Yeah, thats the warning i am seeing . But when i try to do no nslookup from other pods in that namespace it resolves fine..soon after 5th broker starts nslookup is failing as well, SERVER FAIL.

And its trying to resolve the headless services of bootstrap.

[Sai-Charan-Madhvaraj]

Any checks that we need to verify from kubernetes side here ?

[scholzj]

Not really an expert on Kubernetes DNS I'm afraid. It could be for example related to a particular worker node - that is something I saw in the past. But I'm sure there might be other issues as well causing this.