Need Help - Producer failure with NOT_LEADER_OR_FOLLOWER - constantly refreshing metadata.

[pk-hugo]

Hey Community!

I'm here hoping to find the root cause for this and learn more.

We have a strimzi kafka cluster in our k8s cluster. We are using strimzi 0.45.0 with KRAFT.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: {{ .Values.clusterName }}
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled # change from "migration" to "enabled" to finalize the migration process
  labels:
    helm-revision: "{{ .Release.Revision }}"
spec:
  clusterCa:
    renewalDays: 30
    validityDays: 365
    generateCertificateAuthority: true
  clientsCa:
    renewalDays: 30
    validityDays: 365
    generateCertificateAuthority: true
  kafka:
    version: {{ .Values.kafka.version }}
    replicas: {{ .Values.kafka.replicas }}
    resources:
      requests:
        memory: {{ .Values.kafka.memory }}
        cpu: {{ .Values.kafka.cpu }}
      limits:
        memory: {{ .Values.kafka.memory }}
        cpu: {{ .Values.kafka.cpu }}
    authorization:
      type: simple
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
    config:
      log.retention.hours: 720 #30 days
      log.flush.interval.ms: 3000
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      auto.create.topics.enable: false
      socket.request.max.bytes: 2097152000
      config.providers: secrets,configmaps
      config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider
      config.providers.configmaps.class: io.strimzi.kafka.KubernetesConfigMapConfigProvider
    storage:
      type: persistent-claim
      size: {{ .Values.kafka.storageSize }}
      class: {{ .Values.storageClass.name }}
      deleteClaim: false
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: {{ .Values.clusterMetrics.name }}
          key: {{ .Values.clusterMetrics.kafkaMetricsKeyName }}
    template:
      pod:
        metadata:
          annotations:
            prometheus.io/should_be_scraped: "true"
            prometheus.io/scrape_port: "9404"
            prometheus.io/metric_path: "/metrics"
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: node-type
                      operator: In
                      values:
                        - {{ .Values.nodeType }}
          topologySpreadConstraints:
            - maxSkew: 1
              minDomains: 2
              topologyKey: "topology.kubernetes.io/zone"
              whenUnsatisfiable: DoNotSchedule
              labelSelector:
                matchLabels:
                  strimzi.io/name: {{ .Values.clusterName }}-kafka
              matchLabelKeys:
                - pod-template-hash
  entityOperator:
    topicOperator: {}
    userOperator: {}
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: node-type
                      operator: In
                      values:
                        - {{ .Values.nodeType }}
          topologySpreadConstraints:
            - maxSkew: 1
              minDomains: 2
              topologyKey: "topology.kubernetes.io/zone"
              whenUnsatisfiable: DoNotSchedule
              labelSelector:
                matchLabels:
                  strimzi.io/name: {{ .Values.clusterName }}-entity-operator
              matchLabelKeys:
                - pod-template-hash

Our producers are failing constantly with the below error. This log keeps repeating

2025-06-12 16:52:07 WARN Sender - [Producer clientId=producer-1] Got error produce response with correlation id 3829 on topic-partition topic-a-0, retrying (2147483599 attempts left). Error: NOT_LEADER_OR_FOLLOWER

2025-06-12 16:52:07 WARN Sender - [Producer clientId=producer-1] Received invalid metadata error in produce request on partition topic-a-0 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now

We thought the issue is with the leader broker for that partition and restarted that broker. But even when the partition leader changed to a different broker we are not able to produce any messages and the error pops up again. Surprisingly what we noticed is whenever we restart our brokers and try producing the first few messages will be pushed and consumed. Then once the error pops up the producer starts failing again.

Then we thought the error is with one partition. So we tried pushing to other partitions it worked initially but started failing again after some time.

We have also tried deleting the topic and creating it again. Even then the same issue started reproducing.

We tried increasing the delay between fetching the metadata from 100 ms to 1000ms - did not work

We checked if any consumer is constantly reconnecting making the brokers to keep shuffling between partitions always - we did not find any consumer doing that.

We restarted all the brokers to reset the state again for all of them - did not work the error came again.

[srinivasmummareddy]

@pk-hugo Did you find the fix?

[katheris]

Hey @pk-hugo, can you share some more information about the problem, specifically:

• Is this a new cluster, or one that has been running a while?
• What client library are you using for the producers? (e.g. Java or another language, what version etc)
• Where are your producer applications running? I assume they must be within the same Kubernetes cluster, which bootstrap service are they using to connect?
• Can you see from collecting metrics which broker is leader for the topic-partition you are producing to? Does the leader keep changing or is it remaining static?

The cluster metrics are listed here: https://kafka.apache.org/documentation/#kraft_monitoring