Skip to content

[BUG] kafka separated stop -> start broker crash #9938

@JashBook

Description

@JashBook

Describe the bug
A clear and concise description of what the bug is.

helm get notes -n kb-system kb-addon-kafka  
NOTES:
CHART NAME: kafka
CHART VERSION: 1.1.0-alpha.0
APP VERSION: 3.3.2

KubeBlocks Kafka server cluster definition, start create your Kafka Server Cluster with following command:

    kbcli cluster create kafka


Release Information:
  Commit ID: "900fa110fb5bc1b3b09ebe48c35a22f9b6eb0a5c"
  Commit Time: "2025-12-03 14:43:50 +0800"
  Release Branch: "v1.1.0-alpha.5"
  Release Time:  "2025-12-15 09:39:54 +0800"
  Enterprise: "false"

➜  ~ 
➜  ~ kbcli version
Kubernetes: v1.30.4-vke.10
KubeBlocks: 1.1.0-alpha.5
kbcli: 1.0.2-beta.0

To Reproduce
Steps to reproduce the behavior:

  1. create cluster
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: kafka-gornmp
  namespace: default
  annotations:
    "kubeblocks.io/extra-env": '{"KB_KAFKA_ENABLE_SASL":"false","KB_KAFKA_BROKER_HEAP":"-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64","KB_KAFKA_CONTROLLER_HEAP":"-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64","KB_KAFKA_PUBLIC_ACCESS":"false"}'
spec:
  clusterDef: kafka
  topology: separated_monitor
  terminationPolicy: Delete
  componentSpecs:
    - name: kafka-broker
      tls: false
      disableExporter: true
      replicas: 3
      serviceVersion: 3.3.2
      services:
        - name: advertised-listener
          serviceType: ClusterIP
          podService: true
      resources:
        requests:
          cpu: 500m
          memory: 1Gi
        limits:
          cpu: 500m
          memory: 1Gi
      env:
        - name: KB_BROKER_DIRECT_POD_ACCESS
          value: "false"
        - name: KB_KAFKA_ENABLE_SASL_SCRAM
          value: "false"
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: 
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
        - name: metadata
          spec:
            storageClassName: 
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
    - name: kafka-controller
      tls: false
      replicas: 3
      serviceVersion: 3.3.2
      disableExporter: true
      resources:
        requests:
          cpu: 500m
          memory: 1Gi
        limits:
          cpu: 500m
          memory: 1Gi
      volumeClaimTemplates:
        - name: metadata
          spec:
            storageClassName: 
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
    - name: kafka-exporter
      replicas: 1
      disableExporter: true
      env:
        - name: KB_KAFKA_ENABLE_SASL_SCRAM
          value: "false"
      resources:
        requests:
          cpu: 500m
          memory: 1Gi
        limits:
          cpu: 500m
          memory: 1Gi

  1. stop
kbcli cluster stop kafka-gornmp --auto-approve 
OpsRequest kafka-gornmp-stop-nh8f8 created successfully, you can view the progress:
	kbcli cluster describe-ops kafka-gornmp-stop-nh8f8 -n default

kubectl get cluster 
NAME           CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS    AGE
kafka-gornmp   kafka                Delete               Stopped   2m22s
  1. start
kbcli cluster start kafka-gornmp 
  1. See error
kubectl get cluster kafka-gornmp 
NAME           CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
kafka-gornmp   kafka                Delete               Abnormal   2m52s
➜  ~ 
➜  ~ kubectl get pod 
NAME                          READY   STATUS             RESTARTS      AGE
kafka-gornmp-kafka-broker-0   1/2     CrashLoopBackOff   1 (14s ago)   32s
kafka-gornmp-kafka-broker-1   1/2     CrashLoopBackOff   1 (14s ago)   32s
kafka-gornmp-kafka-broker-2   1/2     CrashLoopBackOff   1 (14s ago)   32s

logs error pod

kubectl logs kafka-gornmp-kafka-broker-0 --tail 50 --previous 
Defaulted container "kafka" out of: kafka, jmx-exporter, kafkatool (init)
convert prop 
line '' has no value; skipped
convert prop delegation.token.max.lifetime.ms=604800000
[cfg]export KAFKA_CFG_DELEGATION_TOKEN_MAX_LIFETIME_MS=604800000
convert prop delegation.token.expiry.time.ms=86400000
[cfg]export KAFKA_CFG_DELEGATION_TOKEN_EXPIRY_TIME_MS=86400000
convert prop delegation.token.expiry.check.interval.ms=3600000
[cfg]export KAFKA_CFG_DELEGATION_TOKEN_EXPIRY_CHECK_INTERVAL_MS=3600000
convert prop password.encoder.cipher.algorithm=AES/CBC/PKCS5Padding
[cfg]export KAFKA_CFG_PASSWORD_ENCODER_CIPHER_ALGORITHM=AES/CBC/PKCS5Padding
convert prop password.encoder.key.length=128
[cfg]export KAFKA_CFG_PASSWORD_ENCODER_KEY_LENGTH=128
convert prop password.encoder.iterations=4096
[cfg]export KAFKA_CFG_PASSWORD_ENCODER_ITERATIONS=4096
convert prop 
line '' has no value; skipped
convert prop 
line '' has no value; skipped
convert prop 
line '' has no value; skipped
convert prop 
line '' has no value; skipped
convert prop allow.everyone.if.no.acl.found=true
[cfg]export KAFKA_CFG_ALLOW_EVERYONE_IF_NO_ACL_FOUND=true
convert prop 
line '' has no value; skipped
convert prop 
line '' has no value; skipped
KAFKA_KRAFT_CLUSTER_ID=08e7ab09-a68f-4881-a9aa-50e2fef2835c
export KAFKA_KRAFT_CLUSTER_ID=08e7ab09-a68f-4881-a9a
find advertised_ports:kafka-gornmp-kafka-broker-advertised-listener-0:9092
find pod_name_ordinal:0
find svc_name:kafka-gornmp-kafka-broker-advertised-listener-0,port:9092,svc_name_ordinal:0
Found matching svcName and port for podName 'kafka-gornmp-kafka-broker-0', BROKER_ADVERTISED_PORT: kafka-gornmp-kafka-broker-advertised-listener-0:9092,kafka-gornmp-kafka-broker-advertised-listener-1:9092,kafka-gornmp-kafka-broker-advertised-listener-2:9092. svcName: kafka-gornmp-kafka-broker-advertised-listener-0, port: 9092.
[cfg]KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL://kafka-gornmp-kafka-broker-0.kafka-gornmp-kafka-broker-headless.default.svc.cluster.local:9094,CLIENT://kafka-gornmp-kafka-broker-0.kafka-gornmp-kafka-broker-headless.default.svc.cluster.local:9092
[cfg]KAFKA_CFG_NODE_ID=100
[cfg]export KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=,for kafka-broker.
[cfg]log to /bitnami/kafka/logs log4j configuration added.
kafka 10:10:14.67 
kafka 10:10:14.67 Welcome to the Bitnami kafka container
kafka 10:10:14.67 Subscribe to project updates by watching https://github.com/bitnami/containers
kafka 10:10:14.67 Submit issues and feature requests at https://github.com/bitnami/containers/issues
kafka 10:10:14.67 
kafka 10:10:14.67 INFO  ==> ** Starting Kafka setup **
kafka 10:10:14.69 DEBUG ==> Validating settings in KAFKA_* env vars...
kafka 10:10:14.70 ERROR ==> KRaft requires KAFKA_CFG_CONTROLLER_QUORUM_VOTERS to be set
kafka 10:10:14.70 WARN  ==> KAFKA_CFG_NODE_ID must match what is set in KAFKA_CFG_CONTROLLER_QUORUM_VOTERS
kafka 10:10:14.70 WARN  ==> KAFKA_CFG_PROCESS_ROLES must include 'controller' for KRaft
kafka 10:10:14.70 WARN  ==> KAFKA_CFG_LISTENERS must include a listener for CONTROLLER
kafka 10:10:14.70 WARN  ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment.
kubectl logs kafka-gornmp-kafka-broker-1 --tail 20 --previous
Defaulted container "kafka" out of: kafka, jmx-exporter, kafkatool (init)
find pod_name_ordinal:1
find svc_name:kafka-gornmp-kafka-broker-advertised-listener-0,port:9092,svc_name_ordinal:0
find svc_name:kafka-gornmp-kafka-broker-advertised-listener-1,port:9092,svc_name_ordinal:1
Found matching svcName and port for podName 'kafka-gornmp-kafka-broker-1', BROKER_ADVERTISED_PORT: kafka-gornmp-kafka-broker-advertised-listener-0:9092,kafka-gornmp-kafka-broker-advertised-listener-1:9092,kafka-gornmp-kafka-broker-advertised-listener-2:9092. svcName: kafka-gornmp-kafka-broker-advertised-listener-1, port: 9092.
[cfg]KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL://kafka-gornmp-kafka-broker-1.kafka-gornmp-kafka-broker-headless.default.svc.cluster.local:9094,CLIENT://kafka-gornmp-kafka-broker-1.kafka-gornmp-kafka-broker-headless.default.svc.cluster.local:9092
[cfg]KAFKA_CFG_NODE_ID=101
[cfg]export KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=,for kafka-broker.
[cfg]log to /bitnami/kafka/logs log4j configuration added.
kafka 10:10:24.67 
kafka 10:10:24.67 Welcome to the Bitnami kafka container
kafka 10:10:24.67 Subscribe to project updates by watching https://github.com/bitnami/containers
kafka 10:10:24.67 Submit issues and feature requests at https://github.com/bitnami/containers/issues
kafka 10:10:24.67 
kafka 10:10:24.67 INFO  ==> ** Starting Kafka setup **
kafka 10:10:24.70 DEBUG ==> Validating settings in KAFKA_* env vars...
kafka 10:10:24.70 ERROR ==> KRaft requires KAFKA_CFG_CONTROLLER_QUORUM_VOTERS to be set
kafka 10:10:24.70 WARN  ==> KAFKA_CFG_NODE_ID must match what is set in KAFKA_CFG_CONTROLLER_QUORUM_VOTERS
kafka 10:10:24.70 WARN  ==> KAFKA_CFG_PROCESS_ROLES must include 'controller' for KRaft
kafka 10:10:24.70 WARN  ==> KAFKA_CFG_LISTENERS must include a listener for CONTROLLER
kafka 10:10:24.76 WARN  ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment.
➜  ~ 
➜  ~ kubectl logs kafka-gornmp-kafka-broker-2 --tail 20 --previous
Defaulted container "kafka" out of: kafka, jmx-exporter, kafkatool (init)
find svc_name:kafka-gornmp-kafka-broker-advertised-listener-0,port:9092,svc_name_ordinal:0
find svc_name:kafka-gornmp-kafka-broker-advertised-listener-1,port:9092,svc_name_ordinal:1
find svc_name:kafka-gornmp-kafka-broker-advertised-listener-2,port:9092,svc_name_ordinal:2
Found matching svcName and port for podName 'kafka-gornmp-kafka-broker-2', BROKER_ADVERTISED_PORT: kafka-gornmp-kafka-broker-advertised-listener-0:9092,kafka-gornmp-kafka-broker-advertised-listener-1:9092,kafka-gornmp-kafka-broker-advertised-listener-2:9092. svcName: kafka-gornmp-kafka-broker-advertised-listener-2, port: 9092.
[cfg]KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL://kafka-gornmp-kafka-broker-2.kafka-gornmp-kafka-broker-headless.default.svc.cluster.local:9094,CLIENT://kafka-gornmp-kafka-broker-2.kafka-gornmp-kafka-broker-headless.default.svc.cluster.local:9092
[cfg]KAFKA_CFG_NODE_ID=102
[cfg]export KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=,for kafka-broker.
[cfg]log to /bitnami/kafka/logs log4j configuration added.
kafka 10:10:11.67 
kafka 10:10:11.67 Welcome to the Bitnami kafka container
kafka 10:10:11.67 Subscribe to project updates by watching https://github.com/bitnami/containers
kafka 10:10:11.67 Submit issues and feature requests at https://github.com/bitnami/containers/issues
kafka 10:10:11.67 
kafka 10:10:11.67 INFO  ==> ** Starting Kafka setup **
kafka 10:10:11.70 DEBUG ==> Validating settings in KAFKA_* env vars...
kafka 10:10:11.70 ERROR ==> KRaft requires KAFKA_CFG_CONTROLLER_QUORUM_VOTERS to be set
kafka 10:10:11.70 WARN  ==> KAFKA_CFG_NODE_ID must match what is set in KAFKA_CFG_CONTROLLER_QUORUM_VOTERS
kafka 10:10:11.70 WARN  ==> KAFKA_CFG_PROCESS_ROLES must include 'controller' for KRaft
kafka 10:10:11.70 WARN  ==> KAFKA_CFG_LISTENERS must include a listener for CONTROLLER
kafka 10:10:11.71 WARN  ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

Labels

kind/bugSomething isn't working

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions