部署 3 节点 controller ，broker 状态 alive 显示为 false

[drivebyer]

我用 operator 部署了一个 5.1.4 版本的 3 个 controller 2 个 broker 1 个 nameserver 的集群，等待一段时间后，在 nameserver 里执行命令：

[root@master0 ~]# kubectl -n mcamel-system exec -it name-service-0 -- ./mqadmin clusterList -n 127.0.0.1:9876
#Cluster Name           #Broker Name            #BID  #Addr                  #Version              #InTPS(LOAD)     #OutTPS(LOAD)  #Timer(Progress)        #PCWait(ms)  #Hour         #SPACE    #ACTIVATED
broker                  broker-0                0     192.168.137.126:10911  V5_1_4                 0.00(0,0ms)       0.00(0,0ms)  0-0(0.0w, 0.0, 0.0)               0  474775.65     0.6800          true
broker                  broker-0                2     192.168.84.199:10911   V5_1_4                 0.00(0,0ms)       0.00(0,0ms)  2-0(0.0w, 0.0, 0.0)               0  474775.65     0.6500         false

看起来是正常的。

在 controller 里执行命令：

[root@master0 ~]# kubectl -n mcamel-system exec -it controller-1 -- ./mqadmin getSyncStateSet -a 127.0.0.1:9878 -c broker -b broker-0

#brokerName	broker-0
#MasterBrokerId	1
#MasterAddr	192.168.137.126:10911
#MasterEpoch	1
#SyncStateSetEpoch	1
#SyncStateSetNums	1

InSyncReplica:	ReplicaIdentity{brokerName='broker-0', brokerId=1, brokerAddress='192.168.137.126:10911', alive=true}

NotInSyncReplica:	ReplicaIdentity{brokerName='broker-0', brokerId=2, brokerAddress='192.168.84.199:10911', alive=false}

显示其中一个 broker alive 为 false。

并且在 192.168.137.126:10911 中发现了下面的错误日志：

2024-02-29 15:50:26 ERROR AutoSwitchHAService_Executor_1 - Error happen when change SyncStateSet, broker:broker-0, masterAddress:192.168.137.126:10911, masterEpoch:1, oldSyncStateSet:[1], newSyncStateSet:[1, 2], syncStateSetEpoch:1
org.apache.rocketmq.client.exception.MQBrokerException: CODE: 2006  DESC: Rejecting alter syncStateSet request because the replicas {2} don't alive
For more information, please visit the url, https://rocketmq.apache.org/docs/bestPractice/06FAQ
	at org.apache.rocketmq.broker.out.BrokerOuterAPI.alterSyncStateSet(BrokerOuterAPI.java:1215)
	at org.apache.rocketmq.broker.controller.ReplicasManager.doReportSyncStateSetChanged(ReplicasManager.java:761)
	at org.apache.rocketmq.store.ha.autoswitch.AutoSwitchHAService.lambda$null$0(AutoSwitchHAService.java:263)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.apache.rocketmq.store.ha.autoswitch.AutoSwitchHAService.lambda$notifySyncStateSetChanged$1(AutoSwitchHAService.java:263)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2024-02-29 15:50:30 INFO ReplicasManager_ScheduledService_1 - Update controller leader address to controller-1.controller-svc-headless:9878

ps: 如果使用单节点的 controller 就不会有这个问题

[RongtongJin]

看着192.168.84.199没有像controller发心跳。。。但单节点controller不会有这个问题比较奇怪。。。可以看下192.168.84.199和controller的日志

[caigy]

broker和controller间的网络正常吗？broker中controller相关的环境变量提供一下。

[drivebyer]

在两个 broker 中用 nc 命令测试到 controller 的端口都是通的：

bash-4.4# nc -zv 192.168.84.246 9878
192.168.84.246 (192.168.84.246:9878) open
bash-4.4# nc -zv 192.168.137.114 9878
192.168.137.114 (192.168.137.114:9878) open
bash-4.4# nc -zv 192.168.180.10 9878
192.168.180.10 (192.168.180.10:9878) open

bash-4.4# env | grep -i controller
CONTROLLER_SVC_PORT_9878_TCP=tcp://10.105.182.106:9878
enableControllerMode=true
CONTROLLER_SVC_SERVICE_PORT=9878
CONTROLLER_SVC_PORT_9878_TCP_ADDR=10.105.182.106
CONTROLLER_SVC_PORT=tcp://10.105.182.106:9878
CONTROLLER_SVC_PORT_9878_TCP_PORT=9878
CONTROLLER_SVC_SERVICE_HOST=10.105.182.106
controllerAddr=controller-svc:9878
CONTROLLER_SVC_PORT_9878_TCP_PROTO=tcp
CONTROLLER_SVC_SERVICE_PORT_CONTROLLER=9878

[anguoming]

用operator部署5.2.0的rocketmq，当controller是3节点时也遇到了相同的问题，现象和日志报错跟前边描述的第二个现象相同

[drivebyer]

用operator部署5.2.0的rocketmq，当controller是3节点时也遇到了相同的问题，现象和日志报错跟前边描述的第二个现象相同

看来我不是一个人 :)

[EricKWXue]

5.2.0遇到同样的问题

2024-09-26 12:52:12 INFO ReplicasManager_ScheduledService_2 - Update controller leader address to 172.XX.XXX.XX:XXXX
2024-09-26 12:52:15 WARN ReplicasManager_ExecutorService_1 - Failed to start replicasManager, retry times:77, current state:REGISTER_TO_CONTROLLER_DONE, try it again
2024-09-26 12:52:15 ERROR ReplicasManager_ExecutorService_1 - Failed to try elect
org.apache.rocketmq.client.exception.MQBrokerException: CODE: 2012  DESC: Failed to elect a new master
For more information, please visit the url, https://rocketmq.apache.org/docs/bestPractice/06FAQ
        at org.apache.rocketmq.broker.out.BrokerOuterAPI.brokerElect(BrokerOuterAPI.java:1261)
        at org.apache.rocketmq.broker.controller.ReplicasManager.brokerElect(ReplicasManager.java:378)
        at org.apache.rocketmq.broker.controller.ReplicasManager.startBasicService(ReplicasManager.java:200)
        at org.apache.rocketmq.broker.controller.ReplicasManager.lambda$start$0(ReplicasManager.java:156)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

[drivebyer]

BTW, 我用的 yaml 是：

apiVersion: v1
kind: ConfigMap
metadata:
  name: broker-config
data:
  # BROKER_MEM sets the broker JVM, if set to "" then Xms = Xmx = max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
  BROKER_MEM: " -Xms2g -Xmx2g -Xmn1g "
  broker-common.conf: |
    # brokerClusterName, brokerName, brokerId are automatically generated by the operator and do not set it manually!!!
    deleteWhen=04
    fileReservedTime=48
    flushDiskType=ASYNC_FLUSH
    # set brokerRole to ASYNC_MASTER or SYNC_MASTER. DO NOT set to SLAVE because the replica instance will automatically be set!!!
    brokerRole=ASYNC_MASTER

---
apiVersion: rocketmq.apache.org/v1alpha1
kind: Broker
metadata:
  # name of broker cluster
  name: broker
spec:
  clusterMode: CONTROLLER
  # size is the number of the broker cluster, each broker cluster contains a master broker and [replicaPerGroup] replica brokers.
  size: 1
  # nameServers is the [ip:port] list of name service
  nameServers: ""
  # replicaPerGroup is the number of each broker cluster
  replicaPerGroup: 1
  # brokerImage is the customized docker image repo of the RocketMQ broker
  brokerImage: ghcr.m.daocloud.io/ksmartdata/rocketmq-broker:v5.1.4
  # imagePullPolicy is the image pull policy
  imagePullPolicy: Always
  # resources describes the compute resource requirements and limits
  resources:
    requests:
      memory: "2048Mi"
      cpu: "250m"
    limits:
      memory: "12288Mi"
      cpu: "500m"
  # allowRestart defines whether allow pod restart
  allowRestart: true
  # storageMode can be EmptyDir, HostPath, StorageClass
  storageMode: StorageClass
  # hostPath is the local path to store data
  hostPath: /data/rocketmq/broker
  # scalePodName is [Broker name]-[broker group number]-master-0
  scalePodName: broker-0-0-0
  # env defines custom env, e.g. BROKER_MEM
  env:
    - name: BROKER_MEM
      valueFrom:
        configMapKeyRef:
          name: broker-config
          key: BROKER_MEM
  # volumes defines the broker.conf
  volumes:
    - name: broker-config
      configMap:
        name: broker-config
        items:
          - key: broker-common.conf
            path: broker-common.conf
  # volumeClaimTemplates defines the storageClass
  volumeClaimTemplates:
    - metadata:
        name: broker-storage
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
---
apiVersion: rocketmq.apache.org/v1alpha1
kind: NameService
metadata:
  name: name-service
spec:
  # size is the the name service instance number of the name service cluster
  size: 1
  # nameServiceImage is the customized docker image repo of the RocketMQ name service
  nameServiceImage: ghcr.m.daocloud.io/ksmartdata/rocketmq-nameserver:v5.1.4
  # imagePullPolicy is the image pull policy
  imagePullPolicy: Always
  # hostNetwork can be true or false
  hostNetwork: false
  #  Set DNS policy for the pod.
  #  Defaults to "ClusterFirst".
  #  Valid values are 'ClusterFirstWithHostNet', 'ClusterFirst', 'Default' or 'None'.
  #  DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy.
  #  To have DNS options set along with hostNetwork, you have to specify DNS policy
  #  explicitly to 'ClusterFirstWithHostNet'.
  dnsPolicy: ClusterFirstWithHostNet
  # resources describes the compute resource requirements and limits
  resources:
    requests:
      memory: "512Mi"
      cpu: "250m"
    limits:
      memory: "1024Mi"
      cpu: "500m"
  # storageMode can be EmptyDir, HostPath, StorageClass
  storageMode: EmptyDir
  # hostPath is the local path to store data
  hostPath: /data/rocketmq/nameserver
  # volumeClaimTemplates defines the storageClass
  volumeClaimTemplates:
    - metadata:
        name: namesrv-storage
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
---
apiVersion: rocketmq.apache.org/v1alpha1
kind: Controller
metadata:
  # name of controller cluster
  name: controller
spec:
  # size is the number of controllers.
  size: 3
  # controllerImage is the customized docker image repo of the RocketMQ Controller
  controllerImage: ghcr.m.daocloud.io/ksmartdata/rocketmq-controller:v5.1.4
  # imagePullPolicy is the image pull policy
  imagePullPolicy: IfNotPresent
  # resources describes the compute resource requirements and limits
  resources:
    requests:
      memory: "0.5Gi"
    limits:
      memory: "2Gi"
  # storageMode can be EmptyDir, HostPath, StorageClass
  storageMode: StorageClass
  # hostPath is the local path to store data
  hostPath: /data/rocketmq/controller
  # volumeClaimTemplates defines the storageClass
  volumeClaimTemplates:
    - metadata:
        name: controller-storage
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 1Gi