[Questions] Consumers of quorum queue repeatedly receive messages

[dormanze]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.0.6

Erlang version used

26.2.x

Operating system (distribution) used

linux

How is RabbitMQ deployed?

Community Docker image

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

Details

# PASTE OUTPUT HERE, BETWEEN BACKTICKS

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

Details

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

Details

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

Details

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

Details

# PASTE rabbitmq.conf HERE, BETWEEN BACKTICKS

Steps to deploy RabbitMQ cluster

config rabbitmq and start

Steps to reproduce the behavior in question

I don't know

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

Details

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

Details

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

Details

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

quest 1:
I have 2 consumers but UI show only one consumer.


quest 2:
I pushed a message, and occasionally the same message was sent to both of my consumers at the same time.
trace message(my message have unique message ID, such as :8d5c5105-b313-11f0-a614-68f400000698):

================================================================================
2025-10-27T17:01:40.597+08:00: Message published

Node:         rabbit@rabbitmqservice-2
Connection:   [3fff:75:2006::210:2cd]:60810 -> [3fff:75:2006::210:1ef]:5673
Virtual host: /
User:         rabbitmq_admin
Channel:      2
Exchange:     mateinfo.durable.quorum.queues
Routing keys: [<<"1002.asyn-task-queue-adc-batch">>]
Routed queues: [<<"mateinfo.1002.asyn-task-queue-adc-batch-quorum">>]
Properties:   [{<<"priority">>,signedint,0},
               {<<"delivery_mode">>,signedint,2},
               {<<"headers">>,table,
                [{<<"X-B3-ParentSpanId">>,longstr,<<"d7652dd638406a17">>},
                 {<<"X-B3-Sampled">>,longstr,<<"0">>},
                 {<<"X-B3-SpanId">>,longstr,<<"95e78eb8216e48b9">>},
                 {<<"X-B3-TraceId">>,longstr,<<"b0d9f3dd0c9712be">>},
                 {<<"come_from">>,longstr,
                  <<"adc-schedule-job-adc-schedule-job-5b484ff776-zkwmb-fc18:0:0:0:0:0:0:698">>},
                 {<<"runtimeLanguage">>,void,undefined},
                 {<<"runtimeTimeZone">>,void,undefined},
                 {<<"tenant_id">>,longstr,<<"1002">>},
                 {<<"timestamp">>,long,1761555700595},
                 {<<"x-gde-call-chain-depth">>,longstr,<<"1">>},
                 {<<"x-gde-locale">>,longstr,<<"zh_CN">>},
                 {<<"x-gde-src-app">>,void,undefined},
                 {<<"x-gde-src-app-for-transfer">>,void,undefined},
                 {<<"x-gde-src-app-invoke">>,void,undefined},
                 {<<"x-gde-src-business-field">>,void,undefined},
                 {<<"x-gde-src-item">>,void,undefined},
                 {<<"x-gde-src-item-type">>,void,undefined},
                 {<<"x-gde-src-js-invoke">>,void,undefined},
                 {<<"x-gde-src-module">>,void,undefined},
                 {<<"x-gde-src-module-legacy">>,void,undefined},
                 {<<"x-gde-src-node">>,longstr,<<"ows-pm-middleware02">>},
                 {<<"x-gde-src-service">>,longstr,<<"adc-schedule-job">>},
                 {<<"x-gde-src-user-address">>,longstr,<<"75.254.194.136">>},
                 {<<"x-gde-src-user-agent">>,longstr,
                  <<"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 Edg/126.0.0.0">>},
                 {<<"x-gde-src-user-name">>,longstr,<<"admin">>},
                 {<<"x-gde-tenant-id">>,longstr,<<"1002">>},
                 {<<"x-gde-username">>,longstr,<<"admin">>}]},
               {<<"content_type">>,longstr,<<"text/plain">>}]
Payload: 
{"action":"Creat","taskDetail":{"durable":true,"task_id":"8d5c5105-b313-11f0-a614-68f400000698","task_type":"import-task","target_service":"adc-batch","tenant_id":"1002","user":null,"task_status":"Waiting","is_durable":true,"timeout_seconds":7200,"priority":1,"task_input":"{\"type\":\"excel\",\"source\":\"WEB_REST\",\"output\":null,\"uuid\":null,\"subTaskType\":null,\"templates\":[\"test/test/test_jskd_import\"],\"progressIds\":[\"test/test/test_jskd_import\"],\"user\":\"admin\",\"callback\":null,\"asyncTaskRunningContext\":null,\"removeFile\":false,\"fileToken\":\"eyJleHBpcmVUaW1lIjoxNzYxNjQyMTAwNTgxLCJ0ZW1wRmlsZURpciI6Ii90ZW1wL2FkYy1iYXRjaC8yMDI1LTEwLTI3LTA5LTAxLTQwLzVkNjFhZDg5LWI4MDYtNGZlYy1iYjIxLWJlZjhjNmI3YzY0OSIsImZpbGVEaXIiOiJhZGMtYmF0Y2gvaW1wb3J0LXRhc2stZmlsZS8xNzYxNTU1NzAwNTE1LzhkNTE0NzhkLWIzMTMtMTFmMC05ZTUzLTY4ZjQwMDAwMDZlNCIsInN0YXRlIjoiRVhJU1RJTkciLCJmaWxlQ2hlY2tSdWxlIjp7Im1heEZpbGVDb3VudCI6NSwibWF4RmlsZVNpemUiOjUyNDI4ODAwLCJwZXJtaXNzaW9uIjoiQ1JVRCJ9fUBAQDMwMzAzMzUzODYxMzdiNDhjYjlhNTU1MGZlYmNkNTg0MWFhZGQwYmZhZGYwOGQ1MTBlZTFlNjdhNWFmYjU1ZTk0NGY5Nzc=\",\"fileName\":\"test_jskd_import.xlsx\",\"dirPath\":null,\"batchImport\":false,\"batchImportSize\":1000,\"parameterNode\":null,\"userAddress\":\"75.254.194.136\",\"timeOut\":false,\"truncateModel\":false,\"inMemory\":false,\"dataImportContextList\":[],\"parameters\":{\"from\":\"adc-studio-web\",\"locale\":\"zh_CN\"},\"syncImport\":false}","remark":null,"task_distribute_type":"Queue","create_time":1761555700582,"create_from":"adc-batch-76f56cd448-vzrjn-fc18:0:0:0:0:0:0:6e4","callback_service_uri":null,"callback_parameters":null,"callback_service_method":null}}

================================================================================
2025-10-27T17:01:40.600+08:00: Message received

Node:         rabbit@rabbitmqservice-1
Connection:   [3fff:75:2006::210:312]:54094 -> [3fff:75:2006::210:2cd]:5673
Virtual host: /
User:         rabbitmq_admin
Channel:      12
Exchange:     mateinfo.durable.quorum.queues
Routing keys: [<<"1002.asyn-task-queue-adc-batch">>]
Queue:        mateinfo.1002.asyn-task-queue-adc-batch-quorum
Properties:   [{<<"priority">>,signedint,0},
               {<<"delivery_mode">>,signedint,2},
               {<<"headers">>,table,
                [{<<"X-B3-ParentSpanId">>,longstr,<<"d7652dd638406a17">>},
                 {<<"X-B3-Sampled">>,longstr,<<"0">>},
                 {<<"X-B3-SpanId">>,longstr,<<"95e78eb8216e48b9">>},
                 {<<"X-B3-TraceId">>,longstr,<<"b0d9f3dd0c9712be">>},
                 {<<"come_from">>,longstr,
                  <<"adc-schedule-job-adc-schedule-job-5b484ff776-zkwmb-fc18:0:0:0:0:0:0:698">>},
                 {<<"runtimeLanguage">>,void,undefined},
                 {<<"runtimeTimeZone">>,void,undefined},
                 {<<"tenant_id">>,longstr,<<"1002">>},
                 {<<"timestamp">>,long,1761555700595},
                 {<<"x-gde-call-chain-depth">>,longstr,<<"1">>},
                 {<<"x-gde-locale">>,longstr,<<"zh_CN">>},
                 {<<"x-gde-src-app">>,void,undefined},
                 {<<"x-gde-src-app-for-transfer">>,void,undefined},
                 {<<"x-gde-src-app-invoke">>,void,undefined},
                 {<<"x-gde-src-business-field">>,void,undefined},
                 {<<"x-gde-src-item">>,void,undefined},
                 {<<"x-gde-src-item-type">>,void,undefined},
                 {<<"x-gde-src-js-invoke">>,void,undefined},
                 {<<"x-gde-src-module">>,void,undefined},
                 {<<"x-gde-src-module-legacy">>,void,undefined},
                 {<<"x-gde-src-node">>,longstr,<<"ows-pm-middleware02">>},
                 {<<"x-gde-src-service">>,longstr,<<"adc-schedule-job">>},
                 {<<"x-gde-src-user-address">>,longstr,<<"75.254.194.136">>},
                 {<<"x-gde-src-user-agent">>,longstr,
                  <<"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 Edg/126.0.0.0">>},
                 {<<"x-gde-src-user-name">>,longstr,<<"admin">>},
                 {<<"x-gde-tenant-id">>,longstr,<<"1002">>},
                 {<<"x-gde-username">>,longstr,<<"admin">>}]},
               {<<"content_type">>,longstr,<<"text/plain">>}]
Payload: 
{"action":"Creat","taskDetail":{"durable":true,"task_id":"8d5c5105-b313-11f0-a614-68f400000698","task_type":"import-task","target_service":"adc-batch","tenant_id":"1002","user":null,"task_status":"Waiting","is_durable":true,"timeout_seconds":7200,"priority":1,"task_input":"{\"type\":\"excel\",\"source\":\"WEB_REST\",\"output\":null,\"uuid\":null,\"subTaskType\":null,\"templates\":[\"test/test/test_jskd_import\"],\"progressIds\":[\"test/test/test_jskd_import\"],\"user\":\"admin\",\"callback\":null,\"asyncTaskRunningContext\":null,\"removeFile\":false,\"fileToken\":\"eyJleHBpcmVUaW1lIjoxNzYxNjQyMTAwNTgxLCJ0ZW1wRmlsZURpciI6Ii90ZW1wL2FkYy1iYXRjaC8yMDI1LTEwLTI3LTA5LTAxLTQwLzVkNjFhZDg5LWI4MDYtNGZlYy1iYjIxLWJlZjhjNmI3YzY0OSIsImZpbGVEaXIiOiJhZGMtYmF0Y2gvaW1wb3J0LXRhc2stZmlsZS8xNzYxNTU1NzAwNTE1LzhkNTE0NzhkLWIzMTMtMTFmMC05ZTUzLTY4ZjQwMDAwMDZlNCIsInN0YXRlIjoiRVhJU1RJTkciLCJmaWxlQ2hlY2tSdWxlIjp7Im1heEZpbGVDb3VudCI6NSwibWF4RmlsZVNpemUiOjUyNDI4ODAwLCJwZXJtaXNzaW9uIjoiQ1JVRCJ9fUBAQDMwMzAzMzUzODYxMzdiNDhjYjlhNTU1MGZlYmNkNTg0MWFhZGQwYmZhZGYwOGQ1MTBlZTFlNjdhNWFmYjU1ZTk0NGY5Nzc=\",\"fileName\":\"test_jskd_import.xlsx\",\"dirPath\":null,\"batchImport\":false,\"batchImportSize\":1000,\"parameterNode\":null,\"userAddress\":\"75.254.194.136\",\"timeOut\":false,\"truncateModel\":false,\"inMemory\":false,\"dataImportContextList\":[],\"parameters\":{\"from\":\"adc-studio-web\",\"locale\":\"zh_CN\"},\"syncImport\":false}","remark":null,"task_distribute_type":"Queue","create_time":1761555700582,"create_from":"adc-batch-76f56cd448-vzrjn-fc18:0:0:0:0:0:0:6e4","callback_service_uri":null,"callback_parameters":null,"callback_service_method":null}}

================================================================================
2025-10-27T17:01:40.601+08:00: Message received

Node:         rabbit@rabbitmqservice-2
Connection:   [3fff:75:2006::210:380]:53044 -> [3fff:75:2006::210:1ef]:5673
Virtual host: /
User:         rabbitmq_admin
Channel:      12
Exchange:     mateinfo.durable.quorum.queues
Routing keys: [<<"1002.asyn-task-queue-adc-batch">>]
Queue:        mateinfo.1002.asyn-task-queue-adc-batch-quorum
Properties:   [{<<"priority">>,signedint,0},
               {<<"delivery_mode">>,signedint,2},
               {<<"headers">>,table,
                [{<<"X-B3-ParentSpanId">>,longstr,<<"d7652dd638406a17">>},
                 {<<"X-B3-Sampled">>,longstr,<<"0">>},
                 {<<"X-B3-SpanId">>,longstr,<<"95e78eb8216e48b9">>},
                 {<<"X-B3-TraceId">>,longstr,<<"b0d9f3dd0c9712be">>},
                 {<<"come_from">>,longstr,
                  <<"adc-schedule-job-adc-schedule-job-5b484ff776-zkwmb-fc18:0:0:0:0:0:0:698">>},
                 {<<"runtimeLanguage">>,void,undefined},
                 {<<"runtimeTimeZone">>,void,undefined},
                 {<<"tenant_id">>,longstr,<<"1002">>},
                 {<<"timestamp">>,long,1761555700595},
                 {<<"x-gde-call-chain-depth">>,longstr,<<"1">>},
                 {<<"x-gde-locale">>,longstr,<<"zh_CN">>},
                 {<<"x-gde-src-app">>,void,undefined},
                 {<<"x-gde-src-app-for-transfer">>,void,undefined},
                 {<<"x-gde-src-app-invoke">>,void,undefined},
                 {<<"x-gde-src-business-field">>,void,undefined},
                 {<<"x-gde-src-item">>,void,undefined},
                 {<<"x-gde-src-item-type">>,void,undefined},
                 {<<"x-gde-src-js-invoke">>,void,undefined},
                 {<<"x-gde-src-module">>,void,undefined},
                 {<<"x-gde-src-module-legacy">>,void,undefined},
                 {<<"x-gde-src-node">>,longstr,<<"ows-pm-middleware02">>},
                 {<<"x-gde-src-service">>,longstr,<<"adc-schedule-job">>},
                 {<<"x-gde-src-user-address">>,longstr,<<"75.254.194.136">>},
                 {<<"x-gde-src-user-agent">>,longstr,
                  <<"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 Edg/126.0.0.0">>},
                 {<<"x-gde-src-user-name">>,longstr,<<"admin">>},
                 {<<"x-gde-tenant-id">>,longstr,<<"1002">>},
                 {<<"x-gde-username">>,longstr,<<"admin">>}]},
               {<<"content_type">>,longstr,<<"text/plain">>}]
Payload: 
{"action":"Creat","taskDetail":{"durable":true,"task_id":"8d5c5105-b313-11f0-a614-68f400000698","task_type":"import-task","target_service":"adc-batch","tenant_id":"1002","user":null,"task_status":"Waiting","is_durable":true,"timeout_seconds":7200,"priority":1,"task_input":"{\"type\":\"excel\",\"source\":\"WEB_REST\",\"output\":null,\"uuid\":null,\"subTaskType\":null,\"templates\":[\"test/test/test_jskd_import\"],\"progressIds\":[\"test/test/test_jskd_import\"],\"user\":\"admin\",\"callback\":null,\"asyncTaskRunningContext\":null,\"removeFile\":false,\"fileToken\":\"eyJleHBpcmVUaW1lIjoxNzYxNjQyMTAwNTgxLCJ0ZW1wRmlsZURpciI6Ii90ZW1wL2FkYy1iYXRjaC8yMDI1LTEwLTI3LTA5LTAxLTQwLzVkNjFhZDg5LWI4MDYtNGZlYy1iYjIxLWJlZjhjNmI3YzY0OSIsImZpbGVEaXIiOiJhZGMtYmF0Y2gvaW1wb3J0LXRhc2stZmlsZS8xNzYxNTU1NzAwNTE1LzhkNTE0NzhkLWIzMTMtMTFmMC05ZTUzLTY4ZjQwMDAwMDZlNCIsInN0YXRlIjoiRVhJU1RJTkciLCJmaWxlQ2hlY2tSdWxlIjp7Im1heEZpbGVDb3VudCI6NSwibWF4RmlsZVNpemUiOjUyNDI4ODAwLCJwZXJtaXNzaW9uIjoiQ1JVRCJ9fUBAQDMwMzAzMzUzODYxMzdiNDhjYjlhNTU1MGZlYmNkNTg0MWFhZGQwYmZhZGYwOGQ1MTBlZTFlNjdhNWFmYjU1ZTk0NGY5Nzc=\",\"fileName\":\"test_jskd_import.xlsx\",\"dirPath\":null,\"batchImport\":false,\"batchImportSize\":1000,\"parameterNode\":null,\"userAddress\":\"75.254.194.136\",\"timeOut\":false,\"truncateModel\":false,\"inMemory\":false,\"dataImportContextList\":[],\"parameters\":{\"from\":\"adc-studio-web\",\"locale\":\"zh_CN\"},\"syncImport\":false}","remark":null,"task_distribute_type":"Queue","create_time":1761555700582,"create_from":"adc-batch-76f56cd448-vzrjn-fc18:0:0:0:0:0:0:6e4","callback_service_uri":null,"callback_parameters":null,"callback_service_method":null}}

[dormanze]

After I restarted all my consumer clients, a more strange situation occurred: the consumers were listening normally, but all messages were in the Unacked state, yet my consumers did not receive any messages.

[dormanze]

I have checked the historical metrics of the nodes and found that the disk of one RabbitMQ node was once full. I'm not sure if this is the reason for the queue anomaly.

[michaelklishin]

If a node is out of disk space, you never know what it could not persist to the Raft log.

The recommendation is to decomission and replace such nodes, and have at the very least OS-level monitoring in place.