RabbitMQ Cluster of 3 nodes lose messages if uses quorum queues

[hideowork]

Hi all,

I need your help to find solve of my issue with rabbitmq cluster.

Issue describing:

Rabbitmq cluster has 3 nodes. There are losing messages sometimes (<0.05%) between publishing and receiving by consumer.
Info:
App coded with PHP and uses lib php-amqplib, ver.3.5.4. All queues type = quorum.
Durability is enabled for all exchanges and queues.
All messages are persistent. And uses acks for publishing and receiving messages.
Our test publish and receive ~9000 messages during 2 mins.

Rabbitmq version 3.11.20. Also tried 3.11.19 and 3.12.2. But no difference.
Rabbitmq cluster was installed using rabbitmq-operator in kubernetes cluster. Also created vhosts, users, permissions using operator CRDs. Each of 3 pods has 1 CPU and 3 GiB resources (limits = requests). NetworkPolicy allows app connect to rabbitmq nodes via k8s balancing service. Network problems were not discovered.

Described issue not happens in cases:

1. Uses classic queues and 3 nodes in cluster. Other settings are equal.
2. Uses quorum queues and 1-node cluster. Other settings are equal.

I guess that issue related to combination of settings: quorum queues and multi-node cluster. But I found nothing clear in debug logs of rabbitmq-server and app. Rabbitmq-server logs show only "connections" info. And app logs show only successful publishes and receiving. Nothing wrong or weird. Some messages sometimes just vanished. Also checked dead letters and there is not these "vanished" messages.

Configs of app and rabbitmq:

App config: https://pastebin.com/zjtAw9NT
App class for messages headers: https://pastebin.com/u1a4saxw
rabbitmq-diagnostics status: https://pastebin.com/qrHXKdtV
rabbitmq-diagnostics environment: https://pastebin.com/YvVuT8We
rabbitmq cluster CRD: https://pastebin.com/pqGKjr1z

Expected behaviour:

Cluster of 3 nodes doesn't lose messages when uses quorum queues.

Could you help to fix this?

[lukebakken]

Quorum queues in a cluster have been tested and used in production for a couple years now. It's unlikely you have found a bug. More likely is that you have a bug in your application code, or configuration. My guess is that you are not using publisher confirmations correctly.

Without a reliable way to reproduce what you report, it is impossible for us to assist you. We would need the source code to your test application, as well as an exact series of steps to reproduce the issue.

[hideowork]

Hi @lukebakken
Thanks for your reply.
But how you explain that it works if use classic queues with 3 nodes or quorum queues with 1 node? App code is the same.
I would believe that is some wrong configuration in rabbit cluster config or some related. But that bug is not in app.