StatefulSet RabbitMQ queue msg got cleaned up during upgrade.

[CassieZAHNG]

Hey,

Got question about how RabbitMQ maintain a queue status during upgrade. My RabbitMQ is defined as a statefulset, without specifying any persistent volume or persistent volume claims, using rolling upgrade strategy to upgrade. I assumed during upgrade period, RabbitMQ will take care of migrating queue status to new upgraded image? Is my understanding right? From observation, I find the queue is cleaned up with 0 msg inside after upgrade.

[michaelklishin]

RabbitMQ node data directory must use a persistent volume, otherwise all node data will be gone after a restart. With any reasonably popular RabbitMQ OCI/Docker image, no data is stored in the image itself because images, well, are supposed to be immutable.

If you DIY install RabbitMQ on Kubernetes, see this documentation section, or better yet, use the Kuberenetes Operator instead of rolling your own installation.

[michaelklishin]

I have forgotten that there is a blog post on the subject of what is involved in running RabbitMQ on Kubernetes. Again, the recommendation is to use the cluster Operator (the post is from 3 years ago when the Operator was nowhere near as mature).

[CassieZAHNG]

Thanks for reply @michaelklishin and the recommendation on cluster operator, will definitely have a look! Just got a follow up question regarding the link you provided above, Use a Persistent Volume for Node Data session mentioned that:

"If a transient volume is ..... Syncing all of this data on every node restart would be highly inefficient. In case of a loss of quorum during a rolling restart, this will also lead to data loss...."

What the syncing data on every node restart exactly mean? In what scenarios would the syncing data from other nodes happens? cause if a node is able to automatically sync data during a rolling restart from other existing nodes (e.g., upgrade rabbit image), why we need the RabbitMQ node has persistent volume?

[michaelklishin]

When a node starts without an existing data directory (due to a transient volume) and performs peer discovery, it will have to sync its schema from peers, and replicated queue types would have to sync their contents from scratch, which can lead to massive data transfers for long queues or queues with very large messages.

[michaelklishin]

why we need the RabbitMQ node has persistent volume?

RabbitMQ is a stateful distributed service that stores most of its data on disk. Quorum queues and streams use Raft, a consensus protocol that very explicitly requires durable storage that supports fsync. You cannot run such workloads on transient disks, even if it appears to work at first, you will not be getting any of the data safety guarantees.

While some entities in 3.x can be made transient, starting with 4.x everything will be persistent.

RabbitMQ clusters on Kubernetes must use persistent volumes for node data directories, and stable hostnames on top of that (IP addresses can and will change). If you don't agree with that, please don't use RabbitMQ or do not run it on Kubernetes because it is guaranteed to not end well.

[CassieZAHNG]

Thanks for the detailed reply, since I am quite new to use RabbitMQ, so really want to get more understanding about it in case anything I am missing. In that case will use persistent volumes for sure and appreciate your help!