bitnami rmq statefulset, messages backup and restore

[rahul-captionh]

Hi , I have been trying to backup the node directory and restore it on new server environment, but all my persistent messages are getting lost.

We are planning to upgrade RMQ, but the way our helm chart is configured, through argocd, upgrading one environment, ie, dev, sqa, uat, requires separation of the RMQ bitnami chart files to be separated. ArgoCD considers this as new installation and wipes of all the queue and messages.

I want to preserve the messages, we are able to restore queues through definitions but messages are not getting restored.

We have rmq 3.11.8 in one environment where we are testing it. But we are going from 3.10.9 to 3.11.8

I am scaling down RMQ stateful set to one replica and than stopping rmq , by running rabbitmqctl stop_app and then copy entire node directory to outside the pod.
Delete the environment through ArgoCD and recreate it. After recreation same server names come up , I import the definitions and than stop_app and restore the node directory from previous copy.

the moment I do start_app, all the messges are gone and they don't come up . I though this would work
any suggestions?

[michaelklishin]

We cannot suggest anything specific with this much information.

There can be all kinds of reasons, from node name changing between the two environments (which means the new node will start as a blank one, as if it never had any previous state) to your publishers publishing messages as transient which means RabbitMQ deletes them on next boot, as instructed by your applications.

When a node initializes a new database, it will log a relevant message:

2024-01-20 13:44:08.860319-05:00 [info] <0.234.0> Node database directory at /tmp/rmq.tmp is empty. Assuming we need to join an existing cluster or initialise from scratch...

and a few more messages about seeding the default data (unless Boot time definition import is used).

2024-01-20 13:44:08.913712-05:00 [info] <0.234.0> Will seed default virtual host and user...
2024-01-20 13:44:08.913770-05:00 [info] <0.234.0> Adding vhost '/' (description: 'Default virtual host', tags: [])
2024-01-20 13:44:08.914600-05:00 [info] <0.234.0> Inserted a virtual host record {vhost,<<"/">>,[],
2024-01-20 13:44:08.914600-05:00 [info] <0.234.0>                                       #{description =>
2024-01-20 13:44:08.914600-05:00 [info] <0.234.0>                                             <<"Default virtual host">>,
2024-01-20 13:44:08.914600-05:00 [info] <0.234.0>                                         tags => []}}
2024-01-20 13:44:08.919075-05:00 [info] <0.550.0> Making sure data directory '/tmp/rmq.tmp/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists

So start with inspecting node logs.

[rahul-captionh]

when node initializes original data directory is not there, I shutdown the app using rabbitmqctl stop_app and than restore the node directory and start the app

[rahul-captionh]

Node name does not change, as we are using the same namespace and node name for the cluster with helm charts

[michaelklishin]

RabbitMQ never removes its data directory (even when you reset a node, it removes most of the state there but never the directory itself).

This is a Kubernetes question, not a RabbitMQ one.

[rahul-captionh]

also we are publishing persistent messages and not transient