rabbitmq cluster upgrade fails

[rajbaratht]

Hello - I'm trying to upgrade rabbitmq 3 node cluster using helm from version rabbitmq-8.31.5 to rabbitmq:12.0.1. However, we see the following error

rabbitmq 16:01:36.38 INFO  ==> ** Starting RabbitMQ **
2023-06-16 16:01:38.655744+00:00 [error] <0.235.0> Feature flags: `classic_mirrored_queue_version`: required feature flag not enabled! It must be enabled before upgrading RabbitMQ.
2023-06-16 16:01:38.666226+00:00 [error] <0.235.0> Failed to initialize feature flags registry: {disabled_required_feature_flag,
2023-06-16 16:01:38.666226+00:00 [error] <0.235.0>                                               classic_mirrored_queue_version}

BOOT FAILED
===========
Error during startup: {error,failed_to_initialize_feature_flags_registry}

2023-06-16 16:01:38.674981+00:00 [error] <0.235.0>
2023-06-16 16:01:38.674981+00:00 [error] <0.235.0> BOOT FAILED
2023-06-16 16:01:38.674981+00:00 [error] <0.235.0> ===========
2023-06-16 16:01:38.674981+00:00 [error] <0.235.0> Error during startup: {error,failed_to_initialize_feature_flags_registry}
2023-06-16 16:01:38.674981+00:00 [error] <0.235.0>
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>   crasher:
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>     initial call: application_master:init/4
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>     pid: <0.234.0>
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>     registered_name: []
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>     exception exit: {failed_to_initialize_feature_flags_registry,
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>                         {rabbit,start,[normal,[]]}}
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>       in function  application_master:init/4 (application_master.erl, line 142)
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>     ancestors: [<0.233.0>]
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>     message_queue_len: 1
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>     messages: [{'EXIT',<0.235.0>,normal}]
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>     links: [<0.233.0>,<0.44.0>]
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>     dictionary: []
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>     trap_exit: true
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>     status: running
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>     heap_size: 233
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>     stack_size: 28
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>     reductions: 168
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>   neighbours:
2023-06-16 16:01:39.676052+00:00 [error] <0.234.0>
2023-06-16 16:01:39.683565+00:00 [notice] <0.44.0> Application rabbit exited with reason: {failed_to_initialize_feature_flags_registry,{rabbit,start,[normal,[]]}}
Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{failed_to_initialize_feature_flags_registry,{rabbit,start,[normal,[]]}}})

Crash dump is being written to: erl_crash.dump...

I even tried to enable the feature and I get the same error. Any ideas on how to fix this error ?

[michaelklishin]

Those versions are not RabbitMQ versions. My guess is that they are Bitnami image versions.

See the Upgrade guide, in particular the RabbitMQ Version Upgradeability section.

Bitnami 12.x chart seems to ship RabbitMQ 3.12. By this logic, 8.x ships 3.8, which cannot be upgraded directly to 3.12, or even 3.11. Consider using Blue/Green deployment upgrade strategy or upgrade one major at a time.

[johnsxiong]

hi ,the same

[p0358]

Bruh, I used the official Debian images and this happened after upgrade

[p0358]

Yeah, but people don't expect an apt update to break everything, it doesn't happen with other software unless you explicitly change entry in the apt sources list to a newer major version at the very least. Just signalling it's a kinda bad UX, there should be some way to handle such upgrades automatically. (I'm using Cloudsmith apt repo)

Especially that the docs give "or just delete everything" as one of the solutions, even though that will lead to loss of defined users too (as I understand, but it's not mentioned). The export to avoid this requires a running RabbitMQ, which it's not due to the error etc etc

The thing is, if it's this way, the default instructions should recommend pinning it to particular major version, if upgrading it will fail without some manual action. And the repo, according to docs and its page, has just "main" channel instead.

So do I understand correctly that I now have to read apt logs to figure out what my old version was, then manually pin it to the next major release, and do such upgrades one-by-one until I reach the latest one?

[p0358]

Though trying to do that with apt install rabbitmq-server=3.11.15-1 yields erlang depedency errors, and they don't seem to be solvable with --fix-broken or by uninstalling erlang stuff earlier, even though older Erlang versions should be available on the repo, ehh

[p0358]

Okay it looks like I was able to solve my problem with:

apt-get remove rabbitmq-server erlang-base \
                        erlang-asn1 erlang-crypto erlang-eldap erlang-ftp erlang-inets \
                        erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key \
                        erlang-runtime-tools erlang-snmp erlang-ssl \
                        erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl

apt install --fix-broken rabbitmq-server=3.11.15-1 erlang-base=1:25.3.2.7-1 \
                        erlang-asn1=1:25.3.2.7-1 erlang-crypto=1:25.3.2.7-1 erlang-eldap=1:25.3.2.7-1 erlang-ftp=1:25.3.2.7-1 erlang-inets=1:25.3.2.7-1 \
                        erlang-mnesia=1:25.3.2.7-1 erlang-os-mon=1:25.3.2.7-1 erlang-parsetools=1:25.3.2.7-1 erlang-public-key=1:25.3.2.7-1 \
                        erlang-runtime-tools=1:25.3.2.7-1 erlang-snmp=1:25.3.2.7-1 erlang-ssl=1:25.3.2.7-1 \
                        erlang-syntax-tools=1:25.3.2.7-1 erlang-tftp=1:25.3.2.7-1 erlang-tools=1:25.3.2.7-1 erlang-xmerl=1:25.3.2.7-1

rabbitmqctl enable_feature_flag all

apt-get install rabbitmq-server erlang-base \
                        erlang-asn1 erlang-crypto erlang-eldap erlang-ftp erlang-inets \
                        erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key \
                        erlang-runtime-tools erlang-snmp erlang-ssl \
                        erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl

I think in Debian environment the expected way to handle breaking major updates within same repo channel is to have a news entry with urgency of at least medium that's then shown to admin upon upgrade, at least that's what the system does for its own packages, though not sure how easy/hard to implement...

I can't say my issue is solved though, because apparently in this whole ordeal the config/data was lost anyway... which is kinda a pity, as I don't remember the exact permissions I've set up on users back then :| It's probably because I shouldn't have used --purge (I forgot it purges configs and not dependencies xD)
nvm I restored a backup (forgot I have these), and did enable feature flags before update lol

well then my issue is solved, have a nice day all

edited my thing to remove --purge so nobody else stumbling upon this does that for themselves :D

[p0358]

Also I correct myself, I think actually the best course of action would be for the package to just forefully enable these features after upgrade instead of bailing out when service tries to start. Would there by any counter-arguments to do that?

[mkuratczyk]

We have an issue with some ideas for how to make feature flags more user friendly #9677.