Unable to Upgrade from Running Instance of 3.11.0 to 3.12.0 - single node - missing feature flag error (not found in documentation)

[sgoerres]

Describe the bug

Hi,

In total honesty I expected issues since I am running some upgrades starting from 3.8.9 and targeting 3.12.0...
However, the documentation did not state anything with regard to this feature flag being an issue so I am assuming this error report fits here.
In my opinion this is mainly a documentation thing but I might be wrong here and it's just me.

I am running a single node on a windows machine. Upgrade path in the past was mainly:

• Check correct environment settings and Erlang versions (install/update if needed)
• Check for major changes in the release logs of second digit changes and main page documentation
• Copy latest Release Zip file and extract (adjust Environment variables where applicable)
• Stop the old and run the new version (as a service after successful CLI start)

Before upgrading I checked the release logs (though not all third digit increments in thorough detail), I checked especially the feature_flags page https://www.rabbitmq.com/feature-flags.html since I identified this as my major hurdle in upgrading for my case.

Stated there is:
Feature Flag classic_queue_type_delivery_support is stable from 3.10.9 (so I would assume for it to be available in 3.11.0 as well. Did not check release timestamps though...)
Feature Flag stream_sac_coordinator_unblock_group is stable from 3.12.0 but it appears in 3.11.18 as well which is confusing when just looking at the main documentation.

I am aware that my upgrade path is not that common (and that I should have upgraded much earlier and more frequently).

If this is a non issue please close.

Reproduction steps

I started with 3.8.9, upgraded to 3.8.11. Then I enabled all feature flags.
I then went from there to 3.11.0 which started properly.
Then I enabled all feature flags again.
Then tried to start 3.12.0.
This cannot start with the error message that feature "classic_queue_type_delivery_support" was not enabled.
Went back and installed 3.11.18.
Now two more feature flags appeared: Aforementioned classic_queue_type_delivery_support and stream_sac_coordinator_unblock_group.
I enabled both feature flags and now 3.12.0 starts without issues.

PS: Somewhere during my upgrades the instance lost all configured logins as well. Since I cannot fully reproduce this I am unable to create a separate issue for this...

Expected behavior

Clear statement on the Rabbitmq page which versions introduced which feature flags would have helped a lot. I would have targeted/used correct versions as intermediate versions as well.
e.g.
classic_queue_type_delivery_support : 3.10.9 / 3.11.??

Additional context

No response

[michaelklishin]

Two days ago the docs were upgraded to explain that we expect 3.11 users to run the latest patch release before upgrading to 3.12.0. This is true for all release series. Sometimes new series introduces the kind of changes that require update to the previously released ones, and they ship in patch releases.

[michaelklishin]

There is a doc guide that lists feature flags. It's usually updated every so often, and feature flags are introduced whenever there is a need to retain mixed version cluster compatibility or safety.

When jumping multiple release series, it is usually a good idea to

• Consider a Blue/Green deployment upgrade
• Use the latest patch releases for every series

In other words, not going from 3.8.9 to 3.9.0 then 3.10.0 then 3.11.0 then 3.12.0
but from 3.8.9 to 3.8.35, then 3.9.29, then 3.10.24, then 3.11.18.

Sometimes jumping two series will work fine and sometimes it won't, e.g. because some feature flags
become required. Our pipelines only test upgrades from N-1 (the previous release series), usually involving a reasonably recent patch release. That is, for 3.12.0 it was 3.11.11 or so, not 3.11.0.

Every step of the way, you need to enable all feature flags once all nodes are updated. Feature
flags in RabbitMQ are primarily used as a safety mechanism during rolling upgrades, not to disable
what you don't use right now (that's what plugins are for).

Blue/Green deployments may seem complicated at first but it is much easier to upgrade that way from 3.8, or even 3.7, 3.6, 3.5. And the original cluster is still there should you need it.

[sgoerres]

Thanks for the clarification.
I will consider this in my future upgrades.