How to ensure "feature flag states written to disk"?

[inikulshin]

We have a script for RabbitMQ repair + upgrade.
One of the repair techniques when RabbitMQ fails to start, is to empty db (rename mnesia directory) and try to start RabbitMQ.

The flow (reproduction scenario) is:

00:18:57 TRC: rabbitmq_server-3.10.8\sbin: rabbitmq-service.bat stop {
00:19:01 INF: The RabbitMQ service is stopping.
00:19:01 INF: The RabbitMQ service was stopped successfully.
00:19:01 TRC: rabbitmq_server-3.10.8\sbin: rabbitmq-service.bat stop }
00:19:03 INF: Renaming db\rabbit@localhost-mnesia to rabbit@localhost-mnesia-20240124_001834
00:19:03 TRC: rabbitmq_server-3.10.8\sbin: rabbitmq-service.bat start {
00:19:07 INF: The RabbitMQ service is starting.
00:19:07 INF: The RabbitMQ service was started successfully.
00:19:07 TRC: rabbitmq_server-3.10.8\sbin: rabbitmq-service.bat start }
...
00:19:23 TRC: rabbitmq_server-3.10.8\sbin: rabbitmq-diagnostics.bat is_running {
00:19:25 TRC: rabbitmq_server-3.10.8\sbin: rabbitmq-diagnostics.bat is_running }
00:19:25 TRC: rabbitmq_server-3.10.8\sbin: rabbitmqctl.bat enable_feature_flag all {
00:19:27 INF: Enabling all feature flags ...
00:19:27 TRC: rabbitmq_server-3.10.8\sbin: rabbitmqctl.bat enable_feature_flag all }
00:19:32 TRC: rabbitmq_server-3.10.8\sbin: rabbitmqctl.bat stop_app {
00:19:39 INF: Stopping rabbit application on node rabbit@localhost ...
00:19:39 TRC: rabbitmq_server-3.10.8\sbin: rabbitmqctl.bat stop_app }
00:19:39 TRC: rabbitmq_server-3.10.8\sbin: rabbitmq-service.bat stop {
00:19:42 INF: The RabbitMQ service is stopping.
00:19:42 INF: The RabbitMQ service was stopped successfully.
00:19:42 TRC: rabbitmq_server-3.10.8\sbin: rabbitmq-service.bat stop }
00:19:48 TRC: : uninstall.exe /S {
00:19:55 TRC: : uninstall.exe /S }
00:19:57 TRC: setup: rabbitmq-server-3.11.26.exe /S /NOSTARTMENU /NOSERVICEINSTALL /NOSERVICERUN {
00:20:16 TRC: setup: rabbitmq-server-3.11.26.exe /S /NOSTARTMENU /NOSERVICEINSTALL /NOSERVICERUN }

And after successful installation rabbitmq-server-3.11.26 crashes on start with:

Slogan: Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{failed_to_initialize_feature_flags_registry,{rabbit,start,[normal,[]]}}})

I think it happens because RabbitMQ didn't wrote feature flag states to disk. There are no Feature flags: feature flag states written to disk: yes lines in [email protected] file, the lines I can see if I give RabbitMQ service more time to run between rabbitmqctl.bat enable_feature_flag all and rabbitmqctl.bat stop_app.

And now the question: how to ensure "feature flag states written to disk" and it's safe to stop RabbitMQ before uninstalling it?

• I think, rabbitmqctl.bat enable_feature_flag should not return success until feature flag states are written to disk.

[michaelklishin]

You cannot stop a node a few milliseconds after asking it to perform a data migration (which some feature flags will do). Give it some time to finish what it is doing before shutting the node down.

#9206, #10027, #9681 are somewhat relevant but ultimately, only a mechanism similar to pre-enabling plugins would really help avoid a fundamental race condition your script is creating:

• A node is asked to do X
• A node is asked to terminate immediately after
• Update: there is a third option listed below (for a very narrow set of use cases)

[michaelklishin]

Since feature flags are cluster-wide state, you cannot enable them the same way you enable plugins: nodes reach out to other nodes to see if this feature flag is even available there (in a mixed version cluster during an upgrade, that will not be the case).

So an enabled_feature_flags option is very unlikely to materialize, and scripts like the one above must wait for a few seconds before shutting the node down after enabling all feature flags.

[michaelklishin]

@dumbbell mentioned it to me that for a very narrow set of use cases, this is already possible.

Single node clusters and integration test environments (so, brand new clusters where none of the nodes have previous state) are the primary use case for that env var. We use it in RabbitMQ's own test suites, which are, of course, often really short lived compared to any real production installation.

But that might be an option for some, and comes with a warning. Use at your own risk.

[inikulshin]

@michaelklishin thanks, but "give it some time to finish" is not something that I can translate to the script language. I gave it 1 minute and nothing happened. 0 log lines, same crash after upgrade.

But the root cause of my problem was an old "repair by emptying db" routine that worked for RabbitMQ 3.8.x. I renamed db\rabbit@localhost-mnesia folder, but kept db\rabbit@localhost-feature_flags unchanged. After renaming whole db folder upgrade succeeded.

(Probably there should be a default location for rabbit@localhost-feature_flags file, e.g. RABBITMQ_FEATURE_FLAGS_FILE similarly to RABBITMQ_ENABLED_PLUGINS_FILE?).