Homebrew: node fails to boot, required feature flag not enabled

[amitbarat]

Describe the bug

After installing RabbitMQ - brew install rabbitmq on Mac, I tried to run it in the foreground - rabbitmq-server, but got the following error:

2023-12-24 14:16:55.673611+02:00 [error] <0.234.0> Feature flags: `stream_single_active_consumer`: required feature flag not enabled! It must be enabled before upgrading RabbitMQ.
2023-12-24 14:16:55.677879+02:00 [error] <0.234.0> Failed to initialize feature flags registry: {disabled_required_feature_flag,
2023-12-24 14:16:55.677879+02:00 [error] <0.234.0>                                               stream_single_active_consumer}

BOOT FAILED
===========
Error during startup: {error,failed_to_initialize_feature_flags_registry}

2023-12-24 14:16:55.681441+02:00 [error] <0.234.0> 
2023-12-24 14:16:55.681441+02:00 [error] <0.234.0> BOOT FAILED
2023-12-24 14:16:55.681441+02:00 [error] <0.234.0> ===========
2023-12-24 14:16:55.681441+02:00 [error] <0.234.0> Error during startup: {error,failed_to_initialize_feature_flags_registry}
2023-12-24 14:16:55.681441+02:00 [error] <0.234.0> 
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0>   crasher:
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0>     initial call: application_master:init/4
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0>     pid: <0.233.0>
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0>     registered_name: []
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0>     exception exit: {failed_to_initialize_feature_flags_registry,
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0>                         {rabbit,start,[normal,[]]}}
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0>       in function  application_master:init/4 (application_master.erl, line 142)
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0>     ancestors: [<0.232.0>]
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0>     message_queue_len: 1
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0>     messages: [{'EXIT',<0.234.0>,normal}]
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0>     links: [<0.232.0>,<0.44.0>]
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0>     dictionary: []
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0>     trap_exit: true
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0>     status: running
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0>     heap_size: 376
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0>     stack_size: 28
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0>     reductions: 173
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0>   neighbours:
2023-12-24 14:16:56.682103+02:00 [error] <0.233.0> 
2023-12-24 14:16:56.694773+02:00 [notice] <0.44.0> Application rabbit exited with reason: {failed_to_initialize_feature_flags_registry,{rabbit,start,[normal,[]]}}
Kernel pid terminated (application_controller) ("{application_start_failure,rabbit,{failed_to_initialize_feature_flags_registry,{rabbit,start,[normal,[]]}}}")

Crash dump is being written to: erl_crash.dump...done

I tried brew uninstall rabbitmq and deleting the /opt/homebrew/etc/rabbitmq before installing again (I might have had an older version installed in the past), as well as restarting the machine - but still no luck.
The installed version was 3.12.11.

Reproduction steps

1. brew install rabbitmq
2. rabbitmq-server

Expected behavior

RabbitMQ to run successfully

Additional context

No response

[michaelklishin]

You must enable all feature flags before upgrading to 3.12.x.
Otherwise 3.12 nodes will refuse to start and log a message.

In that case, with Homebrew you must uninstall the formula, delete any existing node data and then re-install it.
What's the data directory location depends on when Homebrew was installed.
Likely it is /opt/homebrew/etc/rabbitmq/ and /opt/homebrew/opt/rabbitmq/.

failed_to_initialize_feature_flags_registry, however, likely means that data directory is not writable by the node feature flag state cannot be stored. That's just a guess, there really aren't much to "initializing the registery".

[michaelklishin]

I haven't noticed the topmost log line at first:

Feature flags: `stream_single_active_consumer`: required feature flag not enabled! It must be enabled before upgrading RabbitMQ

confirms the hypothesis that this node didn't have all feature flags enabled at the time it was upgraded to 3.12.

[michaelklishin]

When you uninstall the formula, Homebrew visibly logs:

Warning: The following rabbitmq configuration files have not been removed!
If desired, remove them manually with `rm -rf`:
  /opt/homebrew/etc/rabbitmq
  /opt/homebrew/etc/rabbitmq/enabled_plugins
  /opt/homebrew/etc/rabbitmq/rabbitmq-env.conf
  /opt/homebrew/etc/rabbitmq/rabbitmq-env.conf.default

(your paths may vary), suggesting the formula authors expect these files to be manually removed
if the user decides so.

[amitbarat]

I uninstalled and deleted /opt/homebrew/etc/rabbitmq - what else should have been wiped?

[michaelklishin]

@amitbarat I have documented everything I could find in the Homebrew doc guide. I am not a Homebrew expert, you must wipe the node's data directory, that's it.

[lukebakken]

@amitbarat do you have any environment variables that start with RABBITMQ_ that may be relocating data?

The other suggestion I have is to forget using homebrew and just use the generic-unix package.

[amitbarat]

Alright, now I got it working - looks like the missing piece was this line from the Reinstalling the Formula section:
rm -rf /opt/homebrew/var/lib/rabbitmq/

The other ones mentioned there were already covered in previous attempts - I deleted /opt/homebrew/etc/rabbitmq/ manually, and /opt/homebrew/opt/rabbitmq/ was automatically deleted when uninstalling.
Once I deleted the var/lib folder too, I was able to start the service and enable the feature flags.

Thanks!

[michaelklishin]

@amitbarat one of the Homebrew maintainers does not consider this scenario to be an issue, you are welcome to make your case to them in the Homebrew core tap repo.

There are several options with various degrees of effectiveness but this person refuses to acknowledge it is an issue that specifically affects Homebrew users because there are no relevant post-installation steps.