Node fails to start after reset if rabbitmq_delayed_message_exchange is installed

[trollshiro]

Describe the bug

On a fresh install of RMQ 4.2.0 if you run a standard

stop_app
reset
start_app

as if you want to remove the node from a cluster of whatever other reason, the app crashes with the following error

{:rabbit, {{:schema_integrity_check_failed, [table_missing: :rabbit_user, table_missing: :rabbit_user_permission, table_missing: :rabbit_runtime_parameters, table_missing: :rabbit_topic_permission, table_missing: :rabbit_vhost, table_missing: :rabbit_durable_queue, table_missing: :rabbit_queue, table_missing: :rabbit_durable_exchange, table_missing: :rabbit_exchange, table_missing: :rabbit_exchange_serial, table_missing: :rabbit_durable_route, table_missing: :rabbit_semi_durable_route, table_missing: :rabbit_route, table_missing: :rabbit_reverse_route, table_missing: :rabbit_index_route, table_missing: :rabbit_topic_trie_node, table_missing: :rabbit_topic_trie_edge, table_missing: :rabbit_topic_trie_binding, table_missing: :gm_group, table_missing: :mirrored_sup_childspec, table_missing: :rabbit_node_maintenance_states]}, {:rabbit, :start, [:normal, []]}}}

To make it work again you must remove the mnesia folder that should not be in use since khepri was in use

Reproduction steps

1. Install rabbitmq 4.2
2. Install and enable delay plugin 4.2
3. Run stop_app, reset, start_app
4. Observe the crash
   ...

Expected behavior

App should start normally using khepri, (maybe disabling the plugin if needed?)

Additional context

Tested on:
Debian 12.12
Rabbit 4.2.0
Erlang 27.3.4.2

Other plugin installed:
rabbitmq_top
rabbitmq_management
rabbitmq_management_agent
rabbitmq_prometheus
rabbitmq_web_dispatch

I was a bit unsure if open the bug here or on the delay plugin page but I think rabbit should not crash regardless of what could be wrong with the plugin. Looks like it somehow reverts back to mnesia...

[michaelklishin]

@trollshiro the "standard" stop_app, reset, start_app is no longer necessary, as the clustering docs explain.

Assuming that the "delay plugin" is rabbitmq_delayed_message_exchange one, you must have missed its release notes, that since 4.1.0 (and in 4.2.0) explicitly state:

This plugin is RabbitMQ 4.2.x compatible with the following limitations:

1. If the cluster uses Mnesia for schema store, it works exactly as it did against RabbitMQ 3.13.x
2. If the cluster uses Khepri and the plugin is enabled after Khepri, it will start Mnesia, set up a node-local Mnesia replica and schema, and works as in scenario 1
3. Important: if the cluster uses Mnesia, then the plugin is enabled, and then Khepri is enabled, the plugin must be disabled and re-enabled, or the node must be restarted. Then it will start Mnesia and works as in scenario 2

[michaelklishin]

Also, that "delay plugin" is no longer maintained by the core team. We will produce releases for some time but as the README of that plugin explains, that design is a dead end with serious limitations.

Tanzu RabbitMQ 4.3 will ship with a distributed delayed queue (a new queue type).

[michaelklishin]

think rabbit should not crash regardless of what could be wrong with the plugin. Looks like it somehow reverts back to mnesia..

Nodes fail to start when one of the plugins fail to start. The plugin you are relying on is entirely Mnesia-based, and Mnesia in RabbitMQ is going away in the next release.

When Khepri is enabled, depending on the state of the plugin you may or may not need to re-enable it, as the (plugin) release notes since 4.1.0 explain.

We do not have plans to allow nodes run with plugins that have failed to start.

[trollshiro]

Hi,
thanks for the explanation, just as a note the stop -> reset -> start is also mentioned in the doc about khepri/clustering here that's why I used it. I'll use the new simpler way now.

We have now enough info to survive until 4.3 is released, thanks again for your help.

[michaelklishin]

Understood. Here's a doc update that should eliminate the confusion: rabbitmq/rabbitmq-website@4f4378f.

Thank you.