4.0.7 nodes fail to boot with a 4.0.2 'rabbitmq_delayed_message_exchange' plugin

[bihla]

Describe the bug

When performing an rolling upgrade of RabbitMQ on Ubuntu 22.04 (erlang 1:27.2.4-1) from 4.0.6 to 4.0.7 the RabbitMQ server refuse to start: "Application rabbit exited with reason: {{invalid_boot_step_dependency,external_infrastructure,dat
abase,{bad_edge,[database,rabbit_registry,rabbit_delayed_message,external_infrastructure]}},{rabbit,start,[normal,[]]}}"

After downgrading it to 4.0.6 it starts without any issues.
Something seems related to "rabbitmq delayed message plugin" and RabbitMQ 4.0.7 in combination.

Reproduction steps

1. install RabbitMQ 4.0.6 in a 3 node cluster.
2. verify its working
3. install "rabbitmq delayed message plugin" version 4.0.2 on each node and restart
4. enable the plugin "rabbitmq-plugins enable rabbitmq_delayed_message_exchange"
5. verify its working
6. apt upgrade (or apt install rabbitmq-server=4.0.7)
7. start node (will fail and dump provided messages into log file)
8. apt install rabbitmq-server=4.0.6
9. start node (works again)

Expected behavior

RabbitMQ node will refuse to start and spew this into the log:

2025-03-04 09:23:27.463459+01:00 [error] <0.227.0> Exception during startup:
2025-03-04 09:23:27.463459+01:00 [error] <0.227.0> 
2025-03-04 09:23:27.463459+01:00 [error] <0.227.0> exit:{invalid_boot_step_dependency,external_infrastructure,database,{bad_edge,[database,rabbit_re
gistry,rabbit_delayed_message,external_infrastructure]}}
2025-03-04 09:23:27.463459+01:00 [error] <0.227.0> 
2025-03-04 09:23:27.463459+01:00 [error] <0.227.0>     rabbit_boot_steps:sort_boot_steps/1, line 102
2025-03-04 09:23:27.463459+01:00 [error] <0.227.0>     rabbit_boot_steps:find_steps/1, line 44
2025-03-04 09:23:27.463459+01:00 [error] <0.227.0>     rabbit_boot_steps:run_boot_steps/1, line 23
2025-03-04 09:23:27.463459+01:00 [error] <0.227.0>     rabbit:start/2, line 994
2025-03-04 09:23:27.463459+01:00 [error] <0.227.0>     application_master:start_it_old/4, line 295
2025-03-04 09:23:27.463459+01:00 [error] <0.227.0> 
2025-03-04 09:23:28.466616+01:00 [notice] <0.45.0> Application rabbit exited with reason: {{invalid_boot_step_dependency,external_infrastructure,dat
abase,{bad_edge,[database,rabbit_registry,rabbit_delayed_message,external_infrastructure]}},{rabbit,start,[normal,[]]}}
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>   crasher:
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>     initial call: application_master:init/3
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>     pid: <0.226.0>
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>     registered_name: []
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>     exception exit: {{invalid_boot_step_dependency,external_infrastructure,
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>                          database,
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>                          {bad_edge,
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>                              [database,rabbit_registry,rabbit_delayed_message,
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>                               external_infrastructure]}},
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>                      {rabbit,start,[normal,[]]}}
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>       in function  application_master:init/3 (application_master.erl, line 143)
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>     ancestors: [application_controller,<0.10.0>]
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>     message_queue_len: 1
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>     messages: [{'EXIT',<0.227.0>,normal}]
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>     links: [<0.45.0>]
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>     dictionary: []
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>     trap_exit: true
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>     status: running
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>     heap_size: 2586
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>     stack_size: 29
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>     reductions: 80
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0>   neighbours:
2025-03-04 09:23:28.468195+01:00 [error] <0.226.0> 
2025-03-04 09:23:28.568139+01:00 [notice] <0.84.0>     alarm_handler: {clear,system_memory_high_watermark}
2025-03-04 09:23:42.763915+01:00 [notice] <0.222.0> Logging: configured log handlers are now ACTIVE
2025-03-04 09:23:42.770724+01:00 [info] <0.222.0> ra: starting system quorum_queues

Additional context

No response

[gomoripeti]

you need to install a new version of the delayed message plugin for 4.0.7 https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/releases/tag/v4.0.7

see GitHub issues: rabbitmq/rabbitmq-delayed-message-exchange#298, #13381

[michaelklishin]

It was a bug in the plugin (a couple of boot steps badly needed adjustments for 2025), not RabbitMQ server but I'll leave it here for visibility.