disabling classic_queue_mirroring prevents rabbit from starting

[markusgust]

Hello!

We've been trying to set the deprecated_features.permit.classic_queue_mirroring = false in rabbitmq.conf to be able to enable the khepri_db feature flag quicker, but doing so looks to be preventing RMQ from booting. I guess this make sense since there are mirroring policies on the cluster, but I would expect that deprecating the feature would trump that?

On RabbitMQ version: 3.13.0-alpha.256

To reproduce, rabbitmq.conf only has one line:
deprecated_features.permit.classic_queue_mirroring = false

and I've created one classic queue, with a HA policy applied to it.

2023-11-01 02:43:28.905113+00:00 [error] <0.335.0> Classic mirrored queues are deprecated.
2023-11-01 02:43:28.905113+00:00 [error] <0.335.0> Their use is not permitted per the configuration (overriding the default, which is permitted):
2023-11-01 02:43:28.905113+00:00 [error] <0.335.0>     "deprecated_features.permit.classic_queue_mirroring = false"
2023-11-01 02:43:28.905113+00:00 [error] <0.335.0> Their use will not be permitted by default in the next minor RabbitMQ version (if any) and they will be removed from RabbitMQ 4.0.0.
2023-11-01 02:43:28.905113+00:00 [error] <0.335.0> To continue using classic mirrored queues when they are not permitted by default, set the following parameter in your configuration:
2023-11-01 02:43:28.905113+00:00 [error] <0.335.0>     "deprecated_features.permit.classic_queue_mirroring = true"
2023-11-01 02:43:28.909596+00:00 [notice] <0.44.0> Application mnesia exited with reason: stopped
2023-11-01 02:43:28.909828+00:00 [error] <0.250.0>
2023-11-01 02:43:28.909828+00:00 [error] <0.250.0> BOOT FAILED
2023-11-01 02:43:28.909828+00:00 [error] <0.250.0> ===========
2023-11-01 02:43:28.909828+00:00 [error] <0.250.0> Error during startup: {error,
2023-11-01 02:43:28.909828+00:00 [error] <0.250.0>                           {failed_to_deny_deprecated_features,
2023-11-01 02:43:28.909828+00:00 [error] <0.250.0>                               [classic_queue_mirroring]}}
2023-11-01 02:43:28.909828+00:00 [error] <0.250.0>
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>   crasher:
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>     initial call: application_master:init/4
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>     pid: <0.249.0>
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>     registered_name: []
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>     exception exit: {{failed_to_deny_deprecated_features,
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>                          [classic_queue_mirroring]},
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>                      {rabbit,start,[normal,[]]}}
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>       in function  application_master:init/4 (application_master.erl, line 142)
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>     ancestors: [<0.248.0>]
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>     message_queue_len: 1
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>     messages: [{'EXIT',<0.250.0>,normal}]
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>     links: [<0.248.0>,<0.44.0>]
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>     dictionary: []
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>     trap_exit: true
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>     status: running
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>     heap_size: 2586
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>     stack_size: 28
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>     reductions: 176
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>   neighbours:
2023-11-01 02:43:29.911328+00:00 [error] <0.249.0>
2023-11-01 02:43:29.918874+00:00 [notice] <0.44.0> Application rabbit exited with reason: {{failed_to_deny_deprecated_features,[classic_queue_mirroring]},{rabbit,start,[normal,[]]}}

I also can't find a reference to what the format would be for the advanced config file? Tried below to no avail:

  {deprecated_features, [
    {permit, [{classic_queue_mirroring, false }]}]},

Thanks!

[michaelklishin]

@markus812498 you are explicitly telling RabbitMQ to not permit a deprecated feature. See ac05652.

[markusgust]

yeah, maybe we're going at this the wrong way. The idea being that when migrating to khepri, we often get the error stating that we need to disable classic mirrored queues beforehand (which is fine and understandable). We were hoping that instead of manually disabling all policies with HA arguments, we can deny classic mirrored queues in config before migrating.

My understanding is classic mirror queues are deprecated, but permitted by default, and the idea of adding this config is to deny the usage, based on:

rabbitmq-server/deps/rabbit/src/rabbit_mirror_queue_misc.erl

Line 42 in 4eda2a8

-rabbit_deprecated_feature(

[markusgust]

once I remove current HA policy, I am able to add deprecated_features.permit.classic_queue_mirroring = false and boot rabbitmq with the feature denied, so it seems that it fails if there are existing HA policies. Maybe that's expected?

[michaelklishin]

This is expected: you tell RabbitMQ that you don't want to allow CMQs in this cluster. When it detects that there are CMQ policies, it refuses to start to make it very visible that the requirement you have opted in for was violated.

[michaelklishin]

"Deny the usage" means that nodes won't start if an explicitly disabled feature is used. Ignoring a feature would mean that you'd have to have monitoring in place since most users don't pay any attention to logs before it hits the fan.

In 4.0, classic mirrored queue policies won't have any effect. With 3.13, they can still be used but you can opt in to prohibit their usage, with very visible consequences.

If you cannot accept the consequences, do not disable the feature.

[markusgust]

ok, that's good to know, we'll figure out a way to accommodate. Thanks @michaelklishin

And I suspect if there's a minor version before 4.0 where CMQ will be denied by default, clusters upgrading to this minor version without disabling HA policies will get the same error (unless we explicitly permit usage in config)?

Just based on the comment below:

Their use will not be permitted by default in the next minor"
        "RabbitMQ version (if any) and they will be removed from "
        "RabbitMQ 4.0.0.\n"

link

[michaelklishin]

I cannot speculate on whether there will be a 3.14 release or whether CMQs will be disabled by default there. The error message tries to communicate that this feature is really on its way out, even though it won't be removed over night.

[markusgust]

ok I see, that's fair enough. We'll keep our eyes on it. Thanks!

[michaelklishin]

I don't see why anyone would configure this in advanced.config but as with all other rabbitmq.conf keys not provided by plugins, the mapping can be found in rabbit.schema.