Problems with Sharding Plugin in RabbitMQ version 3.9.5

[reichowc]

Hello, I'm using sharding plugin in RabbitMQ version 3.8.18 without problems, but, when I change to version 3.9.5 the following problem happens: When one node (RabbitMQ) is down, I can't get it up again if I have some policy created.

So, if I delete the policy and try to start the node again, this is possible. Then I need to recreate the policy with the node up to make things work again.

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

We cannot suggest anything with this much information. Consider sharing an executable way to reproduce and node logs. The sharing plugin hasn't changed at all in 3.9, here is the entire diff:

diff --git a/deps/rabbitmq_sharding/src/rabbit_sharding_shard.erl b/deps/rabbitmq_sharding/src/rabbit_sharding_shard.erl
index 20a0df66fb..16a4d1ccd7 100644
--- a/deps/rabbitmq_sharding/src/rabbit_sharding_shard.erl
+++ b/deps/rabbitmq_sharding/src/rabbit_sharding_shard.erl
@@ -90,8 +90,8 @@ declare_queue(XName, Durable, N, Node) ->
             ok
     catch
         _Error:Reason ->
-            _ = rabbit_log:error("sharding failed to declare queue for exchange ~p"
-                             " - soft error:~n~p~n",
+            rabbit_log:error("sharding failed to declare queue for exchange ~p"
+                             " - soft error:~n~p",
                              [exchange_bin(XName), Reason]),
             error
     end.
@@ -119,7 +119,7 @@ binding_action(F, XName, RoutingKey, N, Node, ErrMsg) ->
            ?SHARDING_USER) of
         ok              -> ok;
         {error, Reason} ->
-            _ = rabbit_log:error(ErrMsg, [QBin, exchange_bin(XName), Reason]),
+            rabbit_log:error(ErrMsg, [QBin, exchange_bin(XName), Reason]),
             error
     end.

Try it for yourself:

git diff v3.8.x..v3.9.x -- deps/rabbitmq_sharding/src

[reichowc]

The node error is:

2021-10-05 20:29:45.407081+00:00 [noti] <0.44.0> Application rabbit exited with reason: {{{{try_clause,{protocol_error,internal_error,"Cannot declare a queue '~s' on node '~s': ~255p",["queue 'sharding: testing - rabbit@rabbit2 - 0' in vhost '/'",rabbit@rabbit2,{vhost_supervisor_not_running,<<"/">>}]}},[{rabbit_sharding_shard,declare_queue,4,[{file,"src/rabbit_sharding_shard.erl"},{line,87}]},{rabbit_sharding_shard,'-add_queues/1-lc$^0/1-0-',4,[{file,"src/rabbit_sharding_shard.erl"},{line,70}]},{rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',2,[{file,"src/rabbit_sharding_shard.erl"},{line,130}]},{rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',2,[{file,"src/rabbit_sharding_shard.erl"},{line,130}]},{rabbit_sharding_shard,ensure_sharded_queues,1,[{file,"src/rabbit_sharding_shard.erl"},{line,41}]},{rabbit_sharding_exchange_decorator,create,2,[{file,"src/rabbit_sharding_exchange_decorator.erl"},{line,39}]},{rabbit_exchange_decorator,'-maybe_recover/1-lc$^0/1-0-',2,[{file,"src/rabbit_exchange_decorator.erl"},{line,103}]},{rabbit_exchange_decorator,maybe_recover,1,[{file,"src/rabbit_exchange_decorator.erl"},{line,103}]}]},{gen_server,call,[rabbit_registry,{register,exchange_decorator,<<"sharding">>,rabbit_sharding_exchange_decorator},infinity]}},{rabbit,start,[normal,[]]}}

I created an application with a step-by-step to reproduce it:

https://github.com/reichowc/rabbitmq-sharding-plugin

[reichowc]

Can someone support with it?

[gaima8]

We've also been seeing this problem.
Originally we tried setting up a RabbitMQ 3.10 cluster and that failed with this error during setup. A RabbitMQ 3.9 cluster provisioned first time without issue.
Today after some server reboots a 3.9 sharded cluster has partially failed.

Error log

2022-08-30 14:23:34.208021+00:00 [error] <0.231.0> BOOT FAILED
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0> ===========
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0> Exception during startup:
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0> 
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0> exit:{{{try_clause,{protocol_error,internal_error,"Cannot declare a queue '~s' on node '~s': ~255p",["queue 'sharding: shard.mcshardface - rabbit@bddplgeshot2 - 0' in vhost 'mcshardface'",rabbit@bddplgeshot2,{vhost_supervisor_not_running,<<"mcshardface">>}]}},[{rabbit_sharding_shard,declare_queue,4,[{file,"rabbit_sharding_shard.erl"},{line,87}]},{rabbit_sharding_shard,'-add_queues/1-lc$^0/1-0-',4,[{file,"rabbit_sharding_shard.erl"},{line,70}]},{rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',2,[{file,"rabbit_sharding_shard.erl"},{line,130}]},{rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',2,[{file,"rabbit_sharding_shard.erl"},{line,130}]},{rabbit_sharding_shard,ensure_sharded_queues,1,[{file,"rabbit_sharding_shard.erl"},{line,41}]},{rabbit_sharding_exchange_decorator,create,2,[{file,"rabbit_sharding_exchange_decorator.erl"},{line,39}]},{rabbit_exchange_decorator,'-maybe_recover/1-lc$^0/1-0-',2,[{file,"rabbit_exchange_decorator.erl"},{line,103}]},{rabbit_exchange_decorator,maybe_recover,1,[{file,"rabbit_exchange_decorator.erl"},{line,103}]}]},{gen_server,call,[rabbit_registry,{register,exchange_decorator,<<"sharding">>,rabbit_sharding_exchange_decorator},infinity]}}
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0> 
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0>     gen_server:call/3, line 382
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0>     rabbit_boot_steps:-run_step/2-lc$^0/1-0-/2, line 41
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0>     rabbit_boot_steps:run_step/2, line 46
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0>     rabbit_boot_steps:-run_boot_steps/1-lc$^0/1-0-/1, line 19
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0>     rabbit_boot_steps:run_boot_steps/1, line 20
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0>     rabbit:start/2, line 897
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0>     application_master:start_it_old/4, line 293
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0> 
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>   crasher:
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     initial call: application_master:init/4
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     pid: <0.230.0>
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     registered_name: []
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     exception exit: {{{{try_clause,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                            {protocol_error,internal_error,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                                "Cannot declare a queue '~s' on node '~s': ~255p",
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                                ["queue 'sharding: shard.mcshardface - rabbit@bddplgeshot2 - 0' in vhost 'mcshardface'",
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                                 rabbit@bddplgeshot2,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                                 {vhost_supervisor_not_running,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                                     <<"mcshardface">>}]}},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                        [{rabbit_sharding_shard,declare_queue,4,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             [{file,"rabbit_sharding_shard.erl"},{line,87}]},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                         {rabbit_sharding_shard,'-add_queues/1-lc$^0/1-0-',4,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             [{file,"rabbit_sharding_shard.erl"},{line,70}]},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                         {rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             2,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             [{file,"rabbit_sharding_shard.erl"},{line,130}]},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                         {rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             2,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             [{file,"rabbit_sharding_shard.erl"},{line,130}]},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                         {rabbit_sharding_shard,ensure_sharded_queues,1,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             [{file,"rabbit_sharding_shard.erl"},{line,41}]},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                         {rabbit_sharding_exchange_decorator,create,2,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             [{file,"rabbit_sharding_exchange_decorator.erl"},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                              {line,39}]},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                         {rabbit_exchange_decorator,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             '-maybe_recover/1-lc$^0/1-0-',2,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             [{file,"rabbit_exchange_decorator.erl"},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                              {line,103}]},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                         {rabbit_exchange_decorator,maybe_recover,1,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             [{file,"rabbit_exchange_decorator.erl"},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                              {line,103}]}]},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                       {gen_server,call,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                           [rabbit_registry,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                            {register,exchange_decorator,<<"sharding">>,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                                rabbit_sharding_exchange_decorator},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                            infinity]}},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                      {rabbit,start,[normal,[]]}}
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>       in function  application_master:init/4 (application_master.erl, line 142)
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     ancestors: [<0.229.0>]
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     message_queue_len: 1
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     messages: [{'EXIT',<0.231.0>,normal}]
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     links: [<0.229.0>,<0.44.0>]
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     dictionary: []
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     trap_exit: true
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     status: running
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     heap_size: 2586
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     stack_size: 28
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     reductions: 330
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>   neighbours:
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0> 
2022-08-30 14:23:35.212011+00:00 [notice] <0.44.0> Application rabbit exited with reason: {{{{try_clause,{protocol_error,internal_error,"Cannot declare a queue '~s' on node '~s': ~255p",["queue 'sharding: shard.mcshardface - rabbit@bddplgeshot2 - 0' in vhost 'mcshardface'",rabbit@bddplgeshot2,{vhost_supervisor_not_running,<<"mcshardface">>}]}},[{rabbit_sharding_shard,declare_queue,4,[{file,"rabbit_sharding_shard.erl"},{line,87}]},{rabbit_sharding_shard,'-add_queues/1-lc$^0/1-0-',4,[{file,"rabbit_sharding_shard.erl"},{line,70}]},{rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',2,[{file,"rabbit_sharding_shard.erl"},{line,130}]},{rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',2,[{file,"rabbit_sharding_shard.erl"},{line,130}]},{rabbit_sharding_shard,ensure_sharded_queues,1,[{file,"rabbit_sharding_shard.erl"},{line,41}]},{rabbit_sharding_exchange_decorator,create,2,[{file,"rabbit_sharding_exchange_decorator.erl"},{line,39}]},{rabbit_exchange_decorator,'-maybe_recover/1-lc$^0/1-0-',2,[{file,"rabbit_exchange_decorator.erl"},{line,103}]},{rabbit_exchange_decorator,maybe_recover,1,[{file,"rabbit_exchange_decorator.erl"},{line,103}]}]},{gen_server,call,[rabbit_registry,{register,exchange_decorator,<<"sharding">>,rabbit_sharding_exchange_decorator},infinity]}},{rabbit,start,[normal,[]]}}

Our 3.8.29 cluster is completely fine, and has been through multiple point release upgrades.

[faithfull]

i also have this problem， 3.8.x is fine， but 3.9.x and 3.10.x is fail to start

[faithfull]

@michaelklishin hello , it 's very easy to reproduce it. when the rabbitmq cluster set the policy about exchange .

for example:
rabbitmqctl set_policy sharding-rule "sharding$" '{"shards-per-node":2}' --apply-to exchanges

then the node is failed to restart

[lukebakken]

@reichowc @faithfull @gaima8 I finally have been able to reproduce this issue. Please see #6429 where I will continue discussion.