Cannot declare a queue '~s' on node '~s': ~255p rabbitmq:3.10.10

[faithfull]

`
Dear developer，i am using sharding plugin in rabbitmq 3.10.10,when i set the following policy

rabbitmqctl set_policy sharding-rule "sharding$" '{"shards-per-node":2}' --apply-to exchanges

then the node is failed to restart

by th way
3.8.x is fine， but 3.9.x and 3.10.x is fail to restart

Error log

2022-08-30 14:23:34.208021+00:00 [error] <0.231.0> BOOT FAILED
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0> ===========
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0> Exception during startup:
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0> 
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0> exit:{{{try_clause,{protocol_error,internal_error,"Cannot declare a queue '~s' on node '~s': ~255p",["queue 'sharding: shard.mcshardface - rabbit@bddplgeshot2 - 0' in vhost 'mcshardface'",rabbit@bddplgeshot2,{vhost_supervisor_not_running,<<"mcshardface">>}]}},[{rabbit_sharding_shard,declare_queue,4,[{file,"rabbit_sharding_shard.erl"},{line,87}]},{rabbit_sharding_shard,'-add_queues/1-lc$^0/1-0-',4,[{file,"rabbit_sharding_shard.erl"},{line,70}]},{rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',2,[{file,"rabbit_sharding_shard.erl"},{line,130}]},{rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',2,[{file,"rabbit_sharding_shard.erl"},{line,130}]},{rabbit_sharding_shard,ensure_sharded_queues,1,[{file,"rabbit_sharding_shard.erl"},{line,41}]},{rabbit_sharding_exchange_decorator,create,2,[{file,"rabbit_sharding_exchange_decorator.erl"},{line,39}]},{rabbit_exchange_decorator,'-maybe_recover/1-lc$^0/1-0-',2,[{file,"rabbit_exchange_decorator.erl"},{line,103}]},{rabbit_exchange_decorator,maybe_recover,1,[{file,"rabbit_exchange_decorator.erl"},{line,103}]}]},{gen_server,call,[rabbit_registry,{register,exchange_decorator,<<"sharding">>,rabbit_sharding_exchange_decorator},infinity]}}
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0> 
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0>     gen_server:call/3, line 382
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0>     rabbit_boot_steps:-run_step/2-lc$^0/1-0-/2, line 41
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0>     rabbit_boot_steps:run_step/2, line 46
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0>     rabbit_boot_steps:-run_boot_steps/1-lc$^0/1-0-/1, line 19
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0>     rabbit_boot_steps:run_boot_steps/1, line 20
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0>     rabbit:start/2, line 897
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0>     application_master:start_it_old/4, line 293
2022-08-30 14:23:34.208021+00:00 [error] <0.231.0> 
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>   crasher:
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     initial call: application_master:init/4
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     pid: <0.230.0>
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     registered_name: []
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     exception exit: {{{{try_clause,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                            {protocol_error,internal_error,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                                "Cannot declare a queue '~s' on node '~s': ~255p",
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                                ["queue 'sharding: shard.mcshardface - rabbit@bddplgeshot2 - 0' in vhost 'mcshardface'",
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                                 rabbit@bddplgeshot2,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                                 {vhost_supervisor_not_running,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                                     <<"mcshardface">>}]}},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                        [{rabbit_sharding_shard,declare_queue,4,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             [{file,"rabbit_sharding_shard.erl"},{line,87}]},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                         {rabbit_sharding_shard,'-add_queues/1-lc$^0/1-0-',4,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             [{file,"rabbit_sharding_shard.erl"},{line,70}]},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                         {rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             2,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             [{file,"rabbit_sharding_shard.erl"},{line,130}]},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                         {rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             2,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             [{file,"rabbit_sharding_shard.erl"},{line,130}]},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                         {rabbit_sharding_shard,ensure_sharded_queues,1,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             [{file,"rabbit_sharding_shard.erl"},{line,41}]},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                         {rabbit_sharding_exchange_decorator,create,2,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             [{file,"rabbit_sharding_exchange_decorator.erl"},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                              {line,39}]},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                         {rabbit_exchange_decorator,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             '-maybe_recover/1-lc$^0/1-0-',2,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             [{file,"rabbit_exchange_decorator.erl"},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                              {line,103}]},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                         {rabbit_exchange_decorator,maybe_recover,1,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                             [{file,"rabbit_exchange_decorator.erl"},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                              {line,103}]}]},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                       {gen_server,call,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                           [rabbit_registry,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                            {register,exchange_decorator,<<"sharding">>,
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                                rabbit_sharding_exchange_decorator},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                            infinity]}},
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>                      {rabbit,start,[normal,[]]}}
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>       in function  application_master:init/4 (application_master.erl, line 142)
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     ancestors: [<0.229.0>]
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     message_queue_len: 1
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     messages: [{'EXIT',<0.231.0>,normal}]
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     links: [<0.229.0>,<0.44.0>]
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     dictionary: []
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     trap_exit: true
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     status: running
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     heap_size: 2586
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     stack_size: 28
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>     reductions: 330
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0>   neighbours:
2022-08-30 14:23:35.210282+00:00 [error] <0.230.0> 
2022-08-30 14:23:35.212011+00:00 [notice] <0.44.0> Application rabbit exited with reason: {{{{try_clause,{protocol_error,internal_error,"Cannot declare a queue '~s' on node '~s': ~255p",["queue 'sharding: shard.mcshardface - rabbit@bddplgeshot2 - 0' in vhost 'mcshardface'",rabbit@bddplgeshot2,{vhost_supervisor_not_running,<<"mcshardface">>}]}},[{rabbit_sharding_shard,declare_queue,4,[{file,"rabbit_sharding_shard.erl"},{line,87}]},{rabbit_sharding_shard,'-add_queues/1-lc$^0/1-0-',4,[{file,"rabbit_sharding_shard.erl"},{line,70}]},{rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',2,[{file,"rabbit_sharding_shard.erl"},{line,130}]},{rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',2,[{file,"rabbit_sharding_shard.erl"},{line,130}]},{rabbit_sharding_shard,ensure_sharded_queues,1,[{file,"rabbit_sharding_shard.erl"},{line,41}]},{rabbit_sharding_exchange_decorator,create,2,[{file,"rabbit_sharding_exchange_decorator.erl"},{line,39}]},{rabbit_exchange_decorator,'-maybe_recover/1-lc$^0/1-0-',2,[{file,"rabbit_exchange_decorator.erl"},{line,103}]},{rabbit_exchange_decorator,maybe_recover,1,[{file,"rabbit_exchange_decorator.erl"},{line,103}]}]},{gen_server,call,[rabbit_registry,{register,exchange_decorator,<<"sharding">>,rabbit_sharding_exchange_decorator},infinity]}},{rabbit,start,[normal,[]]}}

[lukebakken]

Please provide all commands that you run after starting a fresh 3.10.10 cluster. You have only provided one command which is insufficient for us to reproduce this error.

For instance, you are probably enabling the plugin, creating a cluster, creating queues, and doing other steps prior to restarting the node.

[faithfull]

hello developer， after create a fresh 3.10.10 cluster（3 node use helm bitnami/rabbitmq） ,you can do the follow thing to reproduce
1
create vhost test
create user test and password is test2022

2
curl -i -u test:test2022 -H "content-type:application/json" -XPUT -d'{"type":"x-modulus-hash","durable":true}' http://localhost:15672/api/exchanges/test/scst.sharding

3
rabbitmqctl set_policy -p test scst.sharding "^scst.sharding$" '{"shards-per-node": 3}'

4
restart one of the nodes and you can see the errors

[lukebakken]

I can't reproduce the issue given the above steps using 3.10. Restarting a node went fine.

NOTE after I created the test user, I ensured that user has permissions to the test vhost. My guess is that you are not giving that user any permissions, and thus when it tries to redeclare a queue it can't.

[faithfull]

i have already set the user has permissions to the test vhost, and when i restart one node , wo still have the following error,fail to start.

Error log

2022-12-05 00:05:20.714868+08:00 [error] <0.223.0> BOOT FAILED
2022-12-05 00:05:20.714868+08:00 [error] <0.223.0> ===========
2022-12-05 00:05:20.714868+08:00 [error] <0.223.0> Exception during startup:
2022-12-05 00:05:20.714868+08:00 [error] <0.223.0>
2022-12-05 00:05:20.714868+08:00 [error] <0.223.0> exit:{{{try_clause,{protocol_error,internal_error,"Cannot declare a queue '~s' on node '~s': ~255p",["queue 'sharding: scst.sharding - rabbit@rabbitmq-ad-1.rabbitmq-ad-headless.rabbitmq.svc.cluster.local - 0' in vhost 'test'",'rabbit@rabbitmq-ad-1.rabbitmq-ad-headless.rabbitmq.svc.cluster.local',{vhost_supervisor_not_running,<<"test">>}]}},[{rabbit_sharding_shard,declare_queue,4,[{file,"rabbit_sharding_shard.erl"},{line,87}]},{rabbit_sharding_shard,'-add_queues/1-lc$^0/1-0-',4,[{file,"rabbit_sharding_shard.erl"},{line,70}]},{rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',2,[{file,"rabbit_sharding_shard.erl"},{line,130}]},{rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',2,[{file,"rabbit_sharding_shard.erl"},{line,130}]},{rabbit_sharding_shard,ensure_sharded_queues,1,[{file,"rabbit_sharding_shard.erl"},{line,41}]},{rabbit_sharding_exchange_decorator,create,2,[{file,"rabbit_sharding_exchange_decorator.erl"},{line,39}]},{rabbit_exchange_decorator,'-maybe_recover/1-lc$^0/1-0-',2,[{file,"rabbit_exchange_decorator.erl"},{line,103}]},{rabbit_exchange_decorator,maybe_recover,1,[{file,"rabbit_exchange_decorator.erl"},{line,103}]}]},{gen_server,call,[rabbit_registry,{register,exchange_decorator,<<"sharding">>,rabbit_sharding_exchange_decorator},infinity]}}
2022-12-05 00:05:20.714868+08:00 [error] <0.223.0>
2022-12-05 00:05:20.714868+08:00 [error] <0.223.0>     gen_server:call/3, line 385
2022-12-05 00:05:20.714868+08:00 [error] <0.223.0>     rabbit_boot_steps:-run_step/2-lc$^0/1-0-/2, line 41
2022-12-05 00:05:20.714868+08:00 [error] <0.223.0>     rabbit_boot_steps:run_step/2, line 48
2022-12-05 00:05:20.714868+08:00 [error] <0.223.0>     rabbit_boot_steps:-run_boot_steps/1-lc$^0/1-0-/1, line 19
2022-12-05 00:05:20.714868+08:00 [error] <0.223.0>     rabbit_boot_steps:run_boot_steps/1, line 20
2022-12-05 00:05:20.714868+08:00 [error] <0.223.0>     rabbit:start/2, line 906
2022-12-05 00:05:20.714868+08:00 [error] <0.223.0>     application_master:start_it_old/4, line 293
2022-12-05 00:05:20.714868+08:00 [error] <0.223.0>
2022-12-05 00:05:20.717360+08:00 [info] <0.497.0> Management plugin: to stop collect_statistics.
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0> Set stop reason to: {error,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                         {{{try_clause,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                               {protocol_error,internal_error,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                   "Cannot declare a queue '~s' on node '~s': ~255p",
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                   ["queue 'sharding: scst.sharding - rabbit@rabbitmq-ad-1.rabbitmq-ad-headless.rabbitmq.svc.cluster.local - 0' in vhost 'test'",
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                    'rabbit@rabbitmq-ad-1.rabbitmq-ad-headless.rabbitmq.svc.cluster.local',
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                    {vhost_supervisor_not_running,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                        <<"test">>}]}},
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                           [{rabbit_sharding_shard,declare_queue,4,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                [{file,"rabbit_sharding_shard.erl"},{line,87}]},
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                            {rabbit_sharding_shard,'-add_queues/1-lc$^0/1-0-',
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                4,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                [{file,"rabbit_sharding_shard.erl"},{line,70}]},
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                            {rabbit_sharding_shard,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                '-foreach_node/1-lc$^0/1-0-',2,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                [{file,"rabbit_sharding_shard.erl"},
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                 {line,130}]},
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                            {rabbit_sharding_shard,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                '-foreach_node/1-lc$^0/1-0-',2,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                [{file,"rabbit_sharding_shard.erl"},
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                 {line,130}]},
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                            {rabbit_sharding_shard,ensure_sharded_queues,1,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                [{file,"rabbit_sharding_shard.erl"},{line,41}]},
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                            {rabbit_sharding_exchange_decorator,create,2,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                [{file,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                     "rabbit_sharding_exchange_decorator.erl"},
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                 {line,39}]},
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                            {rabbit_exchange_decorator,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                '-maybe_recover/1-lc$^0/1-0-',2,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                [{file,"rabbit_exchange_decorator.erl"},
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                 {line,103}]},
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                            {rabbit_exchange_decorator,maybe_recover,1,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                [{file,"rabbit_exchange_decorator.erl"},
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                 {line,103}]}]},
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                          {gen_server,call,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                              [rabbit_registry,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                               {register,exchange_decorator,<<"sharding">>,
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                                   rabbit_sharding_exchange_decorator},
2022-12-05 00:05:21.716688+08:00 [debug] <0.223.0>                               infinity]}}}
2022-12-05 00:05:21.717557+08:00 [debug] <0.223.0> Change boot state to `stopped`
2022-12-05 00:05:21.717935+08:00 [debug] <0.134.0> Boot state/systemd: sending non-systemd state (stopped) as status description: "Standing by"
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>   crasher:
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>     initial call: application_master:init/4
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>     pid: <0.222.0>
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>     registered_name: []
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>     exception exit: {{{{try_clause,
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                            {protocol_error,internal_error,
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                                "Cannot declare a queue '~s' on node '~s': ~255p",
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                                ["queue 'sharding: scst.sharding - rabbit@rabbitmq-ad-1.rabbitmq-ad-headless.rabbitmq.svc.cluster.local - 0' in vhost 'test'",
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                                 'rabbit@rabbitmq-ad-1.rabbitmq-ad-headless.rabbitmq.svc.cluster.local',
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                                 {vhost_supervisor_not_running,<<"test">>}]}},
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                        [{rabbit_sharding_shard,declare_queue,4,
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                             [{file,"rabbit_sharding_shard.erl"},{line,87}]},
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                         {rabbit_sharding_shard,'-add_queues/1-lc$^0/1-0-',4,
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                             [{file,"rabbit_sharding_shard.erl"},{line,70}]},
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                         {rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                             2,
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                             [{file,"rabbit_sharding_shard.erl"},{line,130}]},
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                         {rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                             2,
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                             [{file,"rabbit_sharding_shard.erl"},{line,130}]},
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                         {rabbit_sharding_shard,ensure_sharded_queues,1,
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                             [{file,"rabbit_sharding_shard.erl"},{line,41}]},
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                         {rabbit_sharding_exchange_decorator,create,2,
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                             [{file,"rabbit_sharding_exchange_decorator.erl"},
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                              {line,39}]},
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                         {rabbit_exchange_decorator,
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                             '-maybe_recover/1-lc$^0/1-0-',2,
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                             [{file,"rabbit_exchange_decorator.erl"},
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                              {line,103}]},
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                         {rabbit_exchange_decorator,maybe_recover,1,
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                             [{file,"rabbit_exchange_decorator.erl"},
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                              {line,103}]}]},
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                       {gen_server,call,
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                           [rabbit_registry,
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                            {register,exchange_decorator,<<"sharding">>,
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                                rabbit_sharding_exchange_decorator},
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                            infinity]}},
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>                      {rabbit,start,[normal,[]]}}
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>       in function  application_master:init/4 (application_master.erl, line 142)
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>     ancestors: [<0.221.0>]
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>     message_queue_len: 1
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>     messages: [{'EXIT',<0.223.0>,normal}]
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>     links: [<0.221.0>,<0.44.0>]
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>     dictionary: []
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>     trap_exit: true
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>     status: running
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>     heap_size: 4185
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>     stack_size: 28
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>     reductions: 273
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>   neighbours:
2022-12-05 00:05:21.718312+08:00 [error] <0.222.0>
2022-12-05 00:05:21.719870+08:00 [notice] <0.44.0> Application rabbit exited with reason: {{{{try_clause,{protocol_error,internal_error,"Cannot declare a queue '~s' on node '~s': ~255p",["queue 'sharding: scst.sharding - rabbit@rabbitmq-ad-1.rabbitmq-ad-headless.rabbitmq.svc.cluster.local - 0' in vhost 'test'",'rabbit@rabbitmq-ad-1.rabbitmq-ad-headless.rabbitmq.svc.cluster.local',{vhost_supervisor_not_running,<<"test">>}]}},[{rabbit_sharding_shard,declare_queue,4,[{file,"rabbit_sharding_shard.erl"},{line,87}]},{rabbit_sharding_shard,'-add_queues/1-lc$^0/1-0-',4,[{file,"rabbit_sharding_shard.erl"},{line,70}]},{rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',2,[{file,"rabbit_sharding_shard.erl"},{line,130}]},{rabbit_sharding_shard,'-foreach_node/1-lc$^0/1-0-',2,[{file,"rabbit_sharding_shard.erl"},{line,130}]},{rabbit_sharding_shard,ensure_sharded_queues,1,[{file,"rabbit_sharding_shard.erl"},{line,41}]},{rabbit_sharding_exchange_decorator,create,2,[{file,"rabbit_sharding_exchange_decorator.erl"},{line,39}]},{rabbit_exchange_decorator,'-maybe_recover/1-lc$^0/1-0-',2,[{file,"rabbit_exchange_decorator.erl"},{line,103}]},{rabbit_exchange_decorator,maybe_recover,1,[{file,"rabbit_exchange_decorator.erl"},{line,103}]}]},{gen_server,call,[rabbit_registry,{register,exchange_decorator,<<"sharding">>,rabbit_sharding_exchange_decorator},infinity]}},{rabbit,start,[normal,[]]}}
2022-12-05 00:05:21.720649+08:00 [debug] <0.44.0> Running rabbit_prelaunch:shutdown_func() as part of `kernel` shutdown
2022-12-05 00:05:21.720737+08:00 [debug] <0.44.0> Deleting PID file: /bitnami/rabbitmq/mnesia/rabbit@rabbitmq-ad-1.rabbitmq-ad-headless.rabbitmq.svc.cluster.local.pid

This problem ,i find some user had memtion one years before:
#3415

[lukebakken]

@faithfull I have been able to use a project and instructions provided by @reichowc here to reproduce this issue (my Docker Compose project).

I will open a new issue soon with more details and will link to it here.

[lukebakken]

cc @reichowc @faithfull @gaima8 thanks for your patience. I have found and fixed the issue -

• Issue - rabbitmq_sharding plugin race condition on node restart #6582
• Fix - Modify rabbit_sharding_exchange_decorator boot requirements #6583

[faithfull]

I appreciate you help