Shovel plugin crash when upgrading from 3.9.8 to 3.10.17 #7294

brycechesternewman · 2023-02-14T17:02:10Z

brycechesternewman
Feb 14, 2023

Shovel plugin crash when upgrading from 3.9.8 (Erlang 24.1.2) to 3.10.17 (Erlang 24.3.4.7). There are only two nodes in the cluster. The shovel is stuck in restarting on both nodes after 3.9.8 nodes were removed from the cluster. Any ideas on why this occurred or how to prevent it? Thank you.

2023-02-14 17:00:34.772211+00:00 [warning] <0.5030.0> Recurring shovel spec clean up failed with exit:{noproc,
2023-02-14 17:00:34.772211+00:00 [warning] <0.5030.0>                                                  {gen_server,call,
2023-02-14 17:00:34.772211+00:00 [warning] <0.5030.0>                                                   [rabbit_shovel_dyn_worker_sup_sup,
2023-02-14 17:00:34.772211+00:00 [warning] <0.5030.0>                                                    which_children,infinity]}}

...

2023-02-14 17:02:43.501918+00:00 [error] <0.21281.2> Failed to restart shovel 'shovel-test-bryce' on vhost 'dsa', reason: {exception,
2023-02-14 17:02:43.501918+00:00 [error] <0.21281.2>                                                                       {noproc,
2023-02-14 17:02:43.501918+00:00 [error] <0.21281.2>                                                                        {gen_server,
2023-02-14 17:02:43.501918+00:00 [error] <0.21281.2>                                                                         call,
2023-02-14 17:02:43.501918+00:00 [error] <0.21281.2>                                                                         [rabbit_shovel_dyn_worker_sup_sup,
2023-02-14 17:02:43.501918+00:00 [error] <0.21281.2>                                                                          which_children,
2023-02-14 17:02:43.501918+00:00 [error] <0.21281.2>                                                                          infinity]}}}

...

023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>     supervisor: {<0.27047.0>,rabbit_shovel_dyn_worker_sup}
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>     errorContext: start_error
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>     reason: {badarg,
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                 [{erlang,binary_to_list,
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                      [{encrypted,
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                           <<"qkEcGM1zpgqBhhfvzK+ifL1xmTJAo+Ojr2q64KfDKhdDHb4BJwlozMySZkvCfULO1H73xK+D89YTNRzcNdhVe8cmeUjg2pf4+20S33t9PTA=">>}],
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                      [{error_info,#{module => erl_erts_errors}}]},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                  {rabbit_shovel_parameters,'-deobfuscated_uris/2-lc$^1/1-1-',
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                      1,
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                      [{file,"rabbit_shovel_parameters.erl"},{line,447}]},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                  {rabbit_shovel_parameters,parse_amqp091_source,1,
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                      [{file,"rabbit_shovel_parameters.erl"},{line,400}]},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                  {rabbit_shovel_parameters,parse,3,
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                      [{file,"rabbit_shovel_parameters.erl"},{line,268}]},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                  {rabbit_shovel_worker,init,1,
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                      [{file,"rabbit_shovel_worker.erl"},{line,50}]},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                  {gen_server2,init_it,6,[{file,"gen_server2.erl"},{line,565}]},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                  {proc_lib,init_p_do_apply,3,
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                      [{file,"proc_lib.erl"},{line,226}]}]}
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>     offender: [{pid,undefined},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                {id,{<<"dsa">>,<<"shovel-test-bryce">>}},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                {mfargs,
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                    {rabbit_shovel_worker,start_link,
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                        [dynamic,
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                         {<<"dsa">>,<<"shovel-test-bryce">>},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                         [{<<"dest-uri">>,
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                           [{encrypted,
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                                <<"CzD9Iu/l6FtMKj0II67BCm6cZCOtnI9UGrfAwm+SUTTHv6HG+ckHvwM8MDBx9ixhTUeICaXOn1vNo0ujh8N/e4+6HLzax8n6YN0mIFcZUPg=">>}]},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                          {<<"src-uri">>,
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                           [{encrypted,
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                                <<"qkEcGM1zpgqBhhfvzK+ifL1xmTJAo+Ojr2q64KfDKhdDHb4BJwlozMySZkvCfULO1H73xK+D89YTNRzcNdhVe8cmeUjg2pf4+20S33t9PTA=">>}]},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                          {<<"ack-mode">>,<<"on-confirm">>},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                          {<<"dest-add-forward-headers">>,false},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                          {<<"dest-protocol">>,<<"amqp091">>},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                          {<<"dest-queue">>,<<"tmp-bryce-sink-shovel">>},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                          {<<"src-delete-after">>,<<"never">>},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                          {<<"src-exchange">>,<<"dsa">>},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                          {<<"src-exchange-key">>,<<"me">>},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                          {<<"src-protocol">>,<<"amqp091">>}]]}},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                {restart_type,{permanent,5}},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                {shutdown,4294967295},
2023-02-14 16:33:45.707213+00:00 [error] <0.27047.0>                {child_type,worker}]
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>     supervisor: {local,rabbit_shovel_dyn_worker_sup_sup}
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>     errorContext: child_terminated
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>     reason: {shutdown,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>              [{{shutdown,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                 {failed_to_start_child,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                  {<<"dsa">>,<<"shovel-test-bryce">>},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                  {badarg,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                   [{erlang,binary_to_list,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                     [{encrypted,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                       <<"qkEcGM1zpgqBhhfvzK+ifL1xmTJAo+Ojr2q64KfDKhdDHb4BJwlozMySZkvCfULO1H73xK+D89YTNRzcNdhVe8cmeUjg2pf4+20S33t9PTA=">>}],
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                     [{error_info,#{module => erl_erts_errors}}]},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                    {rabbit_shovel_parameters,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                     '-deobfuscated_uris/2-lc$^1/1-1-',1,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                     [{file,"rabbit_shovel_parameters.erl"},{line,447}]},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                    {rabbit_shovel_parameters,parse_amqp091_source,1,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                     [{file,"rabbit_shovel_parameters.erl"},{line,400}]},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                    {rabbit_shovel_parameters,parse,3,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                     [{file,"rabbit_shovel_parameters.erl"},{line,268}]},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                    {rabbit_shovel_worker,init,1,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                     [{file,"rabbit_shovel_worker.erl"},{line,50}]},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                    {gen_server2,init_it,6,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                     [{file,"gen_server2.erl"},{line,565}]},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                    {proc_lib,init_p_do_apply,3,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                     [{file,"proc_lib.erl"},{line,226}]}]}}},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                {child,undefined,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                 {<<"dsa">>,<<"shovel-test-bryce">>},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                 {rabbit_shovel_dyn_worker_sup,start_link,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                  [{<<"dsa">>,<<"shovel-test-bryce">>},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                   [{<<"dest-uri">>,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                     [{encrypted,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                       <<"CzD9Iu/l6FtMKj0II67BCm6cZCOtnI9UGrfAwm+SUTTHv6HG+ckHvwM8MDBx9ixhTUeICaXOn1vNo0ujh8N/e4+6HLzax8n6YN0mIFcZUPg=">>}]},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                    {<<"src-uri">>,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                     [{encrypted,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                       <<"qkEcGM1zpgqBhhfvzK+ifL1xmTJAo+Ojr2q64KfDKhdDHb4BJwlozMySZkvCfULO1H73xK+D89YTNRzcNdhVe8cmeUjg2pf4+20S33t9PTA=">>}]},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                    {<<"ack-mode">>,<<"on-confirm">>},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                    {<<"dest-add-forward-headers">>,false},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                    {<<"dest-protocol">>,<<"amqp091">>},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                    {<<"dest-queue">>,<<"tmp-bryce-sink-shovel">>},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                    {<<"src-delete-after">>,<<"never">>},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                    {<<"src-exchange">>,<<"dsa">>},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                    {<<"src-exchange-key">>,<<"me">>},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                    {<<"src-protocol">>,<<"amqp091">>}]]},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                 transient,300000,worker,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                 [rabbit_shovel_dyn_worker_sup]}}]}
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>     offender: [{pid,<0.4715.0>},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                {id,mirroring},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                {mfargs,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                    {mirrored_supervisor,start_internal,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                        [rabbit_shovel_dyn_worker_sup_sup,
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                         fun rabbit_misc:execute_mnesia_transaction/1,[]]}},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                {restart_type,permanent},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                {shutdown,4294967295},
2023-02-14 16:33:45.708583+00:00 [error] <0.4713.0>                {child_type,worker}]
2023-02-14 16:33:45.709475+00:00 [error] <0.4713.0>     supervisor: {local,rabbit_shovel_dyn_worker_sup_sup}
2023-02-14 16:33:45.709475+00:00 [error] <0.4713.0>     errorContext: shutdown
2023-02-14 16:33:45.709475+00:00 [error] <0.4713.0>     reason: reached_max_restart_intensity
2023-02-14 16:33:45.709475+00:00 [error] <0.4713.0>     offender: [{pid,<0.4715.0>},
2023-02-14 16:33:45.709475+00:00 [error] <0.4713.0>                {id,mirroring},
2023-02-14 16:33:45.709475+00:00 [error] <0.4713.0>                {mfargs,
2023-02-14 16:33:45.709475+00:00 [error] <0.4713.0>                    {mirrored_supervisor,start_internal,
2023-02-14 16:33:45.709475+00:00 [error] <0.4713.0>                        [rabbit_shovel_dyn_worker_sup_sup,
2023-02-14 16:33:45.709475+00:00 [error] <0.4713.0>                         fun rabbit_misc:execute_mnesia_transaction/1,[]]}},
2023-02-14 16:33:45.709475+00:00 [error] <0.4713.0>                {restart_type,permanent},
2023-02-14 16:33:45.709475+00:00 [error] <0.4713.0>                {shutdown,4294967295},
2023-02-14 16:33:45.709475+00:00 [error] <0.4713.0>                {child_type,worker}]

michaelklishin · 2023-02-14T17:17:25Z

michaelklishin
Feb 14, 2023
Maintainer

I recall seeing something like this once. Re-creating the Shovel should address it. There have been changes in how credentials are encrypted by Shovels.

@lukebakken do you remember any details?

1 reply

brycechesternewman Feb 14, 2023
Author

Disabling and reenabling rabbitmq_shovel_management and rabbitmq_shovel resolved the issue. I needed to do this on both nodes.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Shovel plugin crash when upgrading from 3.9.8 to 3.10.17 #7294

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Shovel plugin crash when upgrading from 3.9.8 to 3.10.17 #7294

Uh oh!

Uh oh!

brycechesternewman Feb 14, 2023

Replies: 1 comment · 1 reply

Uh oh!

michaelklishin Feb 14, 2023 Maintainer

Uh oh!

brycechesternewman Feb 14, 2023 Author

brycechesternewman
Feb 14, 2023

Replies: 1 comment 1 reply

michaelklishin
Feb 14, 2023
Maintainer

brycechesternewman Feb 14, 2023
Author