Сompletion of shovel with the status terminated and endless errors in the logs

[sbulyigin]

1. The shovel is configured according to these settings from the center, however, it connects and goes to terminated.
   The other shovel from the center normally connects to us and works. What's wrong?
   Our shovel is configured by adding a config advanced.config.
   

2. Tons of errors of this kind are pouring in, why and what to do with them?

2022-05-16 09:37:19.396147+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396147+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396151+04:00 [error] <0.225.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2692626030.1821376515.175644>}} from <0.225.0> to <0.6835.17> in an old incarnation (1652678813) of this node (1652679434) 2022-05-16 09:37:19.396151+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396151+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396161+04:00 [error] <0.225.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2692626030.1821376515.175644>}} from <0.225.0> to <0.6838.17> in an old incarnation (1652678813) of this node (1652679434) 2022-05-16 09:37:19.396161+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396161+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396167+04:00 [error] <0.225.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2692626030.1821376515.175644>}} from <0.225.0> to <0.6823.17> in an old incarnation (1652678813) of this node (1652679434) 2022-05-16 09:37:19.396167+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396167+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396176+04:00 [error] <0.225.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2692626030.1821376515.175644>}} from <0.225.0> to <0.6826.17> in an old incarnation (1652678813) of this node (165267 ![image_2022-05-23_11-57-49](https://user-images.githubusercontent.com/106066658/169790714-441c32ff-a215-4d31-b0ef-0049719d227a.png) ![image_2022-05-23_11-57-48](https://user-images.githubusercontent.com/106066658/169790719-2b037008-7ed9-4919-ad56-13a27693ebe6.png) 9434) 2022-05-16 09:37:19.396176+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396176+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396181+04:00 [error] <0.225.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2692626030.1821376515.175644>}} from <0.225.0> to <0.6829.17> in an old incarnation (1652678813) of this node (1652679434) 2022-05-16 09:37:19.396181+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396181+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396186+04:00 [error] <0.225.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2692626030.1821376515.175644>}} from <0.225.0> to <0.6820.17> in an old incarnation (1652678813) of this node (1652679434) 2022-05-16 09:37:19.396186+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396186+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396192+04:00 [error] <0.225.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2692626030.1821376515.175644>}} from <0.225.0> to <0.6808.17> in an old incarnation (1652678813) of this node (1652679434) 2022-05-16 09:37:19.396192+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396192+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396201+04:00 [error] <0.225.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2692626030.1821376515.175644>}} from <0.225.0> to <0.6811.17> in an old incarnation (1652678813) of this node (1652679434) 2022-05-16 09:37:19.396201+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396201+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396206+04:00 [error] <0.225.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2692626030.1821376515.175644>}} from <0.225.0> to <0.6814.17> in an old incarnation (1652678813) of this node (1652679434) 2022-05-16 09:37:19.396206+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396206+04:00 [error] <0.225.0> 2022-05-16 09:37:19.396215+04:00 [error] <0.225.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2692626030.1821376515.175644>}} from <0.225.0> to <0.6817.17> in an old incarnation (1652678813) of this node (1652679434) 2022-05-16 09:37:19.396215+04:00 [error] <0.225.0>

The size of the logs is growing at a high rate, since the launch, it has grown several gigabytes in a few hours.

[michaelklishin]

My best guess judging from just the logs is something like #3072.

We don't have any RabbitMQ version information on either ends of the Shovel, so it is impossible to
suggest much.

Deleting this Shovel and re-creating it may help. Unfortunately you use static shovels, so it would be less convenient compared to dynamic shovels.

[sbulyigin]

On 12/30/2021, the test passed and the queue reached. After that, for some reason, rabbitmq stopped working, the web console also. I couldn't revive him. Deleted it completely and reinstalled it.
I installed the latest version 3.10.2. (I didn't touch erlang).
At that time it was 3.9.8.
If you believe the web interface, version 3.8.19 is on the other side.

[sbulyigin]

My best guess judging from just the logs is something like #3072.

It turns out I have the following scheme: The Main Rabbitmq throws queues with My Rabbitmq. And in turn, 53 "clients" of Rabbitmq will connect with my Rabbitmq.
I understand correctly, I need to do the following:

The code at the moment makes two assumptions that are not correct or make much sense to me:

That there is a single consumer in the SAC queue case. This, in fact, will almost never be the case in practice (why would you use SAC if there is only one consumer in total?)
That we have to emit a consumers.created internal event only for the active consumer. We emit them for all consumers as they are registered, so it makes sense to do this in this "forced event refresh" case invoked by the management plugin when it starts

[sbulyigin]

Lowered the version to 3.9.8. It did not give any results.