Consumers are not reconnecting after lost of internet connection

[Zdunke]

Hello,
i am using your rabbit server in my project. After some DDOS attack consumers in my project had a problem to recconect to servers. I am using version 3.9.11. I created some envrionement to reproduce this error. On my localhost i simulate the loss of and resumption of the connection. Consumers stop connecting to the server after some time. In rabbit logs i am getting this kind of errror:

2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> supervisor: {<0.4420.8>,rabbit_channel_sup} 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> errorContext: shutdown_error 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> reason: noproc 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> offender: [{pid,<0.4424.8>}, 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> {id,channel}, 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> {mfargs, 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> {rabbit_channel,start_link, 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> [1,<0.4340.8>,<0.4421.8>,<0.4340.8>, 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> <<"127.0.0.1:49601 -> 127.0.0.1:5672">>, 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> rabbit_framing_amqp_0_9_1, 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> {user,<<"guest">>, 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> [administrator], 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> [{rabbit_auth_backend_internal,none}]}, 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> <<"/">>, 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> [{<<"publisher_confirms">>,bool,true}, 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> {<<"exchange_exchange_bindings">>,bool,true}, 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> {<<"basic.nack">>,bool,true}, 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> {<<"consumer_cancel_notify">>,bool,true}, 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> {<<"connection.blocked">>,bool,true}, 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> {<<"authentication_failure_close">>,bool,true}], 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> <0.4389.8>,<0.4422.8>]}}, 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> {restart_type,intrinsic}, 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> {shutdown,70000}, 2022-03-03 13:35:16.337000+01:00 [error] <0.4420.8> {child_type,worker}]

Is it known issue. I am able to give you my source code and script to simulate this problem.
kill.txt.txt
start.txt.txt

[Zdunke]

2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> supervisor: {<0.14627.8>,rabbit_channel_sup} 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> errorContext: shutdown_error 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> reason: noproc 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> offender: [{pid,<0.14630.8>}, 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> {id,channel}, 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> {mfargs, 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> {rabbit_channel,start_link, 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> [2,<0.14552.8>,<0.14628.8>,<0.14552.8>, 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> <<"127.0.0.1:63927 -> 127.0.0.1:5672">>, 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> rabbit_framing_amqp_0_9_1, 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> {user,<<"guest">>, 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> [administrator], 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> [{rabbit_auth_backend_internal,none}]}, 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> <<"/">>, 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> [{<<"publisher_confirms">>,bool,true}, 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> {<<"exchange_exchange_bindings">>,bool,true}, 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> {<<"basic.nack">>,bool,true}, 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> {<<"consumer_cancel_notify">>,bool,true}, 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> {<<"connection.blocked">>,bool,true}, 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> {<<"authentication_failure_close">>,bool,true}], 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> <0.14581.8>,<0.14629.8>]}}, 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> {restart_type,intrinsic}, 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> {shutdown,70000}, 2022-03-03 14:21:16.351000+01:00 [error] <0.14627.8> {child_type,worker}]

This log is often as well.

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

RabbitMQ nodes do not reconnect to clients, it's the other way around: clients reconnect to nodes. You have provided no information about what clients are used, and with what settings.

The exceptions likely mean that a channel ran into an issue during connection shutdown, e.g. due to missed client heartbeats.

[lukebakken]

There have been recent fixes to the .NET client specifically addressing re-connection and consumers. Please upgrade it to the latest version, 6.2.4, and re-try your tests. There should be no need to adjust the heartbeat value.

[Zdunke]

And what about massTransit, should i skip using it?

[lukebakken]

For the purpose of testing version 6.2.4 of the .NET client in your environment, if you can skip using MassTransit that would narrow down the cause of the issue.

[Zdunke]

Well, for testing I can as much as possible abandon the use of the MassTransit library. I will be back with an answer as soon as possible.

[lukebakken]

No rush! We appreciate you taking the time to test out the latest version of the library.