Possible connection leak when trying to create first connection

[krisz2000]

Describe the bug

Hi,

I'm trying to implement a rabbitmq connection that uses the ConnectionFactory with AutomaticRecoveryEnabled set to true. I was using code similar to what the documentation says. To make automatic recovery work I want to make sure that the first connection is created properly, so when a connection attempt fails I retry again after a 5 second delay. To simulate a rabbitmq that is unreachable I use a docker container running the latest image (rabbitmq:4.1.3-management). I start my dotnet service that tries to create the first connection at startup and as expected it fails at first with this exception:

RabbitMQ connection creation failure.
RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable
---> System.AggregateException: One or more errors occurred. (Connection failed, host 127.0.0.1:5672)
---> RabbitMQ.Client.Exceptions.ConnectFailureException: Connection failed, host 127.0.0.1:5672
---> System.Net.Sockets.SocketException (111): Connection refused
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at RabbitMQ.Client.Impl.SocketFactory.ConnectUsingAddressFamilyAsync(IPEndPoint endpoint, Func2 socketFactory, AddressFamily family, TimeSpan connectionTimeout, CancellationToken cancellationToken) --- End of inner exception stack trace --- at RabbitMQ.Client.Impl.SocketFactory.ConnectUsingAddressFamilyAsync(IPEndPoint endpoint, Func2 socketFactory, AddressFamily family, TimeSpan connectionTimeout, CancellationToken cancellationToken)
at RabbitMQ.Client.Impl.SocketFactory.OpenAsync(AmqpTcpEndpoint amqpTcpEndpoint, Func2 socketFactory, TimeSpan connectionTimeout, CancellationToken cancellationToken) at RabbitMQ.Client.Impl.SocketFrameHandler.CreateAsync(AmqpTcpEndpoint amqpTcpEndpoint, Func2 socketFactory, TimeSpan connectionTimeout, CancellationToken cancellationToken)
at RabbitMQ.Client.ConnectionFactory.CreateFrameHandlerAsync(AmqpTcpEndpoint endpoint, CancellationToken cancellationToken)
at RabbitMQ.Client.EndpointResolverExtensions.SelectOneAsync[T](IEndpointResolver resolver, Func3 selector, CancellationToken cancellationToken) --- End of inner exception stack trace --- at RabbitMQ.Client.EndpointResolverExtensions.SelectOneAsync[T](IEndpointResolver resolver, Func3 selector, CancellationToken cancellationToken)
at RabbitMQ.Client.Framing.AutorecoveringConnection.CreateAsync(ConnectionConfig config, IEndpointResolver endpoints, CancellationToken cancellationToken)
at RabbitMQ.Client.ConnectionFactory.CreateConnectionAsync(IEndpointResolver endpointResolver, String clientProvidedName, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at RabbitMQ.Client.ConnectionFactory.CreateConnectionAsync(IEndpointResolver endpointResolver, String clientProvidedName, CancellationToken cancellationToken)

After I start the rabbitmq container again the following exception is thrown a couple of times

RabbitMQConnection[0]
RabbitMQ connection creation failure.
RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable
---> System.IO.IOException: connection.start was never received, likely due to a network timeout
---> System.IO.EndOfStreamException: Pipe is completed.
at RabbitMQ.Client.Impl.InboundFrame.ReadFromPipeAsync(PipeReader reader, UInt32 maxInboundMessageBodySize, InboundFrame frame, CancellationToken mainLoopCancellationToken)
at RabbitMQ.Client.Framing.Connection.ReceiveLoopAsync(CancellationToken mainLoopCancellationToken)
at RabbitMQ.Client.Framing.Connection.MainLoop()
--- End of inner exception stack trace ---
at RabbitMQ.Client.Framing.Connection.StartAndTuneAsync(CancellationToken cancellationToken)
at RabbitMQ.Client.Framing.Connection.OpenAsync(CancellationToken cancellationToken)
at RabbitMQ.Client.Framing.Connection.OpenAsync(CancellationToken cancellationToken)
at RabbitMQ.Client.Framing.AutorecoveringConnection.CreateAsync(ConnectionConfig config, IEndpointResolver endpoints, CancellationToken cancellationToken)
at RabbitMQ.Client.ConnectionFactory.CreateConnectionAsync(IEndpointResolver endpointResolver, String clientProvidedName, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at RabbitMQ.Client.ConnectionFactory.CreateConnectionAsync(IEndpointResolver endpointResolver, String clientProvidedName, CancellationToken cancellationToken)

The end result is that when I check the rabbitmq management frontend I see multiple connections have been created by my service. These connections are kept alive by automatic recovery, but I have no way to access them because the factory threw an exception while creating them.
Is this a bug or am I missing something?

Reproduction steps

1. Use the code found in the documentation for automatic connection recovery. In the retry logic try connecting again until a connection is created.
2. Start the service while the rabbitmq container is not running. Wait a bit for some connection failures.
3. Start the rabbitmq container
4. Wait until the connection is created
5. See that there are multiple connections in the rabbitmq management frontend.

Expected behavior

Only one rabbitmq connection should be created and kept alive by the factory

Additional context

Rabbitmq management version: 4.1.3
Rabbitmq dotnet client version: 7.1.2
Dotnet version: 8.0

[lukebakken]

Hi, thanks for using RabbitMQ and this client library.

The simplest way for you to help me help you is to provide a git repository I can clone, compile, and run to observe this behavior. Doing that would save me the time it takes to set something up to test this. I can manage starting / stopping RabbitMQ.

Thank you.

[krisz2000]

Hi, thanks for the quick response, and sorry if I created this in the wrong place. Here is a minimal example to reproduce the issue:
https://github.com/krisz2000/rabbitmq-reconnect-test
Lowering the Task.Delay time means getting more rogue connections. Please tell me if you need any more information and I'll try my best to provide that.

[lukebakken]

Thank you, I'll investigate.

[lukebakken]

Yep, I can reproduce it. GH won't allow me to re-open your original issue, so I created this:

#1856

Please feel free to debug and fix this yourself! It would be GREATLY appreciated. I was hoping to get 7.2.0 released this week but actual work has gotten in the way.

[krisz2000]

I'll see what I can do, sadly I'm not familiar with the dotnet rabbitmq client code, but this might be a great opportunity to learn more about it.

[lukebakken]

Feel free to ask questions here or in #1856! I will answer!