Automatic recovery is not working for local rabbitmq-server restart

[chirok11]

Step 1. Spawn RabbitMQ instance docker run -d -p 15672:15672 -p 5672:5672 -e RABBITMQ_DEFAULT_USER=guest -e RABBITMQ_DEFAULT_PASS=guest --name rabbitmq rabbitmq:management
Step 2. Run script
Step 3. Run docker restart rabbitmq
Step 4. Wait for recovery (it will be... forever)

Isn't feature is stabilized and tested before? If so, why it is not reconnecting, or ?

[dependencies]
tokio = { version = "1", features = ["full"] }
futures = "0"
lapin = "4"
tracing-subscriber = "0"

use std::time::Duration;

use lapin::{BasicProperties, ConnectionProperties, options::BasicPublishOptions};

#[tokio::main]
async fn main() {
    tracing_subscriber::fmt::init();

    let conn_properties = ConnectionProperties::default().enable_auto_recover();

    let conn = lapin::Connection::connect("amqp://localhost:5672/%2f", conn_properties)
        .await
        .unwrap();

    let channel = conn.create_channel().await.unwrap();

    loop {
        println!("Status: {:?}", channel.status());

        if let Err(error) = channel
            .basic_publish(
                Default::default(),
                "queue".into(),
                BasicPublishOptions::default(),
                "Hello, world!".as_bytes(),
                BasicProperties::default(),
            )
            .await
        {
            if let Err(error) = channel.wait_for_recovery(error).await {
                println!("Error: {:?}", &error);
            }
        }

        tokio::time::sleep(Duration::from_secs(1)).await;
    }
}

Enabled tracing spamming such logs (many thousands)

2026-02-25T15:16:19.756724Z TRACE io_loop: lapin::io_loop: io_loop run
2026-02-25T15:16:19.756728Z TRACE io_loop: lapin::io_loop: io_loop do_run can_read=true can_write=true has_data=false
2026-02-25T15:16:19.756732Z TRACE io_loop: lapin::io_loop: Should continue? status=Initial connection_status=Reconnecting internal_rpc_empty=true frames_pending=false conn_killed=true ser_frames_empty=true
2026-02-25T15:16:19.756739Z ERROR io_loop: lapin::io_loop: Socket was readable but we read 0. This usually means that the connection is half closed, thus report it as broken.
2026-02-25T15:16:19.756743Z TRACE io_loop: lapin::io_loop: We're in the process of recovering connection, quit reading socket to enter recovery

[chirok11]

I figured it out. Maybe it's worth documenting enable_auto_recover(), or changing the behavior to disable auto recover by default, but when the user enables enable_auto_recover, set a value for max_times other than 0 in ExponentialBuilder.

Also, there is no information here that it is necessary to pass your ExponentialBuilder. https://docs.rs/lapin/latest/lapin/struct.ConnectionProperties.html#method.enable_auto_recover

Even in examples (looks they are outdated).

There is also a special feature:

     .configure_backoff(|backoff| {
            backoff.with_max_times(10);
            backoff.with_min_delay(Duration::from_secs(1));
            backoff.with_max_delay(Duration::from_secs(5));
        })

Will not work

.with_backoff(
            ExponentialBuilder::default()
                .with_max_times(10)
                .with_min_delay(Duration::from_secs(1))
                .with_max_delay(Duration::from_secs(5)),
        )

In this case, reconnection will occur.

I hope this issue helps someone to figure out why reconnect does not happen.

[chirok11]

I'll leave it open because reconnecting is very unstable, with spam occurring in half of 10 cases.

Socket was readable but we read 0. This usually means that the connection is half closed, thus report it as broken.

And reconnections do not occur.

[stefa168]

Yup, with the same settings as @chirok11 I still cannot get the connection back up in any way... I'm getting the same "spammed" message

[Ks89]

I'm migrating my code to lapin 4 with automatic recovery, but I'm having the same issues.
If I restart my rabbitmq docker container I see a huge amounts of:

026-02-27T15:24:07.737716Z ERROR io_loop: lapin::io_loop: Socket was readable but we read 0. This usually means that the connection is half closed, thus report it as broken

but no auto recovery at all.

Is there an official example updated to v4.x.x with automatic recovery?

[Keruspe]

Hello,
First of all, thanks for the report.

Isn't feature is stabilized and tested before? If so, why it is not reconnecting, or ?
It obviously was tested by me and a other volunteers who confirmed it did not introduce any regression, and it fixed at least some of the failures they were encountering.

The ConnectionProperties::configure_backoff was introduced quite late, it was supposed to be a simple helper to simplify things, but I definitely got bitten by implicit Copy here. I've taken the decision to break this specific method in terms of API in 4.1.0 (which is bad, I know, it's not something I usually do) because it was necessary to fix it, and to break existing callers on purpose so that they somehow get noticed that it was broken.

The backoff is used at the lowest level possible, for the TCP connection itself, while the recovery happens at the protocol level. While it would be feasible, the place where TCP connection happen knows as little as possible about the protocol, and isn't aware of anything recovery-related, we only pass it the backoff to use for retries, which is why max_tries was set to 0 by default. We now set it to the arbitrary value of 16 by default when enabling recovery, if it wasn't already configured by the user.

4.1.0 will at least fix this configuration glitch.

Note that all the testing I personally made was targetting network issues, and not rabbitmq server restarts. They were all done on WAN network as this is where most issues happen. I understand the usecase for a local rabbitmq-server restart though and will take it into account in my further testing to fix the rest of you issue.

[chirok11]

They were all done on WAN network as this is where most issues happen

Error mentioned in my first message will be on LAN/WAN/Local, does not matter, I tried three cases:

1. local rabbitmq server
2. rabbitmq server in lan
3. rabbitmq server in wan

after restart script will eat cpu and spam this errors

2026-02-28T14:38:28.050316Z TRACE io_loop: lapin::io_loop: Should continue? status=Initial connection_status=Reconnecting internal_rpc_empty=true frames_pending=false conn_killed=true ser_frames_empty=true
2026-02-28T14:38:28.050324Z ERROR io_loop: lapin::io_loop: Socket was readable but we read 0. This usually means that the connection is half closed, thus report it as broken.
2026-02-28T14:38:28.050330Z TRACE io_loop: lapin::io_loop: We're in the process of recovering connection, quit reading socket to enter recovery

[Ks89]

I tryed to restart a local docker container to mimic a pod restart on a production kubernetes cluster.

If I need to redeploy with something like Helm It could happen that some pods and rabbitmq will restart, but not the service with lapin (because unchanged).
So I need to be sure the existing rust service with lapin will be capable to auto reconnect to rabbitmq server without manually restarting It.

[chirok11]

@Keruspe
I will add more details: to easily reproduce issue just remove last line tokio::time::sleep(Duration::from_secs(1)).await; and it will not reconnect in 10/10 times and spam errors.

You can see my attempt to fix this; at least, I can no longer reproduce the error when using the example from the first message.
chirok11@4987a42

[Keruspe]

Yeah, I honestly mostly tested recovery of protocol errors and wan interrupts, rabbitmq server restart was not part of my testing. I was sponsored by cloudamqp for this feature and never had such issues with them outside of planned maintenances so it wasn't in the list of priorities.

I'll definitely try to find some time to work on it soon and I'll give your work a look too

[stefa168]

No worries, it is very hard to foresee all possible failure cases!

If any help with testing or contributions is needed, just ask