All requests with client fail after initial requests failed due to network error #2956
Description
This issue is a minimization of a behavior we observed in uv: When the initial request fails due to the network going away, all subsequent retries fails.
In the MRE below, we reproduce something that can observed with uv. Start the command below.
uv venv -c # Clear an existing torch installation if any
uv pip install torch --no-cache
Disconnect the network while it's running. Reconnect the network. The currently ongoing downloads will fail (expectedly), but unexpectedly, all retries now fail, even if there is internet and other clients can fetch the file just fine.
This seems to be some kind of problem with the client's internal HTTP2 connection pool getting tainted, but I couldn't find any option to reset the connection pool.
The reproducer is also available at https://github.com/konstin/reqwest-fails-after-network-error. It contains commands that worked for me on Ubuntu 24.04 with the default network manager.
[package]
name = "reqwest-fails-after-network-error"
version = "0.1.0"
edition = "2024"
[dependencies]
tokio = { version = "1.49.0", features = ["full", "fs"] }
reqwest = { version = "0.13.1", default-features = false, features = ["http2", "default-tls"] }use std::path::Path;
use std::time::Duration;
use tokio::io::AsyncWriteExt;
use tokio::time::sleep;
async fn download(
client: &reqwest::Client,
url: &str,
path: &Path,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
// Like uv: send HEAD first
println!("Sending HEAD request");
let head_resp = client.head(url).send().await?.error_for_status()?;
println!(
"HEAD response: {} {:?}",
head_resp.status(),
head_resp.version()
);
let mut last_err = match download_inner(client, url, path).await {
Ok(()) => return Ok(()),
Err(err) => err,
};
println!("Initial download failed: {last_err}");
for retry in 0..3 {
println!("Backoff of retry");
sleep(Duration::from_secs(3)).await;
println!("Retrying");
match download_inner(client, url, path).await {
Ok(()) => return Ok(()),
Err(err) => {
println!("Download retry {retry} failed: {err:?}");
last_err = err;
}
}
}
Err(last_err)
}
async fn download_inner(
client: &reqwest::Client,
url: &str,
path: &Path,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
println!("Sending GET request");
let response = client.get(url).send().await?.error_for_status()?;
println!(
"GET response: {} {:?}",
response.status(),
response.version()
);
let bytes = response.bytes().await?;
let mut file = tokio::fs::File::create(path).await?;
file.write_all(&bytes).await?;
Ok(())
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
// Large file
let url = "https://files.pythonhosted.org/packages/ae/30/a3a2120621bf9c17779b169fc17e3dc29b230c29d0f8222f499f5e159aa8/torch-2.10.0-cp311-cp311-manylinux_2_28_x86_64.whl";
let client = reqwest::Client::builder()
.read_timeout(Duration::from_secs(3))
.build()?;
download(
&client,
url,
Path::new("torch-2.10.0-cp311-cp311-manylinux_2_28_x86_64.whl"),
)
.await?;
println!("Success");
Ok(())
}Below, replace nmcli device [dis]connect enp3s0 with the correct definition for your system to disconnect and reconnect the network:
#!/bin/bash
cargo run &
sleep 2
nmcli device disconnect enp3s0 # Deactivate network
sleep 4
nmcli device connect enp3s0 # Reactivate network
waitExample log:
Sending HEAD request
HEAD response: 200 OK HTTP/2.0
Sending GET request
GET response: 200 OK HTTP/2.0
Device 'enp3s0' successfully disconnected.
Initial download failed: error decoding response body
Backoff of retry
Device 'enp3s0' successfully activated with '<id>'.
Retrying
Sending GET request
Download retry 0 failed: reqwest::Error { kind: Request, url: "https://files.pythonhosted.org/packages/ae/30/a3a2120621bf9c17779b169fc17e3dc29b230c29d0f8222f499f5e159aa8/torch-2.10.0-cp311-cp311-manylinux_2_28_x86_64.whl", source: TimedOut }
Backoff of retry
Retrying
Sending GET request
Download retry 1 failed: reqwest::Error { kind: Request, url: "https://files.pythonhosted.org/packages/ae/30/a3a2120621bf9c17779b169fc17e3dc29b230c29d0f8222f499f5e159aa8/torch-2.10.0-cp311-cp311-manylinux_2_28_x86_64.whl", source: TimedOut }
Backoff of retry
Retrying
Sending GET request
Download retry 2 failed: reqwest::Error { kind: Request, url: "https://files.pythonhosted.org/packages/ae/30/a3a2120621bf9c17779b169fc17e3dc29b230c29d0f8222f499f5e159aa8/torch-2.10.0-cp311-cp311-manylinux_2_28_x86_64.whl", source: TimedOut }
Error: reqwest::Error { kind: Request, url: "https://files.pythonhosted.org/packages/ae/30/a3a2120621bf9c17779b169fc17e3dc29b230c29d0f8222f499f5e159aa8/torch-2.10.0-cp311-cp311-manylinux_2_28_x86_64.whl", source: TimedOut }