Too many open files in reverse-proxy example #2010

ghost · 2023-05-18T09:10:24Z

ghost
May 18, 2023

Hi everyone!

Here is a problem that I've been stuck with for a few days now. If I try implementing a simple reverse proxy based on the axum example, I inevitably stumble upon the open sockets limit (with or without concurrency limit layer).

use hyper::Uri;
use hyper::{Body, Request, Server};
use log::error;
use std::net::SocketAddr;

type Client = hyper::Client<HttpConnector, Body>;

#[tokio::main(flavor = "multi_thread")]
async fn main() {
    let addr = SocketAddr::from(([0, 0, 0, 0], 8080));
    let client = hyper::Client::builder().build_http();
    let app = Router::new()
        .route("/", get(proxy))
        .with_state(client)
        .layer(tower::ServiceBuilder::new().concurrency_limit(64))
        .into_make_service();
    let server = Server::bind(&addr).serve(app);

    if let Err(e) = server.await {
        error!("Proxy error: {}", e);
    }
}

async fn proxy(State(client): State<Client>, mut req: Request<Body>) -> Response {
    let path = req.uri().path();
    let path_query = req
        .uri()
        .path_and_query()
        .map(|v| v.as_str())
        .unwrap_or(path);
    let uri = format!("http://127.0.0.1{}", path_query);
    *req.uri_mut() = Uri::try_from(uri).unwrap();
    client.request(req).await.unwrap().into_response()
}

There's an nginx on 80, that just responds with default text. I'm using wrk for tests.


wrk -t12 -c1000 -d30s http://localhost:8080/
...
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: hyper::Error(Connect, ConnectError("tcp open error", Os { code: 24, kind: Uncategorized, message: "Too many open files" }))', src/main.rs:40:31

But then, using hyper::Client as the handling service, rewriting the request URL at the middleware level works just fine.

use hyper::Server;
use log::{error, info};
use std::net::SocketAddr;
use tower::Service;
use core::task::{Poll, Context};
use http::{Request, Response};

#[tokio::main(flavor = "multi_thread")]
async fn main() {
    let addr = SocketAddr::from(([0, 0, 0, 0], 8080));
    let client = hyper::Client::builder().build_http();
    let service = tower::builder::ServiceBuilder::new()
        .layer(RewriteLayer {})
        .service(client);
    let server = Server::bind(&addr)
        .serve(tower::make::Shared::new(service));

    info!("Listening on http://{}", addr);
    if let Err(e) = server.await {
        error!("Proxy error: {}", e);
    }
}

struct RewriteLayer;

impl<S> tower::Layer<S> for RewriteLayer<> {
    type Service = RewriteService<S>;

    fn layer(&self, service: S) -> Self::Service {
        RewriteService {
            service
        }
    }
}

#[derive(Clone)]
struct RewriteService<S> {
    service: S
}

impl <S, ReqBody, ResBody> Service<Request<ReqBody>> for RewriteService<S> where
S: Service<Request<ReqBody>, Response = Response<ResBody>>, ReqBody: std::fmt::Debug {
    type Response = S::Response;
    type Error = S::Error;
    type Future = S::Future;

    fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
        self.service.poll_ready(cx)
    }

    fn call(&mut self, mut request: Request<ReqBody>) -> Self::Future {
        *request.uri_mut() = "http://127.0.0.1:80/".parse().unwrap();
        self.service.call(request)
    }
}


$ wrk -t12 -c1000 -d30s http://localhost:8080/
Running 30s test @ http://localhost:8080/
  12 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    37.97ms  161.82ms   1.68s    94.42%
    Req/Sec    25.92k     6.78k   66.29k    80.48%
  9215749 requests in 30.05s, 7.32GB read
  Socket errors: connect 0, read 386, write 0, timeout 114
Requests/sec: 306715.47
Transfer/sec:    249.51MB

Is there something here I don't understand?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Too many open files in reverse-proxy example #2010

Uh oh!

{{title}}

Uh oh!

Replies: 0 comments

Select a reply

Uh oh!

Too many open files in reverse-proxy example #2010

Uh oh!

ghost May 18, 2023

Replies: 0 comments

ghost
May 18, 2023