how to handle multiple base urls with rest-client?

[PyAntony]

Hi,

I have an application where I get multiple urls with different base urls. I don't know the base URLs so I can't add them all in the config file. What I am currently doing is creating a new client for each base URL. Something like:

String[] baseAndRelativePaths = splitUrl(newUrl)

// cache is a mapping from String (base URL) -> web client
cache.putIfAbsent(baseAndRelativePaths[0], getNewClient())

// WebClient.class is the interface
private WebClient getNewClient(String baseUrl) {
  return RestClientBuilder.newBuilder()
        .baseUri(URI.create(basseUrl))
        .property("io.quarkus.rest.client.connection-pool-size", 50)
        .build(WebClient.class)

cache.get(baseAndRelativePaths[0]).doGETrequest(baseAndRelativePaths[1])
}

This is working, but I am not sure if this is inefficient. I guess if there is a new thread pool being created for ach base URL this is very inefficient. So:

1. How are connection pools created? one for each base URL or one single connection pool for all the clients?
2. is my approach correct? can it be optimized?
3. Also, does Netty create a new connection for each host or for each full URL? For example, given these 2 urls with the same host:

• https://example.com/api/endpoint1
• https://example.com/api/endpoint2

Is one connection created or 2?

[quarkus-bot[bot]]

/cc @michalszynkiewicz

[michalszynkiewicz]

To save resources, you can share the http client between two rest clients.
To do this, set the QuarkusRestClientProperties.SHARED and QuarkusRestClientProperties.NAME properties on the client builder:

RestClientBuilder.newBuilder()
        .baseUri(URI.create(basseUrl))
        .property("io.quarkus.rest.client.connection-pool-size", 50)
        .build(WebClient.class)
        .property(QuarkusRestClientProperties.NAME, "my-client-group")
        .property(QuarkusRestClientProperties.SHARED, true);

Under the hood, with this setting, Quarkus will use shared Vert.x HTTP Client

[michalszynkiewicz]

then quarkus.rest-client.logging.scope=all should enable netty logging... Maybe some logging category needs to be set to TRACE/DEBUG...

[manofthepeace]

I am curious about this as well. I know netty is not the apache httpclient. But I wonder if they have the same recommendation as apache here;
Generally it is recommended to have a single instance of HttpClient per communication component or even per application

I tried just "for fun" bundling all my RestClients under the same QuarkusRestClientProperties.NAME and set shared to true. I do not see any difference in the netty debug logs, well, as @PyAntony said, not enough information to say if it acutally helps. Via tcpdump I see the same with or without that change.

Also unsure how the connection-pool-size work in that case. My rest clients are all configured via config file, and all have a pool size set, if we share some of them which connection-pool would be taken?

[michalszynkiewicz]

@PyAntony can you attach a very simple project for me to try it out? I'd be happy to check it but don't have much time...

[PyAntony]

A very simple project of the rest-client? do you need to see the logs?

[michalszynkiewicz]

one project, two clients, with similar base uris and paths as you use.

[PyAntony]

@michalszynkiewicz
Project here: https://github.com/PyAntony/quarkus-rest-client-demo
You can just git-clone it and run it with quarkus dev.

Notice the application.properties file:

quarkus.rest-client.logging.scope=all

quarkus.log.min-level=TRACE
quarkus.log.category."org.acme".level=TRACE
quarkus.log.category."org.jboss.resteasy.reactive.client.logging".level=TRACE

It only logs info about the request and response, but no low level info about Netty.

All this project does is to send a GET request every 5 seconds:

@Scheduled(
        every = "5s",
        concurrentExecution = SKIP
    )
    public void runClient() {
        var urls = List.of(
            "https://docs.python.org/3/whatsnew/3.10.html",
            "https://docs.python.org/3/whatsnew/3.9.html",
            "https://docs.scala-lang.org/tour/higher-order-functions.html",
            "https://docs.scala-lang.org/tour/traits.html"
        );

        urls.forEach(
            url -> webClient.doGETRequest(url)
                .subscribe().with(resp -> {
                    if (resp != null)
                        Log.debugf("Received %s bytes...", resp.getBytes().length);
                    else
                        Log.debugf("UNI produced null...");
                })
        );
    }

[akil-rails]

I've the same scenario, was there a resolution to this?