Quarkus gRPC runs all request in the same thread

[ctampak]

I am developing a service using Quarkus 2.16.5.Final and while admitedly each request is extremely fast, I noticed the service does not scale in all cores.
I am using ghz.sh to benchmark the service and JProfiler to profile. What I noticed in the profiler thread history view is that for each bechmark run ALL requests (we are talking 100k requests with a concurency of 10 using 2 cores for the benchmark tool) are executed in the same vertx-eventloop-thread. Interestingly each run picks a different thread, but all requests are executed in that thread.
This is what my config looks like:
quarkus.grpc.server.use-separate-server=false quarkus.grpc.server.xds.enabled=false quarkus.vertx.event-loops-pool-size=20 quarkus.grpc.server.instances=20
I set the event loops pool size to ensure I am not accidentaly exceeding it from grpc config. I tested all different combinations of seperate server and xds and saw no difference.
Am I missing something or the grpc.server.instances is ignored?

[quarkus-bot[bot]]

/cc @alesj (grpc), @cescoffier (grpc)

[alesj]

how do we handle the multiple server with the new gRPC server?

@cescoffier multiple server? As before, there is just one gRPC server instance -- in the new case, it's the existing Vert.x HTTP server.
Before it was a new separate instance of Netty-based gRPC server -- in addition to HTTP server.

Afair, we already had a discussion that there will not be multi-server support in gRPC for now.
But I am open for new ideas if the new gRPC can provide this in a simple way / form.

[ctampak]

@alesj In this case I don't want multiple servers, what I wanted to understand is how may threads are goind to handle requests in parallel. As I presented above, I only see 1 vertx-eventloop thread at any time handling all requests regardless of the concurrency I set on the client.

[vietj]

you should use this https://vertx.io/docs/vertx-core/java/#_scaling_sharing_tcp_servers

[vietj]

I thought that the gRPC server was collocated with the Quarkus HTTP server

[ctampak]

@vietj I am probably doing something wrong, but I created a class that extends AbstractVerticle and then in a Lifecucle listener's onStart I called the new class like this:

this.vertx.deployVerticle(new MyVerticle(), new DeploymentOptions().setInstances(10))
                .andThen(res -> logger.infov("Deployed verticles: {0}", res))
                ;

That resulted in an exception:
java.lang.IllegalArgumentException: Can't specify > 1 instances for already created verticle
at io.vertx.core.impl.VertxImpl.deployVerticle(VertxImpl.java:670)

I can only guess that this is because the default verticle(s) are created already

[alesj]

I thought that the gRPC server was collocated with the Quarkus HTTP server

@vietj it is.
This is how we use it

• https://github.com/quarkusio/quarkus/blob/main/extensions/grpc/runtime/src/main/java/io/quarkus/grpc/runtime/GrpcServerRecorder.java#L136

... which might not be completely the right way?

[vietj]

@ctampak since gRPC is using HTTP/2, did you make sure that the client used for testing opens more than one HTTP/2 connection ?

[ctampak]

I thought since I set --insecure this will fallback to http1.1. is it goind to use http2 even without SSL? In that case I can check if multiplexing in at play here

[vietj]

gRPC has to use HTTP/2 by design, it cannot use HTTP/1.1, for clear text it will use h2c (http/2 without SSL)

you can look at the traffic using Wireshark and check now many connections have been opened by the client.

[alesj]

@ctampak what about if you leave out this quarkus.grpc.server.use-separate-server=false config?
This will then give you the old Vert.x gRPC server, where we do use vertices + instances ...

[alesj]

And make sure that the Quarkus profile is not dev -- as we only use vertices with non-dev ...

            if (launchMode == LaunchMode.DEVELOPMENT) {
                // start single server, not in a verticle, regardless of the configuration.instances
                // for reason unknown to me, verticles occasionally get undeployed on dev mode reload
                if (GrpcServerReloader.getServer() != null || (provider != null && provider.serverAlreadyExists())) {
                    devModeReload(grpcContainer, vertx, configuration, provider, blockingMethodsPerService, shutdown);
                } else {
                    devModeStart(grpcContainer, vertx, configuration, provider, blockingMethodsPerService, shutdown,
                            launchMode);
                }
            } else {
                prodStart(grpcContainer, vertx, configuration, provider, blockingMethodsPerService, launchMode);
            }

    private void prodStart(GrpcContainer grpcContainer, Vertx vertx, GrpcServerConfiguration configuration,
            GrpcBuilderProvider<?> provider, Map<String, List<String>> blockingMethodsPerService, LaunchMode launchMode) {
        CompletableFuture<Void> startResult = new CompletableFuture<>();

        vertx.deployVerticle(
                () -> new GrpcServerVerticle(configuration, grpcContainer, provider, launchMode, blockingMethodsPerService),
                new DeploymentOptions().setInstances(configuration.instances),

[ctampak]

I have tried that with quarkus.grpc.server.instances set to a number >1. Same behaviour in JProfiler. What I can say changed though is performance. We are looking from ~5k with seperate-server=false to 15k without. But thread usage looked identical.
I can if you want run 2-3 runs with the aformentioned options and paste a screenshot of what I see here if that helps

[ctampak]

And make sure that the Quarkus profile is not dev -- as we only use vertices with non-dev ...

            if (launchMode == LaunchMode.DEVELOPMENT) {
                // start single server, not in a verticle, regardless of the configuration.instances
                // for reason unknown to me, verticles occasionally get undeployed on dev mode reload
                if (GrpcServerReloader.getServer() != null || (provider != null && provider.serverAlreadyExists())) {
                    devModeReload(grpcContainer, vertx, configuration, provider, blockingMethodsPerService, shutdown);
                } else {
                    devModeStart(grpcContainer, vertx, configuration, provider, blockingMethodsPerService, shutdown,
                            launchMode);
                }
            } else {
                prodStart(grpcContainer, vertx, configuration, provider, blockingMethodsPerService, launchMode);
            }

    private void prodStart(GrpcContainer grpcContainer, Vertx vertx, GrpcServerConfiguration configuration,
            GrpcBuilderProvider<?> provider, Map<String, List<String>> blockingMethodsPerService, LaunchMode launchMode) {
        CompletableFuture<Void> startResult = new CompletableFuture<>();

        vertx.deployVerticle(
                () -> new GrpcServerVerticle(configuration, grpcContainer, provider, launchMode, blockingMethodsPerService),
                new DeploymentOptions().setInstances(configuration.instances),

I always run in prod with a packaged version and the following options:
java -XX:+UseZGC -Xlog:async -jar .\target\quarkus-app\quarkus-run.jar

This is what my java version looks like:

> java -version
openjdk version "17.0.4.1" 2022-08-12 LTS
OpenJDK Runtime Environment Zulu17.36+17-CA (build 17.0.4.1+1-LTS)
OpenJDK 64-Bit Server VM Zulu17.36+17-CA (build 17.0.4.1+1-LTS, mixed mode, sharing)

[alesj]

What I can say changed though is performance. We are looking from ~5k with seperate-server=false to 15k without.

So you're saying that the old one is performing 3x better?
@vietj ^^ uh uh ...

[ctampak]

Some numbers:
Config:

quarkus.grpc.server.use-separate-server=true
quarkus.grpc.server.instances=10

Results:

Summary:
  Count:        10000
  Total:        713.30 ms
  Slowest:      13.48 ms
  Fastest:      0.24 ms
  Average:      2.53 ms
  Requests/sec: 14019.32

Config:

quarkus.grpc.server.use-separate-server=false
quarkus.grpc.server.instances=10

Results:

Summary:
  Count:        10000
  Total:        1.86 s
  Slowest:      59.56 ms
  Fastest:      0.97 ms
  Average:      9.00 ms
  Requests/sec: 5387.69

Both were 2nd runs, meaning I did a round of 10k requests as a small warmup. What I notice now is this is a really small test and probably I should run about 30sec to get a more reliable number, so take the numbers with a grain of salt.

For Both runs I verified: Profile prod activated.
For the run with separate server I had: gRPC Server started on 0.0.0.0:9000 [SSL enabled: false]
This is where I was basing my assumption about HTTP 1.1 and not 2. I can completely wrong though

[alesj]

I'll try / test now if new Vert.x gRPC behaves as expected -- as told by Julien wrt new connections usage, etc.

[ctampak]

Don't lose time, I tested running multiple instances of the benchmark in parallel and the behaviour is as expected. My test was broken. Apologies for the time I cost you

[alesj]

https://github.com/alesj/quarkus_cl130/tree/eventloop1