Thread Exhaustion Under Load in Blocking Quarkus Application

[nitinty]

Hi everyone,

I have a Quarkus application where we’ve written our code in a purely blocking style—we're not leveraging Mutiny or Uni. Here's a simplified example:

@Path("/resource/test")
public class TestController {

    @Inject
    TestService testService;

    @GET
    @Path("/{id}")
    @Produces("application/json")
    public Response getById(@PathParam("id") String id) {
        if (id == null || id.isEmpty()) {
            return Response.status(Response.Status.BAD_REQUEST).entity("ID must not be null or empty").build();
        }
        TestEntity testEntity = testService.getTestEntityById(id);
        if (testEntity == null) {
            return Response.status(Response.Status.NOT_FOUND).entity("Entity not found").build();
        }
        return Response.ok(testEntity).build();
    }
}

// The service and repository layers are also fully blocking using Panache and synchronous access:

@ApplicationScoped
public class TestService {

    @Inject
    TestRepository testRepository;

    public TestEntity getTestEntityById(String id) {
        if (id == null) return null;
        if (ObjectId.isValid(id)) {
            return testRepository.findById(new ObjectId(id));
        } else {
            return testRepository.find("_id", id).firstResult();
        }
    }
}


package com.example.mongo;

import io.quarkus.mongodb.panache.PanacheMongoRepository;
import jakarta.enterprise.context.ApplicationScoped;
import lombok.extern.slf4j.Slf4j;

@Slf4j
@ApplicationScoped
public class TestRepository implements PanacheMongoRepository<TestEntity> {

}

This approach works well under normal load, but during load testing, we observed the following issues:

• Pods get restarted due to thread exhaustion.
• During this time, Kubernetes liveness probes fail, marking the pods unresponsive.
• We’ve configured quarkus.thread-pool.max-threads=500, but it doesn't seem to help under pressure.
• Prometheus metrics don’t provide enough visibility into the executor threads — we’re missing key info like thread wait times, active threads, etc.

Questions:

• Can anyone please suggest, what’s the best way to deal with thread exhaustion in blocking Quarkus apps ?
• Are there any recommended monitoring setups or thread pool metrics we can use for better visibility ?

Any advice or experience shared would be really helpful. Thanks in advance!

[quarkus-bot[bot]]

/cc @geoand (kubernetes), @iocanel (kubernetes)

[geoand]

@yrodiere @cescoffier I wonder if this is an instance of what we discussed the other day where the liveness checks don't get executed because they are just another task in the worker thread executor.

[yrodiere]

IIRC @cescoffier was talking about custom health checks that were using Hibernate. You don't have any custom health check in your case, right @nitinty? Do you know which liveness check is failing exactly?

I can imagine health checks failing if they involve the database and the connection pool is already saturated with requests. If that's the case, we should investigate whether we can give connection requests higher priority when the requester is executing a health check. I'm not sure if Agroal allows that, but... maybe?

The other possibility is that under load, your application has long-running requests getting hold of the whole connection pool, leading to liveness check not being able to get a connection and timing out, simply because no connection was released before the timeout. Your example could never be in that case, but I assume your actual application is more complex than that...

[nitinty]

The other possibility is that under load, your application has long-running requests getting hold of the whole connection pool, leading to liveness check not being able to get a connection and timing out, simply because no connection was released before the timeout. Your example could never be in that case, but I assume your actual application is more complex than that...

Thanks for the response — yes, this seems to be the case we’re observing.

However, our average API response time during load testing stays well under 100 ms (with normal load), and we don’t have any obviously long-running requests. Despite this, we’re seeing thread exhaustion symptoms — like liveness probe failures and pod restarts.

The challenge is, we’re not able to pinpoint the root cause because Prometheus metrics for executor threads (like active thread count, queue wait times, etc.) aren't providing sufficient visibility.

Would really appreciate any pointers on:

• How to monitor executor thread metrics more effectively in Quarkus?
• Any tools or approaches you’d recommend for diagnosing thread blocking or exhaustion in such cases?

And yes, you're right, the actual application is more complex than the minimal example I shared (that was just to illustrate our overall blocking approach and layering).

[geoand]

cc @brunobat and @barreiro as well

[brunobat]

Detailed metrics are published for the Agroal managed DB connections if this property is set:

quarkus.datasource.metrics.enabled=true