Executor thread number goes way above the configured pool size #32621
-
|
Hi, we have an app that does a poll via rest, and processes the response within a ManagedExecutor runAsync bloc. When I run something like this locally (jvm or native) I see the thread number in the log staying in the boundary of the 200 thread pool. (This run in a quartz scheduled method that runs every 3 seconds for example) for (int i = 0; i < 100000; i++) {
int idx = i;
executor.runAsync(() -> {
if (idx % 1000 == 0) {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {}
log().info("Current index is {}", idx);
}
});
}Oddly in our production app, which the schedule method runs every 2 minutes we see the executor threads piling up, now after a few days we see something like this in the logs; I do see something some number reuse, but still not sure why it acts this way, and for sure the processing has time to happen within the 2 minutes of the poller, versus my dummy test above where I force a 5s sleep on a 3s poll. I am afraid this will eventually blow up / hit a certain limit. For sure there are not 10k concurrent threads as our container has only 2 cpu and 256mb memory. Help would be appreciate to understand what is happening above and why this is behaving this way. I am trying to reproduce this locally but so far I an unable to. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 4 replies
-
|
I assume that you're using an injected This executor should indeed leverage the default blocking thread pool. So if you specify the Do you use the same value of |
Beta Was this translation helpful? Give feedback.
-
|
So after more troubleshooting and reading of the javadoc I think I understand now what is happening. To answer you question first, yes I am using What happens is that, by default quarkus.thread-pool.core-threads is 1, meaning that a single thread stays alive forever. When the Why we did not see it in dev, it is because the scheduled method, calling multiple times the ManagedExecutor, is ran much frequently i.e. below the quarkus.thread-pool.keep-alive-time so the thread can be reused properly. To alleviate that, I have made a config interceptor, that set that value to our configured "poller" interval + a small buffer, so more threads can possibly reused. So in our app, each poll, still have the potential of making the thread number by hundreds, but far less now that we reuse threads much more. Also one big factor we saw, we have other scheduler method that runs in a much lower interval, like once a day, thise does some monitoring, and then So the actual solution would be, in our specific case; properly set quarkus.thread-pool.core-threads to something like 20 so at least 20 threads, wont ever be torndown and will always be reused, and the for the rest it will be via the keepalive tuning. What is harder now is this if for the |
Beta Was this translation helpful? Give feedback.
-
Ok, so the thread is terminated and the max limit is never reached. In other words, everything works correctly (the thread name does not really matter). And yes, it makes sense to fine tune the |
Beta Was this translation helpful? Give feedback.
-
This is exact. It just worried me big time at first glance when I saw the thread number in the 10k range in not that long runtime of our app. Not sure if the number is a int or long, but at this rate, without tuning there is a chance we could hit a certain limit. The part I find hard in the tuning is to have different config for both vertx worker threads + quarkus threadpool, without having a way to configure the thread pool used for things like fireAsync and Scheduled methods, and it seem to be also not possible at the moment to create our own threadpools (as far as I can see #6156), a bit like TaskExecutor in spring. Since we cannot exactly trust the thread name in the log for tuning since it could run on a vertx worker, even if the thread name does not represent that makes it harder as well. So as far as this particular discussion I think it is all ok, after proper tuning. I do think it is a bit cumbersome though as highlighted above. Thanks a lot for your time, always super appreciated! |
Beta Was this translation helpful? Give feedback.
-
It seems to be backed by an |
Beta Was this translation helpful? Give feedback.
So after more troubleshooting and reading of the javadoc I think I understand now what is happening. To answer you question first, yes I am using
org.eclipse.microprofile.context.ManagedExecutor, and no I am not changing this for dev/prod;quarkus.thread-pool.max-threads.What happens is that, by default quarkus.thread-pool.core-threads is 1, meaning that a single thread stays alive forever. When the
ManagedExecutordoes multiperunAsync(). Threads are getting created, and those thread lives idle for the configured quarkus.thread-pool.keep-alive-time once the keepalive is reached for those thread, as far as I can see, their thread number is never reused thus incrementing.Why we did not s…