Quarkus - Redis | Implement scheduler for cache refresh - getting ConnectionPoolTooBusyException on key scan

[Jijithnair]

Hello,

I have implemented a scheduler in my Quarkus application that runs every 2 minutes to refresh the cache (The cache items have a TTL, but a higher value, in order of days). The relevant code snippet is as follows:

import io.vertx.mutiny.redis.client.Redis;
import io.vertx.mutiny.redis.client.Request;
import io.vertx.mutiny.redis.client.Command;

private final Redis redisClient;

// Essential code block
redisClient.send(Request.cmd(Command.KEYS).arg("*"));

The issue I'm encountering is that performing an entire key scan like above results in the following exception:
io.vertx.core.http.ConnectionPoolTooBusyException: Connection pool reached max wait queue size of 24
While I understand that increasing the pool size and switching to the SCAN instead of KEY command could be alleviate this, I am seeking the most effective approach to handle this situation. Given that the scheduler runs every 2 minutes, I want to ensure the solution is efficient and doesn't cause connection pool exhaustion.

Additionally, I have observed a near one-to-one relationship between the quarkus.redis.max-pool-waiting property and the number of keys. I need to set this property value higher than the number of keys to avoid the exception, which is not practical for obvious reasons. How does this pile up waiting requests in the queue, though i understand that Redis is doing a heavy job on the server side, but is there any delegation that happens on the client side, that internally batches the request by default which is making concurrent requests? Also i have tried using SCAN but with high waiting queue size, which is working, but wondering why the same shows no difference with the KEYS version, as my expectation is an optimisation on the server side.

Could you please provide guidance on:

• The ideal configuration for quarkus.redis.max-pool-waiting in such scenarios.
• Best practices for handling large key scans efficiently within a short duration refresh interval.
  Thank you for your support.

[quarkus-bot[bot]]

/cc @Ladicek (redis), @cescoffier (redis), @gwenneg (cache), @machi1990 (redis), @manovotn (scheduler), @mkouba (scheduler)

[Jijithnair]

@cescoffier Thanks for the reply, but iam failing to understand, what is causing the issue ConnectionPoolTooBusyException here, as I understand that Redis is doing a heavy job on the server side on KEYS, but is there any delegation that happens on the client side, that internally batches the request by default which is making concurrent requests? Also i have tried using SCAN but with high waiting queue size, which is working, but wondering why the same shows no difference with the KEYS version, as my expectation is an optimisation on the server side.

[Jijithnair]

@cescoffier When i say keys, we are looking for something in the range of thousands (<10000). How would it change if the size is say in range of few thousands? How should i tune the values of pool size and waiting queue size as I am not understanding the above problem statement, as in KEYS, it seems there are concurrent requests being made and hence the connection pool concern, but in essence how are even making concurrent request? Keeping that aside, what could be the ideal configuration if the given context helps you to suggest on it?

[cescoffier]

If you have many keys, the command will take a lot of time to execute, but it keeps the connection open. So, if you have task execution overlap, each takes a connection, and at some point, it won't work anymore.

So, imagine that the command takes 30 seconds to execute, and you run your task every 5 seconds. Very quickly, you will have a connection starvation issue—all the connections are used.

Unlike what you mentioned, the client does not batch the request, except if done explicitly. So, once no connection is available, you get the exception.

SCAN is slightly different. It has the same "pattern", keeping the same connection open until the scan is completed (you read of the keys). So, the same problem will also happen: if you have task execution overlap, you may end up with a connection starvation.

Unfortunately, there is no silver bullet configuration here. I already mentioned that I would avoid task execution overlap to make sure you use at most one connection.

[Jijithnair]

@cescoffier The behaviour I am observing is different as i mentioned in the description. If i keep the defaults for pool size and waiting queue size, and the scheduler is scheduled to run at every 2 minutes. I would get the exception, just when i start the application, which means there is no task overlap, but within the same task, as the next task is scheduled to run only after 2 minutes. I can understand exceptions like server overloaded or timeout etc from Redis server as I am issuing a blocking bulk command to it, but how concurrent requests are made here, as I am issuing only a single KEYS commands and even with SCAN, with default pool config, i have the same issue.

[cescoffier]

Hum, then we would need a reproducer, but if you use a single connection, it might be something happening on the server side.

[Jijithnair]

Closing this as the underlying issue was having multiple Uni instances created for interacting with Redis, and not related to scheduler/ get keys, that exhausted the connection pool resources.