Separating Task Scheduling and Execution

[Blejwi]

Hi,

I’m wondering if it’s possible to separate the scheduling and processing of tasks.

In my scenario, I have an API instance responsible for scheduling tasks, and I would like to have multiple worker instances dedicated to executing those scheduled tasks. This is to keep the API instance under a lighter load.

Additionally, is it possible to configure the executor to pick up only specific types of tasks? For example, I’d like to set up a separate instance for heavy, long-running tasks and scale those independently from the instances handling short tasks.

Thanks in advance!

[osoykan]

In my scenario, I have an API instance responsible for scheduling tasks, and I would like to have multiple worker instances dedicated to executing those scheduled tasks. This is to keep the API instance under a lighter load.

This is possible as far as I know, if you don't start the scheduler, and use the SchedulerClient interface for scheduling-only purposes, it should be fine.

Workers can use Scheduler by starting.

Additionally, is it possible to configure the executor to pick up only specific types of tasks? For example, I’d like to set up a separate instance for heavy, long-running tasks and scale those independently from the instances handling short tasks.

For this one, i am not sure, as far as I have used the library, as soon as something is not KnownTask then it should be removed from the table. What you can do is, create separate schedulers hence separate tables for each scheduler you wanna prioritize. As if a separate queue.

[kagkarlsson]

Yes, this is a good answer. No partitioning, but you can use separate tables to achieve it.

There is a db-scheduler.enable=false you should be able to use as well to disable the task-processing. SchedulerClient can also be instantiated manually, i.e. not via starter, to enable only scheduling

[Blejwi]

That's great. Are you aware if there is a feature to have multiple job queues and dedicated executors for each of them?

[osoykan]

I edited my post to answer that :) @Blejwi

[Blejwi]

Thanks, I'll play with the configuration and verify if that's possible.