Feature: Queue redirecting for timed out jobs

[UmutAkkaya]

The problem:

We have a queue (lets call it default) that's shared by multiple jobs. Some of these jobs may take a long amount of time clogging the queue slowing down the production environment. The timeout for the default queue is long enough that these jobs will not fail but all the messages will pile up in the queue until the long running job is processed.

The solution:

We started by adding a slow-running queue that has a worker with a long timeout. Extending from the Illuminate/Worker we created our prioritizing worker that within its timeout function checks if there is a backup queue the job can be assigned to instead. it looks something like this:

if (is_null($job)) {
       return;
}

$queue = $job->getQueue();
$timeoutQueueList = config('queue.time_out_queues');
if (isset($timeoutQueueList [$queue]) && $queue !== $timeoutQueueList [$queue]) {
       JobExtension::changeQueue($job, $timeoutQueueList[$queue]);
       $job->release();
} else {
       $this->kill(1);
}

We have created JobExtension to be able to set the queue property by simply doing

    public static function changeQueue(Job $job, string $newQueue) : Job
    {
        $job->queue = $newQueue;
        return $job;
    }

In order to keep the implementation generic for every queue, we have added the following settings to the queue.php that's used in the prioritizing worker implementation above:

...
'time_out_queues' => [
        'default' => 'default.slow_running'
        'new_queue_that_requires_timeout' => 'second.slow_running'
 ],
...

This way, for multiple queues, we can define their own time out queues.

Finally, we lowered the timeout for the main worker to make sure long running jobs can be rescheduled to the slow_running queue.

Proposal:

This implementation works great for us and we are already using it in production. In the case that one of the jobs runs longer than expected, it will re-queue itself without any manual intervene. If possible, I'd like to add this to the framework.

Do you think this would be valuable to add to the framework? If so, I'd be more than happy to add a PR for it.

[khepin]

To clarify, when we say some jobs run too long, it's not a job class or type that always runs over the timeout. It's a couple jobs that will usually finish in under a second for most of our users, but for some larger users, the job will take a lot longer.

This setup allows us to keep our main queue running and the flow of messages consumed to keep steady.

The worker for timeout / slow queues is also on a beefier machine and we can keep the standard workers on smaller machines which can have cost advantages in cloud environments.

[UmutAkkaya]

Any thoughts?