Conversation
|
Yes, this feature is required across all backends. |
Great, I'll see what I can do. |
jtwaleson
force-pushed
the
priority-queue
branch
from
c2ab0ab to
5369e57
Compare
|
Hey @kaplanelad , I've added support for sqlite and redis, and rebased my commits. The redis implementation needed some work as the native LPOP operations are incompatible with a priority queue report, and the combination with tags made it still a bit more difficult, hence the updated Lua script. While creating the test cases I found some flaky behavior but I've fixed all the issues. I've been running in production with the postgres implementation for months now, but I've not battle-tested sqlt and redis. I believe it's ready for review & merge. |
|
The priority queue feature is valuable, but a few things need to be addressed:
|
|
Alright, 1 & 3 are simple. For 2 I'll need a bit more time. Thanks for reviewing @kaplanelad . By the way, my startup https://comper.io is now out of the shadows and I can proudly say that it is fully built on Loco, so I wanted to say thanks for creating & maintaining the project! |
As Redis used a pure list, it needed to change to ZSET so we can order on priority when popping new tasks. Added extra tests to test some edge cases here. This required a bit of refactoring, as we now need a peek-and-acquire strategy to prevent concurrency issues. 1. Use `ZRANGE` to scan the queue without removing jobs. 2. Check job tags in the application. 3. If a match is found, use a Lua script (`ACQUIRE_JOB_SCRIPT`) to atomically verify existence, remove from queue, and move to processing set. To test for this, we added a regression test `test_dequeue_skips_mismatched_tags_no_infinite_loop`.
jtwaleson
force-pushed
the
priority-queue
branch
from
015ad29 to
1900fb6
Compare
|
Fixed all three points @kaplanelad . Let me know what you think. Btw noticed some failures while running |
2 participants
Here we add a priority queue to the postgres bg worker. This way, we can prioritize jobs.
Now we can do
Worker::perform_later_with_priority(ctx, params, Some(100))to schedule a high prio job or withSome(0)for a low prio job.I brought this up in #1462
What I'm using this for: we sync and analyze 100s of git repos. Each one has 3 stages: fetch, quick analysis, thorough analysis. As we want to show the results of the quick analysis as soon as possible, we give the following prios:
fetch: 5,quick analysis: 10,thorough analysis: 1. This way, we always fetch first, but after fetching, we do a quick analysis right away. When everything is fetched and "quick analyzed", we handle thethorough analysis.Please let me know what you think, and if we would want this for all queue backends or if we are fine with just postgres. If we don't want this at all, also fine.