Skip to content

Add reties delay logic #161

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
YaroslavStryhun wants to merge 1 commit into from
Closed

Add reties delay logic #161

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions src/Queues/DTO/BackoffStrategyDTO.php
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
<?php

namespace Qless\Queues\DTO;

class BackoffStrategyDTO
{
/**
* @var int
*/
private $initialDelay;

/**
* @var int
*/
private $factor;

public function __construct(int $initialDelay, int $factor)
{
$this->initialDelay = $initialDelay;
$this->factor = $factor;
}
Comment on lines +17 to +21
Copy link

@coderabbitai coderabbitai bot Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Validate inputs (non-negative delay, factor >= 1).

Protect against misconfiguration early.

   public function __construct(int $initialDelay, int $factor)
   {
+        if ($initialDelay < 0) {
+            throw new \InvalidArgumentException('initialDelay must be >= 0');
+        }
+        if ($factor < 1) {
+            throw new \InvalidArgumentException('factor must be >= 1');
+        }
         $this->initialDelay = $initialDelay;
         $this->factor = $factor;
   }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
public function __construct(int $initialDelay, int $factor)
{
$this->initialDelay = $initialDelay;
$this->factor = $factor;
}
public function __construct(int $initialDelay, int $factor)
{
if ($initialDelay < 0) {
throw new \InvalidArgumentException('initialDelay must be >= 0');
}
if ($factor < 1) {
throw new \InvalidArgumentException('factor must be >= 1');
}
$this->initialDelay = $initialDelay;
$this->factor = $factor;
}
🤖 Prompt for AI Agents 
In src/Queues/DTO/BackoffStrategyDTO.php around lines 17 to 21, the constructor
does not validate inputs; add checks to ensure $initialDelay is non-negative and
$factor is >= 1 before assigning to properties, and throw an
InvalidArgumentException (with clear messages like "initialDelay must be >= 0"
and "factor must be >= 1") when validations fail so misconfiguration is caught
early; perform validation first, then set $this->initialDelay and $this->factor
only if values pass.


/**
* @return int
*/
public function getFactor(): int
{
return $this->factor;
}

/**
* @return int
*/
public function getInitialDelay(): int
{
return $this->initialDelay;
}

public function toArray(): array
{
return [
'factor' => $this->getFactor(),
'initial_delay' => $this->getInitialDelay(),
];
}
}
8 changes: 6 additions & 2 deletions src/Queues/Queue.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
use Qless\Exceptions\UnknownPropertyException;
use Qless\Jobs\BaseJob;
use Qless\Jobs\JobData;
use Qless\Queues\DTO\BackoffStrategyDTO;
use Qless\Support\PropertyAccessor;
use Ramsey\Uuid\Uuid;

Expand Down Expand Up @@ -85,7 +86,8 @@ public function put(
?int $retries = null,
?int $priority = null,
?array $tags = null,
?array $depends = null
?array $depends = null,
?BackoffStrategyDTO $backoffStrategyDTO = null
): string {
try {
$jid = $jid ?: str_replace('-', '', Uuid::uuid4()->toString());
Expand Down Expand Up @@ -120,7 +122,9 @@ public function put(
'retries',
is_null($retries) ? 5 : $retries,
'depends',
json_encode($depends ?: [], JSON_UNESCAPED_SLASHES)
json_encode($depends ?: [], JSON_UNESCAPED_SLASHES),
'backoff',
json_encode($backoffStrategyDTO ? $backoffStrategyDTO->toArray() : [], JSON_UNESCAPED_SLASHES)
);

$this->getEventsManager()->fire(new QueueEvent\AfterEnqueue($this, $jid, $data->toArray(), $className));
Expand Down
66 changes: 51 additions & 15 deletions src/qless-core/qless.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -385,7 +385,10 @@ Qless.config.defaults = {
['histogram-history'] = 7,
['jobs-history-count'] = 50000,
['jobs-history'] = 604800,
['jobs-failed-history'] = 604800
['jobs-failed-history'] = 604800,
-- retries logic
['backoff-initial-delay'] = 0, -- Default delay in seconds. 0 means disabled.
['backoff-factor'] = 3 -- Exponential factor.
}

Qless.config.get = function(key, default)
Expand Down Expand Up @@ -1584,19 +1587,25 @@ function QlessQueue:put(now, worker, jid, klass, raw_data, delay, ...)
redis.call('hincrby', 'ql:s:stats:' .. bin .. ':' .. self.name, 'failed' , -1)
end

redis.call('hmset', QlessJob.ns .. jid,
'jid' , jid,
'klass' , klass,
'data' , raw_data,
'priority' , priority,
'tags' , cjson.encode(tags),
'state' , ((delay > 0) and 'scheduled') or 'waiting',
'worker' , '',
'expires' , 0,
'queue' , self.name,
'retries' , retries,
'remaining', retries,
'time' , string.format("%.20f", now))
local job_fields = {
'jid' , jid,
'klass' , klass,
'data' , raw_data,
'priority' , priority,
'tags' , cjson.encode(tags),
'state' , ((delay > 0) and 'scheduled') or 'waiting',
'worker' , '',
'expires' , 0,
'queue' , self.name,
'retries' , retries,
'remaining', retries,
'time' , string.format("%.20f", now)
}
if options['backoff'] then
table.insert(job_fields, 'backoff')
table.insert(job_fields, cjson.encode(options['backoff']))
end
Comment on lines +1604 to +1607
Copy link

@coderabbitai coderabbitai bot Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Avoid double-encoding and ignore empty backoff.

options.backoff arrives JSON-encoded from PHP; encoding again stores a JSON string literal. Decode first and only persist non-empty tables.

-  if options['backoff'] then
-    table.insert(job_fields, 'backoff')
-    table.insert(job_fields, cjson.encode(options['backoff']))
-  end
+  if options['backoff'] then
+    local raw = options['backoff']
+    local backoff = (type(raw) == 'string') and (cjson.decode(raw) or {}) or raw
+    if type(backoff) == 'table' and next(backoff) ~= nil then
+      table.insert(job_fields, 'backoff')
+      table.insert(job_fields, cjson.encode(backoff))
+    end
+  end
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if options['backoff'] then
table.insert(job_fields, 'backoff')
table.insert(job_fields, cjson.encode(options['backoff']))
end
if options['backoff'] then
local raw = options['backoff']
local backoff = (type(raw) == 'string') and (cjson.decode(raw) or {}) or raw
if type(backoff) == 'table' and next(backoff) ~= nil then
table.insert(job_fields, 'backoff')
table.insert(job_fields, cjson.encode(backoff))
end
end
🤖 Prompt for AI Agents 
In src/qless-core/qless.lua around lines 1604-1607, options['backoff'] is
arriving as a JSON-encoded string so re-encoding it stores a JSON string
literal; decode it first (use pcall/cjson.decode), verify the decoded value is a
non-empty table (next(decoded) ~= nil), and only then add 'backoff' and the
properly encoded JSON of that table to job_fields; if decode fails or the table
is empty, do not insert anything.

redis.call('hmset', QlessJob.ns .. jid, job_fields)
Comment on lines +1590 to +1608
Copy link

@coderabbitai coderabbitai bot Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix pipeline breaker: HMSET with a table (must unpack).

This is the CI error (“Lua redis() command arguments must be strings or integers”). HMSET needs flat field/value pairs.

-  redis.call('hmset', QlessJob.ns .. jid, job_fields)
+  redis.call('hmset', QlessJob.ns .. jid, unpack(job_fields))
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
local job_fields = {
'jid' , jid,
'klass' , klass,
'data' , raw_data,
'priority' , priority,
'tags' , cjson.encode(tags),
'state' , ((delay > 0) and 'scheduled') or 'waiting',
'worker' , '',
'expires' , 0,
'queue' , self.name,
'retries' , retries,
'remaining', retries,
'time' , string.format("%.20f", now)
}
if options['backoff'] then
table.insert(job_fields, 'backoff')
table.insert(job_fields, cjson.encode(options['backoff']))
end
redis.call('hmset', QlessJob.ns .. jid, job_fields)
local job_fields = {
'jid' , jid,
'klass' , klass,
'data' , raw_data,
'priority' , priority,
'tags' , cjson.encode(tags),
'state' , ((delay > 0) and 'scheduled') or 'waiting',
'worker' , '',
'expires' , 0,
'queue' , self.name,
'retries' , retries,
'remaining', retries,
'time' , string.format("%.20f", now)
}
if options['backoff'] then
table.insert(job_fields, 'backoff')
table.insert(job_fields, cjson.encode(options['backoff']))
end
redis.call('hmset', QlessJob.ns .. jid, unpack(job_fields))
🤖 Prompt for AI Agents 
In src/qless-core/qless.lua around lines 1590-1608, HMSET is being called with
the job_fields table itself which causes the Redis Lua client error; change the
call to pass the field/value pairs as separate arguments by unpacking the
job_fields table (use table.unpack(job_fields)) so HMSET receives flat
string/integer args instead of a table.


for i, j in ipairs(depends) do
local state = redis.call('hget', QlessJob.ns .. j, 'state')
Expand Down Expand Up @@ -1954,7 +1963,34 @@ function QlessQueue:invalidate_locks(now, count)
redis.call('zadd', 'ql:failed-jobs-list', now, jid)
clearOldFailedJobs(now)
else
table.insert(jids, jid)
local backoff_json = redis.call('hget', Qless.ns .. jid, 'backoff')
local backoff_config = {}
if backoff_json then
backoff_config = cjson.decode(backoff_json)
end

Comment on lines +1966 to +1971
Copy link

@coderabbitai coderabbitai bot Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Read backoff from the correct job key.

Using Qless.ns builds “ql:”, not the job hash.

-        local backoff_json = redis.call('hget', Qless.ns .. jid, 'backoff')
+        local backoff_json = redis.call('hget', QlessJob.ns .. jid, 'backoff')
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
local backoff_json = redis.call('hget', Qless.ns .. jid, 'backoff')
local backoff_config = {}
if backoff_json then
backoff_config = cjson.decode(backoff_json)
end
local backoff_json = redis.call('hget', QlessJob.ns .. jid, 'backoff')
local backoff_config = {}
if backoff_json then
backoff_config = cjson.decode(backoff_json)
end
🤖 Prompt for AI Agents 
In src/qless-core/qless.lua around lines 1966–1971, the code reads backoff JSON
from redis.call('hget', Qless.ns .. jid, 'backoff') which concatenates the
namespace and jid (producing "ql:<jid>") instead of the job hash key; change the
key to the job hash (e.g. Qless.ns .. 'jobs:' .. jid or use the existing helper
like Qless.jobKey(jid) if available) so the hget targets the correct job hash,
then decode backoff_json as before.

local initial_delay = tonumber(backoff_config['initial_delay'])
local backoff_factor = tonumber(backoff_config['factor'])
if initial_delay == nil then
initial_delay = tonumber(Qless.config.get('backoff-initial-delay', 0))
end
if backoff_factor == nil then
backoff_factor = tonumber(Qless.config.get('backoff-factor', 3))
end
if initial_delay == 0 then
table.insert(jids, jid)
else
local job = Qless.job(jid)
local job_history = job:history()
local retry_count = #job_history - 1
if retry_count < 0 then retry_count = 0 end

local delay = initial_delay * (backoff_factor ^ retry_count)

Comment on lines +1972 to +1989
Copy link

@coderabbitai coderabbitai bot Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Derive retry_count from retries/remaining, not history length.

History length is not equal to attempts and can explode delays. Use (retries - remaining), clamped to >= 0.

-            local job = Qless.job(jid)
-            local job_history = job:history()
-            local retry_count = #job_history - 1
-            if retry_count < 0 then retry_count = 0 end
-
-            local delay = initial_delay * (backoff_factor ^ retry_count)
+            local total_retries = tonumber(redis.call('hget', QlessJob.ns .. jid, 'retries')) or 0
+            local remaining     = tonumber(redis.call('hget', QlessJob.ns .. jid, 'remaining')) or 0
+            local retry_count   = math.max(0, total_retries - remaining)
+            local delay         = initial_delay * (backoff_factor ^ retry_count)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
local initial_delay = tonumber(backoff_config['initial_delay'])
local backoff_factor = tonumber(backoff_config['factor'])
if initial_delay == nil then
initial_delay = tonumber(Qless.config.get('backoff-initial-delay', 0))
end
if backoff_factor == nil then
backoff_factor = tonumber(Qless.config.get('backoff-factor', 3))
end
if initial_delay == 0 then
table.insert(jids, jid)
else
local job = Qless.job(jid)
local job_history = job:history()
local retry_count = #job_history - 1
if retry_count < 0 then retry_count = 0 end
local delay = initial_delay * (backoff_factor ^ retry_count)
local initial_delay = tonumber(backoff_config['initial_delay'])
local backoff_factor = tonumber(backoff_config['factor'])
if initial_delay == nil then
initial_delay = tonumber(Qless.config.get('backoff-initial-delay', 0))
end
if backoff_factor == nil then
backoff_factor = tonumber(Qless.config.get('backoff-factor', 3))
end
if initial_delay == 0 then
table.insert(jids, jid)
else
local total_retries = tonumber(redis.call('hget', QlessJob.ns .. jid, 'retries')) or 0
local remaining = tonumber(redis.call('hget', QlessJob.ns .. jid, 'remaining')) or 0
local retry_count = math.max(0, total_retries - remaining)
local delay = initial_delay * (backoff_factor ^ retry_count)
🤖 Prompt for AI Agents 
In src/qless-core/qless.lua around lines 1972 to 1989, the code computes
retry_count from job history length which can be incorrect and inflate backoff;
instead fetch the job's configured retries and current remaining attempts and
compute retry_count = math.max(0, retries - remaining). Update the code to get
numeric retries and remaining from the job (e.g. call the appropriate job API to
read retries and remaining, coerce to numbers), clamp to >= 0, and use that
retry_count when calculating delay.

self.locks.remove(jid)
self.scheduled.add(now + delay, jid)
redis.call('hset', QlessJob.ns .. jid, 'state', 'scheduled')
end
end
end
end
Expand Down
Loading