Skip to content

Commit 0b64d02

Browse files
committed
Using SCAN instead of KEYS at remove_queue
*** Why is this change necessary? If you have a big collection of keys to be fetched and deleted from Redis two problems will occur: 1. The server will be blocked due to the use of KEYS command since it takes a long time to return for big collections; 2. Using the splat ("*") operator for a large array raises a SystemStackError Ultimately the queue is removed but the keys remain there: ``` 2.3.8 :057 > Resque.size :test_queue => 300001 2.3.8 :058 > Resque.remove_queue :test_queue SystemStackError: stack level too deep from /root/project/vendor/bundle/ruby/2.3.0/gems/resque-unique_in_queue-2.0.1/lib/resque/uniqu e_in_queue/queue.rb:68:in `cleanup' from /root/project/vendor/bundle/ruby/2.3.0/gems/resque-unique_in_queue-2.0.1/lib/resque/uniqu e_in_queue/resque_ext/resque.rb:35:in `remove_queue_with_unique_in_queue_cleanup' from /root/project/lib/ext/hacks/resque.rb:115:in `remove_queue_rewrite' from (irb):58 from /root/project/vendor/bundle/ruby/2.3.0/gems/railties-4.2.11.1/lib/rails/commands/console. rb:110:in `start' from /root/project/vendor/bundle/ruby/2.3.0/gems/railties-4.2.11.1/lib/rails/commands/console. rb:9:in `start' from /root/project/vendor/bundle/ruby/2.3.0/gems/railties-4.2.11.1/lib/rails/commands/commands _tasks.rb:68:in `console' from /root/project/vendor/bundle/ruby/2.3.0/gems/railties-4.2.11.1/lib/rails/commands/commands _tasks.rb:39:in `run_command!' from /root/project/vendor/bundle/ruby/2.3.0/gems/railties-4.2.11.1/lib/rails/commands.rb:17:in `<top (required)>' from script/rails:9:in `require' from script/rails:9:in `<main>' 2.3.8 :059 > Resque.size :test_queue => 0 2.3.8 :060 > Resque.redis.keys("#{Resque::UniqueInQueue.configuration&.unique_in_queue_key_base}:queue:test_queue:job:*").count => 300001 ``` *** How does it address the issue? 1. Replacing KEYS by SCAN command 2. Removing splat operator. *** What side effects does this change have? The removal process for large collections will take a lot longer: ``` irb(main):001:0> Benchmark.measure { Resque.remove_queue :test_queue } => #<Benchmark::Tms:0x000055b793b3cd60 @Label="", @real=167.7889306979996, @CsTime=0.0, @cutime=0.0, @stime=0.7600000000000002, @utime=11.500000000000002, @ToTal=12.260000000000002> ```
1 parent 3636a72 commit 0b64d02

File tree

1 file changed

+3
-2
lines changed

1 file changed

+3
-2
lines changed

lib/resque/unique_in_queue/queue.rb

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,8 +64,9 @@ def destroy(queue, klass, *args)
6464
end
6565

6666
def cleanup(queue)
67-
keys = redis.keys("#{Resque::UniqueInQueue.configuration&.unique_in_queue_key_base}:queue:#{queue}:job:*")
68-
redis.del(*keys) if keys.any?
67+
pattern = "#{Resque::UniqueInQueue.configuration&.unique_in_queue_key_base}:queue:#{queue}:job:*"
68+
keys = redis.scan_each(match: pattern, count: 1_000_000).to_a
69+
redis.del(keys) if keys.any?
6970
end
7071

7172
private

0 commit comments

Comments
 (0)