Commit 0b64d02
committed
Using SCAN instead of KEYS at remove_queue
*** Why is this change necessary?
If you have a big collection of keys to be fetched and deleted from
Redis two problems will occur:
1. The server will be blocked due to the use of KEYS command since it
takes a long time to return for big collections;
2. Using the splat ("*") operator for a large array raises a
SystemStackError
Ultimately the queue is removed but the keys remain there:
```
2.3.8 :057 > Resque.size :test_queue
=> 300001
2.3.8 :058 > Resque.remove_queue :test_queue
SystemStackError: stack level too deep
from /root/project/vendor/bundle/ruby/2.3.0/gems/resque-unique_in_queue-2.0.1/lib/resque/uniqu
e_in_queue/queue.rb:68:in `cleanup'
from /root/project/vendor/bundle/ruby/2.3.0/gems/resque-unique_in_queue-2.0.1/lib/resque/uniqu
e_in_queue/resque_ext/resque.rb:35:in `remove_queue_with_unique_in_queue_cleanup'
from /root/project/lib/ext/hacks/resque.rb:115:in `remove_queue_rewrite'
from (irb):58
from /root/project/vendor/bundle/ruby/2.3.0/gems/railties-4.2.11.1/lib/rails/commands/console.
rb:110:in `start'
from /root/project/vendor/bundle/ruby/2.3.0/gems/railties-4.2.11.1/lib/rails/commands/console.
rb:9:in `start'
from /root/project/vendor/bundle/ruby/2.3.0/gems/railties-4.2.11.1/lib/rails/commands/commands
_tasks.rb:68:in `console'
from /root/project/vendor/bundle/ruby/2.3.0/gems/railties-4.2.11.1/lib/rails/commands/commands
_tasks.rb:39:in `run_command!'
from /root/project/vendor/bundle/ruby/2.3.0/gems/railties-4.2.11.1/lib/rails/commands.rb:17:in
`<top (required)>'
from script/rails:9:in `require'
from script/rails:9:in `<main>'
2.3.8 :059 > Resque.size :test_queue
=> 0
2.3.8 :060 > Resque.redis.keys("#{Resque::UniqueInQueue.configuration&.unique_in_queue_key_base}:queue:test_queue:job:*").count
=> 300001
```
*** How does it address the issue?
1. Replacing KEYS by SCAN command
2. Removing splat operator.
*** What side effects does this change have?
The removal process for large collections will take a lot longer:
```
irb(main):001:0> Benchmark.measure { Resque.remove_queue :test_queue }
=> #<Benchmark::Tms:0x000055b793b3cd60 @Label="", @real=167.7889306979996, @CsTime=0.0, @cutime=0.0, @stime=0.7600000000000002, @utime=11.500000000000002, @ToTal=12.260000000000002>
```1 parent 3636a72 commit 0b64d02
1 file changed
+3
-2
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
67 | | - | |
68 | | - | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
69 | 70 | | |
70 | 71 | | |
71 | 72 | | |
| |||
0 commit comments