rake queuebus:subscribe not creating the subscribers

[imi56]

There is no solution available on stackoverflow.
Gem versions:
rails version: 6.0

sidekiq (5.2.8)
      connection_pool (~> 2.2, >= 2.2.2)
      rack (< 2.1.0)
      rack-protection (>= 1.5.0)
      redis (>= 3.3.5, < 5)
sidekiq-bus (0.9.0)
      queue-bus (>= 0.7, < 1)
      sidekiq (~> 5.0, >= 3.0.0)
      sidekiq-scheduler (~> 3.0)

Here is the full trace of rake queuebus:subscribe

** Invoke queuebus:subscribe (first_time)
** Invoke queuebus:preload (first_time)
** Execute queuebus:preload
** Execute queuebus:subscribe
rake aborted!
No subscriptions created
/Users/Imi/.rvm/gems/ruby-2.6.0/gems/queue-bus-0.9.0/lib/queue_bus/tasks.rb:10:in `block (2 levels) in <top (required)>'
/Users/Imi/.rvm/gems/ruby-2.6.0/gems/rake-13.0.1/lib/rake/task.rb:281:in `block in execute'
/Users/Imi/.rvm/gems/ruby-2.6.0/gems/rake-13.0.1/lib/rake/task.rb:281:in `each'
/Users/Imi/.rvm/gems/ruby-2.6.0/gems/rake-13.0.1/lib/rake/task.rb:281:in `execute'
/Users/Imi/.rvm/gems/ruby-2.6.0/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/Users/Imi/.rvm/rubies/ruby-2.6.0/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
/Users/Imi/.rvm/gems/ruby-2.6.0/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/Users/Imi/.rvm/gems/ruby-2.6.0/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke'
/Users/Imi/.rvm/gems/ruby-2.6.0/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task'
/Users/Imi/.rvm/gems/ruby-2.6.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/Users/Imi/.rvm/gems/ruby-2.6.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `each'
/Users/Imi/.rvm/gems/ruby-2.6.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level'
/Users/Imi/.rvm/gems/ruby-2.6.0/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads'
/Users/Imi/.rvm/gems/ruby-2.6.0/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level'
/Users/Imi/.rvm/gems/ruby-2.6.0/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run'
/Users/Imi/.rvm/gems/ruby-2.6.0/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling'
/Users/Imi/.rvm/gems/ruby-2.6.0/gems/rake-13.0.1/lib/rake/application.rb:80:in `run'
/Users/Imi/.rvm/gems/ruby-2.6.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/Users/Imi/.rvm/gems/ruby-2.6.0/bin/rake:23:in `load'
/Users/Imi/.rvm/gems/ruby-2.6.0/bin/rake:23:in `<main>'
/Users/Imi/.rvm/gems/ruby-2.6.0/bin/ruby_executable_hooks:24:in `eval'
/Users/Imi/.rvm/gems/ruby-2.6.0/bin/ruby_executable_hooks:24:in `<main>'
Tasks: TOP => queuebus:subscribe

I am publishing the event (QueueBus.publish('concern_created', phone: '0000000005')) from rails console. As soon as I publish the event processed count in sidekiq is increased by one.

subscribers/base_subscriber.rb

class BaseSubscriber
  include QueueBus::Subscriber
  application :concern_api
end

subscribers/concern_subscriber.rb

class ConcernSubscriber < BaseSubscriber
  subscribe_queue :concern_events, :concern_created

  def concern_created(attr)
    put "******" * 20
    PartnerInvitation.create!({phone: attr['phone']})
  end
end

initializers/sidekiq.rb

require 'sidekiq'
require 'sidekiq/web'

Sidekiq.configure_server do |config|
  config.redis = { url: Rails.application.config_for(:redis)[:url] }
end

Sidekiq.configure_client do |config|
  config.redis = { url: Rails.application.config_for(:redis)[:url] }
end

# https://github.com/queue-bus/sidekiq-bus
if Sidekiq.server?
  # Load the queues into sidekiq:
  weights = {
    'app_events'    => 10,
    'app_heartbeat' => 3,
    'app_refresh'   => 1
  }
  Sidekiq.options[:queues] = SidekiqBus.generate_weighted_queues(overrides: weights, default: 2)
end

config/sidekiq.yml

:verbose: true
:timeout: 25
:queues:
  - critical
  - concern_events
  - default
  - low

Note: rake queuebus:queues returns only bus_incoming

[choubacha]

Hi @imi56

That is a little confusing and I hope we can sort it out! Thank you so much for taking the time to write up such a thorough issue!

Which environment is this loading? Dev? If so, are the subscribers set up to autoload or eagerload?

If they are autoloading, then they may not register in the development environment as that typically autoload all constants and it's the file loading that adds them to the set of subscriptions it could be that they haven't registered themselves yet. In the service that we use that uses subscribers, these files are required explicitly so that they load before we run subscribe.

[imi56]

Hi @kbacha ,
Yes, I'm trying in dev env. Also, rails 6 uses Zeitwerk mode for autoload. In dev env eager loading is disabled by default(config.eager_load = false).
confing/application.rb has following config

config.autoloader = :zeitwerk
config.load_defaults 6.0

By default, the autoload paths of an application consist of all the subdirectories of app that exist when the application boots ---except for assets, javascripts, views,--- plus the autoload paths of engines it might depend on. (https://guides.rubyonrails.org/autoloading_and_reloading_constants.html#autoload-paths)

So my subscribes should also autoload.

[choubacha]

@imi56 Right, autoload may break this since it will only load the files at the time that the constant is referenced, you'll want to make sure that the subscribers are eagerloaded.

edit: This is because the queue-bus framework won't know they exist until the files are loaded. It cannot reference the constants because it doesn't know yet that they are subscribers until the subscribe_queue method is called on the class:

class ConcernSubscriber < BaseSubscriber
  subscribe_queue :concern_events, :concern_created # this line needs to run before the task runs

In our codebase, we do this by requiring all the subscribers via a boot.rb (or an initializer) that we require in our sidekiq.rb. You can certainly do this from an initializer.

In higher environments, which eagerload by default, this should work correctly. You can verify this by turning off autoload in dev and enabling eagerloading.

[imi56]

No luck!
I added Dir["#{Rails.root}/app/subscribers/*.rb"].each { |file| require file } at top of the initializers/sidekiq.rb.
Also, made config.eager_load = true in dev.
Getting same error.

[choubacha]

Oh! Ok, so it seems like I may have forgotten to include something else. In addition to making sure that the files are loaded we should also enhance the task queuebus:preload to require the environment rake task. Here's what we have in our code base:

Rake::Task["queuebus:preload"].enhance do
  # QueueBus doesn't load the environment by default. We want to do that so that the bus
  # is set up correctly during any queuebus loading.
  Rake::Task["environment"].invoke
end

[choubacha]

Thinking on this more, even if the above solves your issue, I think i'm gonna leave this issue open a bit longer. We should likely have some documentation on setting up with Rails and possibly even a queue-bus-rails gem to help with it.

[imi56]

Oh! Ok, so it seems like I may have forgotten to include something else. In addition to making sure that the files are loaded we should also enhance the task queuebus:preload to require the environment rake task. Here's what we have in our code base:

Rake::Task["queuebus:preload"].enhance do
  # QueueBus doesn't load the environment by default. We want to do that so that the bus
  # is set up correctly during any queuebus loading.
  Rake::Task["environment"].invoke
end

Perfect! working peacefully.
And yes please add this to doc, count me in for any help though I'm doing this first time ever.