Changing the throttling config when jobs are running

[roykolak]

I'm currently seeing batches of jobs that are configured to throttle dynamically get stuck when the throttle configuration changes.

For example, here's my throttle config...

    sidekiq_throttle(
      concurrency: {
        limit: ->(search_id, result_id, batch_id, dynamically_assigned_queue) {
          Search.slow_search?(search_id, batch_id) ? 1 : 5
        },
        key_suffix: ->(search_id, result_id, batch_id, dynamically_assigned_queue) {
          Search.slow_search?(search_id, batch_id) ? "#{dynamically_assigned_queue}-slow" : "#{dynamically_assigned_queue}-fast"
        }
      },
      observer: ->(strategy, *args) {
        Sidekiq.logger.warn "Throttled: #{strategy} search_id: #{args[0]} queue: #{args[3]}"
      }
    )

You can see that the jobs belong to a batch and that batch gets assigned to a random shared queue.

My question is, if Search.slow_search? returns a different value in the middle of running a batch of jobs, is that a problem? I'm seeing a lot of logging that the "concurrency" strategy prevented jobs from running after Search.slow_search? changes it's boolean value.