osd/scrub: disable high work-queue priority for h.p. scrub

Prior to this fix, scrub-related messages were queued with
one of four priorities:

- if part of a regular, periodic scrub, the messages were queued
  with a very low priority (osd_scrub_priority. Default: 5), unless:
- if the pool configuration specified a higher priority for the
  scrub, the messages were queued with that priority
  (the SCRUB_PRIORITY pool configuration option);
- if a client operation was waiting for the scrub to complete,
  the messages were queued with the higher priority of the
  client ops (osd_client_op_priority. Default: 63);
  and if the scrub was a high-priority one, e.g. initiated by the
  operator, the messages were queued with the highest priority
  (osd_requested_scrub_priority. Default: 120).

That last priority is too high, as scrubs should not have
precedence over client ops. It is even more disruptive if the
cluster is set to have only manually initiated scrubs, and
all scrubs are high-priority ones.

This patch removes the osd_requested_scrub_priority. Instead,
all scrub messages are queued according to the first three
rules above.

Signed-off-by: Ronen Friedman <[email protected]>

Author: ronen-fr

src/common/options/global.yaml.in

@@ -3705,11 +3705,8 @@ options:
   type: uint
   level: advanced
   default: 5
-  fmt_desc: The priority set for user requested scrub on the work queue.  If
-    this value were to be smaller than ``osd_client_op_priority`` it
-    can be boosted to the value of ``osd_client_op_priority`` when
-    scrub is blocking client operations.
-  with_legacy: true
+  fmt_desc: deprecated.  Use ``osd_scrub_priority`` instead.
+  with_legacy: false
 - name: osd_recovery_priority
   type: uint
   level: advanced

src/osd/scrubber/pg_scrubber.cc

@@ -1671,15 +1671,7 @@ void PgScrubber::set_op_parameters(ScrubPGPreconds pg_cond)
     m_flags.deep_scrub_on_error = false;
   }
 
-  if (ScrubJob::has_high_queue_priority(m_active_target->urgency())) {
-    // specific high priority scrubs - high queue priority
-    /// \todo consider - do we really want high queue priority for any scrub?
-    m_flags.priority = get_pg_cct()->_conf->osd_requested_scrub_priority;
-  } else {
-    // regular, low-priority scrubs - low queue priority - unless blocking
-    // client I/O
-    m_flags.priority = m_pg->get_scrub_priority();
-  }
+  m_flags.priority = m_pg->get_scrub_priority();
 
   // The publishing here is required for tests synchronization.
   // The PG state flags were modified.