Merge pull request ceph#53988 from ljflores/wip-read-balancer-mgr-module

Author: ljflores

PendingReleaseNotes

@@ -113,6 +113,10 @@ CephFS: Disallow delegating preallocated inode ranges to clients. Config
 * RBD: The ``try-netlink`` mapping option for rbd-nbd has become the default
   and is now deprecated. If the NBD netlink interface is not supported by the
   kernel, then the mapping is retried using the legacy ioctl interface.
+* RADOS: Read balancing may now be managed automatically via the balancer
+  manager module. Users may choose between two new modes: ``upmap-read``, which
+  offers upmap and read optimization simultaneously, or ``read``, which may be used
+  to only optimize reads. For more detailed information see https://docs.ceph.com/en/latest/rados/operations/read-balancer/#online-optimization.
 
 >=18.0.0
 

doc/dev/balancer-design.rst

@@ -55,4 +55,3 @@ Plans for the Next Version
 --------------------------
 
 1. Improve behavior for heterogeneous OSDs in a pool
-2. Offer read balancing as an online option to the balancer manager module

doc/rados/operations/balancer.rst

@@ -21,9 +21,9 @@ To check the current status of the balancer, run the following command:
 Automatic balancing
 -------------------
 
-When the balancer is in ``upmap`` mode, the automatic balancing feature is
-enabled by default. For more details, see :ref:`upmap`.  To disable the
-balancer, run the following command:
+When the balancer is in ``upmap`` mode, which is the default, the automatic
+upmap balancing feature is enabled.  For more details, see :ref:`upmap`.
+To disable the balancer, run the following command:
 
    .. prompt:: bash $
 
@@ -34,6 +34,10 @@ The balancer mode can be changed from ``upmap`` mode to ``crush-compat`` mode.
 ``crush-compat`` mode, the balancer automatically makes small changes to the
 data distribution in order to ensure that OSDs are utilized equally.
 
+Additional modes include ``upmap-read`` and ``read``. ``upmap-read`` mode
+combines the upmap balancer with the read balancer so that both writes
+and reads are optimized. ``read`` mode can be used when only read optimization
+is desired. For more details, see :ref:`read_balancer`.
 
 Throttling
 ----------
@@ -102,7 +106,7 @@ and then run the following command:
 Modes
 -----
 
-There are two supported balancer modes:
+There are four supported balancer modes:
 
 #. **crush-compat**. This mode uses the compat weight-set feature (introduced
    in Luminous) to manage an alternative set of weights for devices in the
@@ -135,13 +139,45 @@ There are two supported balancer modes:
 
    To use ``upmap``, all clients must be Luminous or newer.
 
-The default mode is ``upmap``. The mode can be changed to ``crush-compat`` by
-running the following command:
+#. **read**. In Reef and later releases, the OSDMap can store explicit
+   mappings for individual primary OSDs as exceptions to the normal CRUSH
+   placement calculation. These ``pg-upmap-primary`` entries provide fine-grained
+   control over primary PG mappings. This mode optimizes the placement of individual
+   primary PGs in order to achieve balanced reads, or primary PGs, in a cluster.
+   In ``read`` mode, upmap behavior is not excercised, so this mode is best for
+   uses cases in which only read balancing is desired.
+
+   To use ``pg-upmap-primary``, all clients must be Reef or newer. For more
+   details about client compatibility, see :ref:`read_balancer`.
+
+#. **upmap-read**. This balancer mode combines optimization benefits of
+   both ``upmap`` and ``read`` mode. Like in ``read`` mode, ``upmap-read``
+   makes use of ``pg-upmap-primary``. As such, only Reef and later clients
+   are compatible. For more details about client compatibility, see
+   :ref:`read_balancer`.
+
+   ``upmap-read`` is highly recommended for achieving the ``upmap`` mode's
+   offering of balanced PG distribution as well as the ``read`` mode's
+   offering of balanced reads.
+
+The default mode is ``upmap``. The mode can be changed to ``crush-compat`` by running the following command:
 
    .. prompt:: bash $
 
       ceph balancer mode crush-compat
 
+The mode can be changed to ``read`` by running the following command:
+
+   .. prompt:: bash $
+
+      ceph balancer mode read
+
+The mode can be changed to ``upmap-read`` by running the following command:
+
+   .. prompt:: bash $
+
+      ceph balancer mode upmap-read
+
 Supervised optimization
 -----------------------
 

doc/rados/operations/read-balancer.rst

@@ -17,9 +17,50 @@ you may want to try improving your read performance with the read balancer.
 Online Optimization
 ===================
 
-At present, there is no online option for the read balancer. However, we plan to add
-the read balancer as an option to the :ref:`balancer` in the next Ceph version
-so it can be enabled to run automatically in the background like the upmap balancer.
+Enabling
+--------
+
+To enable automatic read balancing, you must turn on the *balancer module*
+(enabled by default in new clusters) and set the mode to ``read`` or ``upmap-read``:
+
+.. prompt:: bash $
+
+   ceph balancer on
+   ceph balancer mode <read|upmap-read>
+
+Both ``read`` and ``upmap-read`` mode make use of ``pg-upmap-primary``. In order
+to use ``pg-upmap-primary``, the cluster cannot have any pre-Reef clients.
+
+If you want to use a different balancer or if you want to make your
+own custom ``pg-upmap-primary`` entries, you might want to turn off the balancer in
+order to avoid conflict:
+
+.. prompt:: bash $
+
+   ceph balancer off
+
+To allow use of the new feature on an existing cluster, you must restrict the
+cluster to supporting only Reef (and newer) clients.  To do so, run the
+following command:
+
+.. prompt:: bash $
+
+   ceph osd set-require-min-compat-client reef
+
+This command will fail if any pre-Reef clients or daemons are connected to
+the monitors. To see which client versions are in use, run the following
+command:
+
+.. prompt:: bash $
+
+   ceph features
+
+Balancer Module
+---------------
+
+The `balancer` module for ``ceph-mgr`` will automatically balance the number of
+primary PGs per OSD if set to ``read`` or ``upmap-read`` mode. See :ref:`balancer`
+for more information.
 
 Offline Optimization
 ====================

qa/suites/rados/thrash/d-balancer/read.yaml

@@ -0,0 +1,10 @@
+tasks:
+- exec:
+    mon.a:
+      - ceph config set mgr mgr/balancer/log_level debug
+      - ceph balancer status
+      - ceph osd set-require-min-compat-client reef
+      - ceph balancer mode read
+      - ceph balancer on
+      - ceph balancer status
+      - ceph balancer status detail

qa/suites/rados/thrash/d-balancer/upmap-read.yaml

@@ -0,0 +1,10 @@
+tasks:
+- exec:
+    mon.a:
+      - ceph config set mgr mgr/balancer/log_level debug
+      - ceph balancer status
+      - ceph osd set-require-min-compat-client reef
+      - ceph balancer mode upmap-read
+      - ceph balancer on
+      - ceph balancer status
+      - ceph balancer status detail

src/mgr/PyOSDMap.cc

@@ -162,6 +162,36 @@ static PyObject *osdmap_calc_pg_upmaps(BasePyOSDMap* self, PyObject *args)
   return PyLong_FromLong(r);
 }
 
+static PyObject *osdmap_balance_primaries(BasePyOSDMap* self, PyObject *args)
+{
+  int pool_id;
+  BasePyOSDMapIncremental *incobj;
+  if (!PyArg_ParseTuple(args, "iO:balance_primaries",
+                        &pool_id, &incobj)) {
+    return nullptr;
+  }
+  auto check_pool = self->osdmap->get_pg_pool(pool_id);
+  if (!check_pool) {
+    derr << __func__ << " pool '" << pool_id
+         << "' does not exist" << dendl;
+    return nullptr;
+  }
+  dout(10) << __func__ << " osdmap " << self->osdmap
+           << " pool_id " << pool_id
+           << " inc " << incobj->inc
+           << dendl;
+  PyThreadState *tstate = PyEval_SaveThread();
+  OSDMap tmp_osd_map;
+  tmp_osd_map.deepish_copy_from(*(self->osdmap));
+  int r = self->osdmap->balance_primaries(g_ceph_context,
+                                 pool_id,
+                                 incobj->inc,
+				 tmp_osd_map);
+  PyEval_RestoreThread(tstate);
+  dout(10) << __func__ << " r = " << r << dendl;
+  return PyLong_FromLong(r);
+}
+
 static PyObject *osdmap_map_pool_pgs_up(BasePyOSDMap* self, PyObject *args)
 {
   int poolid;
@@ -324,6 +354,8 @@ PyMethodDef BasePyOSDMap_methods[] = {
    "Get pools that have CRUSH rules that TAKE the given root"},
   {"_calc_pg_upmaps", (PyCFunction)osdmap_calc_pg_upmaps, METH_VARARGS,
    "Calculate new pg-upmap values"},
+  {"_balance_primaries", (PyCFunction)osdmap_balance_primaries, METH_VARARGS,
+   "Calculate new pg-upmap-primary values"},
   {"_map_pool_pgs_up", (PyCFunction)osdmap_map_pool_pgs_up, METH_VARARGS,
    "Calculate up set mappings for all PGs in a pool"},
   {"_pg_to_up_acting_osds", (PyCFunction)osdmap_pg_to_up_acting_osds, METH_VARARGS,

src/osd/OSDMap.cc

@@ -5143,6 +5143,12 @@ int OSDMap::balance_primaries(
         num_changes++;
       }
     }
+  } else { // clear out any mappings that were made since the score didn't improve
+    for (auto [pg, mapped] : prim_pgs_to_check) {
+      if (mapped) {
+	pending_inc->new_pg_upmap_primary.erase(pg);
+      }
+    }
   }
 
   ldout(cct, 10) << __func__ << " num_changes " << num_changes << dendl;