doc/dev/crimson: clarify and rearrange for userability

Signed-off-by: gitkenan <[email protected]>

Author: gitkenan

doc/dev/crimson/crimson.rst

@@ -16,8 +16,7 @@ Building Crimson
 Crimson is not enabled by default. Enable it at build time by running::
 
   $ WITH_SEASTAR=true ./install-deps.sh
-  $ mkdir build && cd build
-  $ cmake -DWITH_SEASTAR=ON ..
+  $ ./do_cmake.sh -DWITH_SEASTAR=ON
 
 Please note, `ASan`_ is enabled by default if Crimson is built from a source
 cloned using ``git``.
@@ -28,7 +27,7 @@ Testing crimson with cephadm
 ===============================
 
 The Ceph CI/CD pipeline builds containers with
-``crimson-osd`` subsitituted for ``ceph-osd``.
+``crimson-osd`` substituted for ``ceph-osd``.
 
 Once a branch at commit <sha1> has been built and is available in
 ``shaman``, you can deploy it using the cephadm instructions outlined
@@ -44,27 +43,6 @@ use a Crimson build:
 You'll likely need to supply the ``--allow-mismatched-release`` flag to
 use a non-release branch.
 
-Additionally, prior to deploying OSDs, you'll need to
-`Configure Crimson with Bluestore`_ and enable Crimson to
-direct the default pools to be created as Crimson pools.  From the cephadm shell run:
-
-.. prompt:: bash #
-
-   ceph config set global 'enable_experimental_unrecoverable_data_corrupting_features' crimson
-   ceph osd set-allow-crimson --yes-i-really-mean-it
-   ceph config set mon osd_pool_default_crimson true
-
-The first command enables the ``crimson`` experimental feature.  Crimson
-is highly experimental, and malfunctions including crashes
-and data loss are to be expected.
-
-The second enables the ``allow_crimson`` OSDMap flag.  The monitor will
-not allow ``crimson-osd`` to boot without that flag.
-
-The last causes pools to be created by default with the ``crimson`` flag.
-Crimson pools are restricted to operations supported by Crimson.
-``Crimson-osd`` won't instantiate PGs from non-Crimson pools.
-
 Configure Crimson with Bluestore
 ================================
 
@@ -76,7 +54,7 @@ one of the two following configuration options:
 
    #. These two options, along with ``crimson_alien_op_num_threads``,
       can't be changed after deployment.
-   #. `vstart.sh`_ sets this options using the ``--crimson-smp`` flag.
+   #. `vstart.sh`_ sets these options using the ``--crimson-smp`` flag.
 
 
 1) ``crimson_seastar_num_threads``
@@ -103,7 +81,7 @@ one of the two following configuration options:
 
 2) ``crimson_seastar_cpu_cores`` and ``crimson_alien_thread_cpu_cores``.
 
-   Explictily set the CPU core allocation for each ``crimson-osd``
+   Explicitly set the CPU core allocation for each ``crimson-osd``
    and for the BlueStore back end. It's recommended for each set to be mutually exclusive.
 
    For example, for deploying two nodes with eight CPU cores and two OSDs each:
@@ -144,86 +122,38 @@ one of the two following configuration options:
 Running Crimson
 ===============
 
-As you might expect, Crimson does not yet have as extensive a feature set as does ``ceph-osd``.
-
-object store backend
---------------------
-
-At the moment, ``crimson-osd`` offers both native and alienized object store
-backends. The native object store backends perform IO using the SeaStar reactor.
-They are:
-
-.. describe:: cyanstore
-
-   CyanStore is modeled after memstore in the classic OSD.
-
-.. describe:: seastore
-
-   Seastore is still under active development.
-
-The alienized object store backends are backed by a thread pool, which
-is a proxy of the alienstore adaptor running in Seastar. The proxy issues
-requests to object stores running in alien threads, i.e., worker threads not
-managed by the Seastar framework. They are:
-
-.. describe:: memstore
-
-   The memory backed object store
-
-.. describe:: bluestore
-
-   The object store used by the classic ``ceph-osd``
-
-daemonize
----------
+.. note::
+   Crimson is in a tech preview stage. 
+   As you might expect, Crimson does not yet have as extensive a feature set as does ceph-osd. 
+   Malfunctions including crashes and data loss are to be expected. 
 
-Unlike ``ceph-osd``, ``crimson-osd`` does not daemonize itself even if the
-``daemonize`` option is enabled. In order to read this option, ``crimson-osd``
-needs to ready its config sharded service, but this sharded service lives
-in the Seastar reactor. If we fork a child process and exit the parent after
-starting the Seastar engine, that will leave us with a single thread which is
-a replica of the thread that called `fork()`_. Tackling this problem in Crimson
-would unnecessarily complicate the code.
+Enabling Crimson
+================
 
-Since supported GNU/Linux distributions use ``systemd``, which is able to
-daemonize the application, there is no need to daemonize ourselves. 
-Those using sysvinit can use ``start-stop-daemon`` to daemonize ``crimson-osd``.
-If this is does not work out, a helper utility may be devised.
+After building Crimson and starting your cluster, but prior to deploying OSDs, you'll need to
+`Configure Crimson with Bluestore`_ and enable Crimson to
+direct the default pools to be created as Crimson pools.  You can proceed by running the following after you have a running cluster:
 
-.. _fork(): http://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html
+.. note::
+   `vstart.sh`_ enables crimson automatically when `--crimson` is used.
 
-logging
--------
+.. prompt:: bash #
 
-``Crimson-osd`` currently uses the logging utility offered by Seastar. See
-``src/common/dout.h`` for the mapping between Ceph logging levels to
-the severity levels in Seastar. For instance, messages sent to ``derr``
-will be issued using ``logger::error()``, and the messages with a debug level
-greater than ``20`` will be issued using ``logger::trace()``.
+   ceph config set global 'enable_experimental_unrecoverable_data_corrupting_features' crimson
+   ceph osd set-allow-crimson --yes-i-really-mean-it
+   ceph config set mon osd_pool_default_crimson true
 
-+---------+---------+
-| ceph    | seastar |
-+---------+---------+
-| < 0     | error   |
-+---------+---------+
-|   0     | warn    |
-+---------+---------+
-| [1, 6)  | info    |
-+---------+---------+
-| [6, 20] | debug   |
-+---------+---------+
-| >  20   | trace   |
-+---------+---------+
+The first command enables the ``crimson`` experimental feature.  
 
-Note that ``crimson-osd``
-does not send log messages directly to a specified ``log_file``. It writes
-the logging messages to stdout and/or syslog. This behavior can be
-changed using ``--log-to-stdout`` and ``--log-to-syslog`` command line
-options. By default, ``log-to-stdout`` is enabled, and ``--log-to-syslog`` is disabled.
+The second enables the ``allow_crimson`` OSDMap flag.  The monitor will
+not allow ``crimson-osd`` to boot without that flag.
 
+The last causes pools to be created by default with the ``crimson`` flag.
+Crimson pools are restricted to operations supported by Crimson.
+``Crimson-osd`` won't instantiate PGs from non-Crimson pools.
 
 vstart.sh
----------
+=========
 
 The following options can be used with ``vstart.sh``.
 
@@ -246,17 +176,17 @@ The following options can be used with ``vstart.sh``.
 
     for additional Seastar-specific command line options.
 
-``--cyanstore``
-    Use CyanStore as the object store backend.
-
-``--bluestore``
-    Use the alienized BlueStore as the object store backend. This is the default.
-
 ``--crimson-smp``
     The number of cores to use for each OSD.
     If BlueStore is used, the balance of available cores
     (as determined by `nproc`) will be assigned to the object store.
 
+``--bluestore``
+    Use the alienized BlueStore as the object store backend. This is the default (see below section on the `object store backend`_ for more details)
+
+``--cyanstore``
+    Use CyanStore as the object store backend.
+
 ``--memstore``
     Use the alienized MemStore as the object store backend.
 
@@ -279,12 +209,9 @@ The following options can be used with ``vstart.sh``.
 
 To start a cluster with a single Crimson node, run::
 
-  $  MGR=1 MON=1 OSD=1 MDS=0 RGW=0 ../src/vstart.sh -n -x \
-    --without-dashboard --cyanstore \
-    --crimson --redirect-output \
-    --osd-args "--memory 4G"
-
-Here we assign 4 GiB memory and a single thread running on core-0 to ``crimson-osd``.
+  $  MGR=1 MON=1 OSD=1 MDS=0 RGW=0 ../src/vstart.sh \
+    --without-dashboard --bluestore --crimson \
+    --redirect-output
 
 Another SeaStore example::
 
@@ -299,19 +226,93 @@ Stop this ``vstart`` cluster by running::
 
   $ ../src/stop.sh --crimson
 
+Object Store Backend
+====================
+
+At the moment, ``crimson-osd`` offers both native and alienized object store
+backends. The native object store backends perform IO using the SeaStar reactor.
+They are:
+
+.. describe:: cyanstore
+
+   CyanStore is modeled after memstore in the classic OSD.
+
+.. describe:: seastore
+
+   Seastore is still under active development.
+
+The alienized object store backends are backed by a thread pool, which
+is a proxy of the alienstore adaptor running in Seastar. The proxy issues
+requests to object stores running in alien threads, i.e., worker threads not
+managed by the Seastar framework. They are:
+
+.. describe:: memstore
+
+   The memory backend object store
+
+.. describe:: bluestore
+
+   The object store used by the classic ``ceph-osd``
+
+daemonize
+---------
+
+Unlike ``ceph-osd``, ``crimson-osd`` does not daemonize itself even if the
+``daemonize`` option is enabled. In order to read this option, ``crimson-osd``
+needs to ready its config sharded service, but this sharded service lives
+in the Seastar reactor. If we fork a child process and exit the parent after
+starting the Seastar engine, that will leave us with a single thread which is
+a replica of the thread that called `fork()`_. Tackling this problem in Crimson
+would unnecessarily complicate the code.
+
+Since supported GNU/Linux distributions use ``systemd``, which is able to
+daemonize processes, there is no need to daemonize ourselves. 
+Those using sysvinit can use ``start-stop-daemon`` to daemonize ``crimson-osd``.
+If this is does not work out, a helper utility may be devised.
+
+.. _fork(): http://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html
+
+logging
+-------
+
+``Crimson-osd`` currently uses the logging utility offered by Seastar. See
+``src/common/dout.h`` for the mapping between Ceph logging levels to
+the severity levels in Seastar. For instance, messages sent to ``derr``
+will be issued using ``logger::error()``, and the messages with a debug level
+greater than ``20`` will be issued using ``logger::trace()``.
+
++---------+---------+
+| ceph    | seastar |
++---------+---------+
+| < 0     | error   |
++---------+---------+
+|   0     | warn    |
++---------+---------+
+| [1, 6)  | info    |
++---------+---------+
+| [6, 20] | debug   |
++---------+---------+
+| >  20   | trace   |
++---------+---------+
+
+Note that ``crimson-osd``
+does not send log messages directly to a specified ``log_file``. It writes
+the logging messages to stdout and/or syslog. This behavior can be
+changed using ``--log-to-stdout`` and ``--log-to-syslog`` command line
+options. By default, ``log-to-stdout`` is enabled, and ``--log-to-syslog`` is disabled.
 Metrics and Tracing
 ===================
 
 Crimson offers three ways to report stats and metrics.
 
-pg stats reported to mgr
+PG stats reported to mgr
 ------------------------
 
 Crimson collects the per-pg, per-pool, and per-osd stats in a `MPGStats`
 message which is sent to the Ceph Managers. Manager modules can query
 them using the `MgrModule.get()` method.
 
-asock command
+Asock command
 -------------
 
 An admin socket command is offered for dumping metrics::
@@ -334,7 +335,7 @@ see `Prometheus`_ for more details.
 Profiling Crimson
 =================
 
-fio
+Fio
 ---
 
 ``crimson-store-nbd`` exposes configurable ``FuturizedStore`` internals as an
@@ -506,7 +507,7 @@ When a Seastar application crashes, it leaves us with a backtrace of addresses,
 The ``seastar-addr2line`` utility provided by Seastar can be used to map these
 addresses to functions. The script expects input on ``stdin``,
 so we need to copy and paste the above addresses, then send EOF by inputting
-``control-D`` in the terminal.  One might  use ``echo`` or ``cat`` instead`::
+``control-D`` in the terminal.  One might  use ``echo`` or ``cat`` instead::
 
   $ ../src/seastar/scripts/seastar-addr2line -e bin/crimson-osd