Merge pull request ceph#60822 from perezjosibm/wip-perezjos-balance-cpu

[vstart]: add --crimson-balance-cpu option to set CPU distribution policy

Author: perezjosibm

doc/dev/crimson/BalanceCPUCrimson.md

@@ -0,0 +1,75 @@
+# Balance CPU Crimson.
+
+----------
+
+We introduced the following utilities to help analysing the Performance impact of two strategies for
+allocation of CPU cores to Seastar reactor threads. This is limited to a single host deployment at the moment.
+
+- OSD-based: this consists on allocating CPU cores from the same NUMA socket to the same OSD.
+  for simplicity, if the OSD id is even, all its reactor threads are allocated to NUMA socket 0, and 
+  consequently if the OSD is is odd, all its reactor threads are allocated to NUMA socket 1.
+
+- NUMA socket based: this consists of allocating evenly CPU cores from each NUMA socket to the reactors, so 
+  all the OSD end up with reactor on both NUMA sockets.
+
+A new option `--crimson-balance-cpu <osd|socket>` has been implemented in `vstart.sh` to support these strategies.
+
+Worth pointing out, there are *three* CPU allocation strategies:
+
+- when the new flag is not specified (default), Seastar reactors to use CPUs in ascending contiguous order (unbalanced across sockets),
+- osd: distribute across sockets uniformly, don't split within an OSD,
+- socket: distribute across sockets uniformly, split within an OSD.
+
+The utilities introduced are:
+
+- `balance-cpu.py`: a stand-alone script to produce the list of CPU core ids to use by `vstart.sh` when allocating
+  Seastar reactor threads. It uses as input the .json produced by `lscpu.py`.
+- `lscpu.py`: a Python module to parse the .json file created by `lscpu --json`. This produces a Python dictionary
+  with the NUMA details, that is, number of sockets, range of CPU core ids (physical and HT-siblings).
+- `tasksetcpu.py`: a stand-alone script to produce a grid showing the current CPU allocation, useful to quickly
+  visualise the allocation strategy.
+
+## Usage:
+
+The following is a typical example of creating a cluster with three OSDs and three reactors per OSD, and
+the desired CPU allocation policy:
+
+```
+# MDS=0 MON=1 OSD=3 MGR=1 /ceph/src/vstart.sh --new -x --localhost --without-dashboard --cyanstore --redirect-output --crimson --crimson-smp 3 --no-restart --crimson-balance-cpu osd
+```
+
+The following is the corresponding CPU distribution:
+
+![cyan_3osd_3react_bal_osd](./cyan_3osd_3react_bal_osd.png)
+
+The following snippet shows the typical usage of the `balance-cpu.py` script:
+
+```
+lscpu --json > /tmp/numa_nodes.json
+python3 ${CEPH_DIR}/../src/tools/contrib/balance-cpu.py -o $CEPH_NUM_OSD -r $crimson_smp \
+  -b $balance_strategy -u /tmp/numa_nodes.json > /tmp/numa_args.out
+```
+* the accepted balance strategies are "osd" or "socket".
+* the file produced `/tmp/numa_args.out` contains the list of CPU ids that `vstart.sh` consumes to issue the corresponding ceph configuration commands.
+
+The grid can be printed as follows:
+
+```
+  [ ! -f "${NUMA_NODES_OUT}" ] && lscpu --json > ${NUMA_NODES_OUT}
+  python3 /ceph/src/tools/contrib/tasksetcpu.py -c $TEST_NAME -u ${NUMA_NODES_OUT} -d ${RUN_DIR}
+```
+
+## Performance
+
+The following charts show the comparison of IOPs for the three CPU allocation policies: default
+(contiguous allocation, no balance), OSD-based, NUMA socket-based. It is interesting to note that
+there does not seem to be any significant throughput degradation, for this small configuration 
+(3 OSD, 3 reactors). However, the OSD-based allocation requires higher memory utilisation than the other 
+two configurations, which is an interesting finding and requires further investigation.
+
+
+![cyan_3osd_3react_bal_vs_unbal_4krandread_iops_vs_lat](./cyan_3osd_3react_bal_vs_unbal_4krandread_iops_vs_lat.png)
+
+![cyan_3osd_3react_bal_vs_unbal_4krandread_osd_cpu](./cyan_3osd_3react_bal_vs_unbal_4krandread_osd_cpu.png)
+
+![cyan_3osd_3react_bal_vs_unbal_4krandread_osd_mem](./cyan_3osd_3react_bal_vs_unbal_4krandread_osd_mem.png)

doc/dev/crimson/cyan_3osd_3react_bal_osd.png

@@ -234,3 +234,8 @@ else
     [ $stop_rgw -eq 1 ] && do_killall radosgw lt-radosgw apache2
     [ $stop_cephadm -eq 1 ] && do_killcephadm
 fi
+
+# Check whether the --crimson-balance-cpu option was used, if so remove any auxiliary files left:
+if [ "$ceph_osd" == "crimson-osd" ] && [ -f /tmp/numa_args_*.out ]; then
+    rm -f /tmp/numa_args_*.out
+fi

doc/dev/crimson/cyan_3osd_3react_bal_vs_unbal_4krandread_iops_vs_lat.png

@@ -8,3 +8,21 @@ Please do not assume any level of support.  Your mileage may vary.
 
 Each file's header must include a tracker number and an author signed-off-by
 line.
+
+
+- balance-cpu.py. An utility to distribute the Seastar reactor threads over the
+  (physical) CPU cores, according to two strategies:
+  - OSD-based (default): allocates all the reactors of the same OSD in the same 
+    NUMA socket,
+  - NUMA socket: distributes the reactors of each OSD evenly in the NUMA sockets
+    (normally two), so every OSD ends up with reactors running on both NUMA sockets.
+
+- lscpu.py. A Python module to parse the output of  ``lscpu --json`` into a dictionary
+  which is used by balance-cpu and tasksetcpu.py.
+
+- tasksetcpu.py. an utility to print a grid showing the current CPU core allocation
+  of Seastar reactors. Useful to validate that the allocation strategy is correct.
+
+For further details, please see *BalanceCPUCrimson.md* in doc/dev/crimson.
+
+