Improve scheduling configuration of examples_rclcpp_cbg_executor package (#331)

* Reduced the OS-specific priority of the high prio Executor thread on Linux and QNX. Chose the configuration to be below typical threaded interrupt handlers.

Signed-off-by: Ralph Lange <[email protected]>

Author: ralph-lange

rclcpp/executors/cbg_executor/README.md

@@ -29,15 +29,18 @@ ros2 run examples_rclcpp_cbg_executor ping_pong
 Example of a typical output - note the zero pongs received on the low prio path:
 
 ```
-[INFO] [..] [pong_node]: Running experiment from now on for 10s ...
-[INFO] [..] [ping_node]: Both paths: Sent out 982 of configured 1000 pings, i.e. 98%.
-[INFO] [..] [ping_node]: High prio path: Received 980 pongs, i.e. for 99% of the pings.
-[INFO] [..] [ping_node]: High prio path: Average RTT is 17.2ms.
+[INFO] [..] [pong_node]: Running experiment from now on for 10 seconds ...
+[INFO] [..] [ping_node]: Both paths: Sent out 953 of configured 1000 pings, i.e. 95%.
+[INFO] [..] [ping_node]: High prio path: Received 951 pongs, i.e. for 99% of the pings.
+[INFO] [..] [ping_node]: High prio path: Average RTT is 14.0ms.
+[INFO] [..] [ping_node]: High prio path: Jitter of RTT is 7.460ms.
 [INFO] [..] [ping_node]: Low prio path: Received 0 pongs, i.e. for 0% of the pings.
-[INFO] [..] [pong_node]: High priority executor thread ran for 9995ms.
+[INFO] [..] [pong_node]: High priority executor thread ran for 9542ms.
 [INFO] [..] [pong_node]: Low priority executor thread ran for 0ms.
 ```
 
+Note: On Linux, the two Executor threads, which are both scheduled under `SCHED_FIFO`, can consume only 95% of the CPU time due to [RT throttling](https://wiki.linuxfoundation.org/realtime/documentation/technical_basics/sched_rt_throttling).
+
 Running the two nodes in separate processes:
 
 ```bash
@@ -74,11 +77,13 @@ With these values, about (0.033s - 0.025s) / 0.010s = 80% of the ping messages o
 
 ```
 ...
-[INFO] [..] [ping_node]: Both paths: Sent out 303 of configured 303 pings, i.e. 100%.
-[INFO] [..] [ping_node]: High prio path: Received 302 pongs, i.e. for 99% of the pings.
-[INFO] [..] [ping_node]: High prio path: Average RTT is 25.2ms.
-[INFO] [..] [ping_node]: Low prio path: Received 231 pongs, i.e. for 76% of the pings.
-[INFO] [..] [ping_node]: Low prio path: Average RTT is 196.1ms.
+[INFO] [..] [ping_node]: Both paths: Sent out 294 of configured 303 pings, i.e. 97%.
+[INFO] [..] [ping_node]: High prio path: Received 293 pongs, i.e. for 99% of the pings.
+[INFO] [..] [ping_node]: High prio path: Average RTT is 26.2ms.
+[INFO] [..] [ping_node]: High prio path: Jitter of RTT is 7.654ms.
+[INFO] [..] [ping_node]: Low prio path: Received 216 pongs, i.e. for 73% of the pings.
+[INFO] [..] [ping_node]: Low prio path: Average RTT is 202.5ms.
+[INFO] [..] [ping_node]: Low prio path: Jitter of RTT is 36.301ms.
 ...
 ```
 

rclcpp/executors/cbg_executor/src/examples_rclcpp_cbg_executor/utilities.hpp

@@ -113,8 +113,12 @@ bool configure_native_thread(T native_handle, ThreadPriority priority, int cpu_i
   int policy;
   success &= (pthread_getschedparam(native_handle, &policy, &params) == 0);
   if (priority == ThreadPriority::HIGH) {
-    params.sched_priority = sched_get_priority_max(SCHED_FIFO);
+    // Choose a priority value slighly below the middle.
+    params.sched_priority =
+      (sched_get_priority_min(SCHED_FIFO) + sched_get_priority_max(SCHED_FIFO)) / 2 - 1;
   } else {
+    // Choose the lowest priority in SCHED_FIFO. This might still be higher than
+    // the priority of the DDS threads, which are not changed here.
     params.sched_priority = sched_get_priority_min(SCHED_FIFO);
   }
   success &= (pthread_setschedparam(native_handle, SCHED_FIFO, &params) == 0);
@@ -144,8 +148,21 @@ bool configure_native_thread(T native_handle, ThreadPriority priority, int cpu_i
   int policy;
   success &= (pthread_getschedparam(native_handle, &policy, &params) == 0);
   if (priority == ThreadPriority::HIGH) {
-    params.sched_priority = sched_get_priority_max(SCHED_FIFO);
+    // Should be a value of 49 on standard Linux platforms, which is just below
+    // the default priority of 50 for threaded interrupt handling.
+    params.sched_priority =
+      (sched_get_priority_min(SCHED_FIFO) + sched_get_priority_max(SCHED_FIFO)) / 2 - 1;
   } else {
+    // Choose the lowest priority under SCHED_FIFO. This will still be higher than
+    // the priority of the DDS threads, which are not changed here. Normally
+    // the DDS threads will be executed under SCHED_OTHER at nice value 0.
+    // Note that changing the priority below the default user-space priority requires
+    // increasing the nice level. This has not been implemented here for two reasons:
+    // First, the Linux API does not allow to get the Linux-specific thread ID (TID)
+    // for changing the nice value from an arbitrary pthread_t pointer, but only from the
+    // current thread by gettid(). Second, a low prio Executor thread under SCHED_OTHER
+    // would always get 50 ms per second due to RT throttling if not configured
+    // otherwise. This would be difficult to explain in a demo.
     params.sched_priority = sched_get_priority_min(SCHED_FIFO);
   }
   success &= (pthread_setschedparam(native_handle, SCHED_FIFO, &params) == 0);