MB-39815: Change durability_timeout_mode to event-driven

daverigby · jameseh96 · commit 8467cf26bac6 · 2021-10-14T12:56:40.000Z
Change the default durability_timeout_mode from polling to event-driven (see "MB-39815: Add event-driven SyncWrite timeout handling" for details). This has a significant impact on the idle memcached CPU utilisation. Measuring the memcached idle CPU for 30 (empty) buckets via top: Before: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 17121 daver 20 0 5353252 727656 19976 S 89.3 0.6 3:03.80 memcached After: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2531 daver 20 0 5336868 733672 20056 S 10.3 0.6 0:41.41 memcached per-thread details (>0% CPU) Before: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 17224 daver 20 0 5353252 752728 19976 S 11.3 0.6 1:14.40 NonIoPool0 17225 daver 20 0 5353252 752728 19976 S 11.3 0.6 1:14.29 NonIoPool1 17227 daver 20 0 5353252 752728 19976 S 11.3 0.6 1:14.22 NonIoPool3 17230 daver 20 0 5353252 752728 19976 S 11.3 0.6 1:14.41 NonIoPool6 17228 daver 20 0 5353252 752728 19976 S 11.0 0.6 1:14.36 NonIoPool4 17229 daver 20 0 5353252 752728 19976 S 11.0 0.6 1:14.41 NonIoPool5 17226 daver 20 0 5353252 752728 19976 S 10.6 0.6 1:14.22 NonIoPool2 17231 daver 20 0 5353252 752728 19976 S 4.0 0.6 0:26.31 SchedulerPool0 17121 daver 20 0 5353252 752728 19976 S 0.7 0.6 0:06.64 memcached 17187 daver 20 0 5353252 752728 19976 S 0.3 0.6 0:02.59 mc:worker_08 17195 daver 20 0 5353252 752728 19976 S 0.3 0.6 0:03.67 mc:worker_16 After: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2585 daver 20 0 5336868 737204 20056 S 3.7 0.6 0:01.99 mc:worker_08 2531 daver 20 0 5336868 737204 20056 S 1.0 0.6 0:02.31 memcached 2621 daver 20 0 5336868 737204 20056 S 0.7 0.6 0:02.05 NonIoPool0 2623 daver 20 0 5336868 737204 20056 S 0.7 0.6 0:01.97 NonIoPool2 2628 daver 20 0 5336868 737204 20056 S 0.7 0.6 0:02.58 SchedulerPool0 2582 daver 20 0 5336868 737204 20056 S 0.3 0.6 0:02.41 mc:worker_05 2596 daver 20 0 5336868 737204 20056 S 0.3 0.6 0:00.67 mc:worker_19 2622 daver 20 0 5336868 737204 20056 S 0.3 0.6 0:02.06 NonIoPool1 2624 daver 20 0 5336868 737204 20056 S 0.3 0.6 0:02.01 NonIoPool3 2625 daver 20 0 5336868 737204 20056 S 0.3 0.6 0:01.97 NonIoPool4 2626 daver 20 0 5336868 737204 20056 S 0.3 0.6 0:02.15 NonIoPool5 2627 daver 20 0 5336868 737204 20056 S 0.3 0.6 0:02.14 NonIoPool6 We can see that idle CPU drops from ~90% (of 1 CPU core) to 10%. Examiming the pre-thread details, with the original polling timeout mode the 7 NonIo threads were each running at ~11% of a core each; contributing 78% CPU total. Changing to event-driven reduces each of those threads to <0.7% each; contributing just 2.9% CPU total. Change-Id: If6f67b14a35f690f2bd87f7bf688cd5fb06f643b Reviewed-on: http://review.couchbase.org/c/kv_engine/+/162102 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com>
diff --git a/engines/ep/configuration.json b/engines/ep/configuration.json
@@ -468,7 +468,7 @@
             "type": "size_t"
         },
         "durability_timeout_mode": {
-            "default": "polling",
+            "default": "event-driven",
             "descr": "How should durability timeouts be scheduled? polling=periodic task running every 'durability_timeout_task_interval'; event-driven=per-VBucket tasks scheduled based on when next SyncWrite will time out.",
             "dynamic": false,
             "type": "std::string",
diff --git a/engines/ep/tests/ep_testsuite.cc b/engines/ep/tests/ep_testsuite.cc
@@ -6982,7 +6982,6 @@ static enum test_result test_mb19687_fixed(EngineIface* h) {
               "ep_defragmenter_mode",
               "ep_defragmenter_stored_value_age_threshold",
               "ep_durability_timeout_mode",
-              "ep_durability_timeout_task_interval",
               "ep_durability_min_level",
               "ep_executor_pool_backend",
               "ep_exp_pager_enabled",
@@ -7226,7 +7225,6 @@ static enum test_result test_mb19687_fixed(EngineIface* h) {
               "ep_diskqueue_memory",
               "ep_diskqueue_pending",
               "ep_durability_timeout_mode",
-              "ep_durability_timeout_task_interval",
               "ep_durability_min_level",
               "ep_executor_pool_backend",
               "ep_exp_pager_enabled",
