Removed dependency to benchmark_harness leading to a more modular class layout.

Author: simphotonics

CHANGELOG.md

@@ -1,3 +1,13 @@
+## 2.0.0
+Breaking changes:
+- The classes `Benchmark` and `AsyncBenchmark` are now solely responsible for
+generating benchmark scores. The constructor parameters `description` and
+`emitter` have been removed. Generating reports is delegated to `ScoreEmitter`.
+- The functions `benchmark` and `asyncBenchmark` are not longer generic and
+the only optional parameter is: `scoreEmitter`. A custom `ScoreEmitter` can be
+used to generate a custom benchmark report.
+
+
 ## 1.0.0
 Breaking changes:
 - The command `benchmark_runner` now has subcommands `report` and `export`.
@@ -29,8 +39,9 @@ inter-quartile-range `iqr` of the score sample is zero.
 
 ## 0.1.5
 - Made [BenchmarkHelper.sampleSize][sampleSize] a variable assignable with
-defined function. This allows changing the benchmark runtime by customizing
-the relation between score estimate and score sample size.
+a user defined function. This allows changing the score sample generation
+customizing the relation between the single score estimate and the
+score sample size.
 
 ## 0.1.4
 - Fixed bugs in runner (results were listed twice, exit code was always 0).

README.md

@@ -6,8 +6,7 @@
 
 Benchmarking is used to estimate and compare the execution speed of
 numerical algorithms and programs.
-The package [`benchmark_runner`][benchmark_runner] is based on
-[`benchmark_harness`][benchmark_harness] and includes helper
+The package [`benchmark_runner`][benchmark_runner] includes helper
 functions for writing *inline* micro-benchmarks with the option of
 printing a score **histogram** and reporting the score **mean** ±
 **standard deviation**, and score **median** ± **inter quartile range**.
@@ -36,8 +35,6 @@ Write inline benchmarks using the functions:
  asynchronous benchmarks.
 
   ```Dart
-  // ignore_for_file: unused_local_variable
-
   import 'package:benchmark_runner/benchmark_runner.dart';
 
   /// Returns the value [t] after waiting for [duration].
@@ -53,7 +50,7 @@ Write inline benchmarks using the functions:
 
       await asyncBenchmark('5ms', () async {
         await later<int>(27, Duration(milliseconds: 5));
-      }, emitStats: false);
+      }, scoreEmitter: MeanEmitter());
     });
 
     group('Set', () async {
@@ -115,25 +112,27 @@ To export benchmark scores use the sub-command `export`:
 $ dart run benchmark_runner export --outputDir=scores --extension=csv searchDirectory
 ```
 In the example above, `searchDirectory` is scanned for `*_benchmark.dart`
-files. For each benchmark file a corresponding file `*_benchmark.csv` is
-written to the directory `scores`. The directory must exist and the user
-must have write access.
+files. For each benchmark file, a corresponding file `*_benchmark.csv` is
+written to the directory `scores`.
 
-Note: When exporting benchmark scores to a file
+Note: The directory must exist and the user
+must have write access. When exporting benchmark scores to a file
 and the emitter output is colorized,
 it is recommended to use the option `--isMonochrome`, to
 avoid spurious characters due to the use of Ansi modifiers.
 
-Since version 1.0.0, the functions [`benchmark`][benchmark] and
-[`asyncBenchmark`][asyncBenchmark] accept the optional parameters `emitter` and
-`report`. These parameters can be used to customize the score reports e.g.
+The functions [`benchmark`][benchmark] and
+[`asyncBenchmark`][asyncBenchmark] accept the optional parameters `scoreEmitter`.
+The parameter expects an object of type `ScoreEmitter` and
+can be used to customize the score reports e.g.
 to make the score format more suitable for writing to a file:
 
 ```Dart
 import 'package:benchmark_runner/benchmark_runner.dart';
 
-class CustomEmitter extends ColorPrintEmitter {
-  void emitMean({required Score score}) {
+class CustomEmitter implements ScoreEmitter {
+  @override
+  void emit({required description, required Score score}) {
     print('# Mean               Standard Deviation');
     print('${score.stats.mean}  ${score.stats.stdDev}');
   }
@@ -145,11 +144,8 @@ void main(){
       () {
         var list = <int>[for (var i = 0; i < 1000; ++i) i];
       },
-      emitter: CustomEmitter(),
-      report: (instance, emitter) => emitter.emitMean(
-        score: instance.score(),
-      ),
-    );
+      scoreEmitter: CustomEmitter(),
+  );
 }
 ```
 
@@ -170,49 +166,54 @@ as reported by [`benchmark_harness`][benchmark_harness] and the
 score statistics.
 
 - By default, [`benchmark`][benchmark] and
-[`asyncBenchmark`][asyncBenchmark] report score statistics. In order to generate
-the report provided by [`benchmark_harness`][benchmark_harness] use the
-optional argument `report: reportMean`.
+[`asyncBenchmark`][asyncBenchmark] report score statistics. In order to print
+the report similar to that produced by
+[`benchmark_harness`][benchmark_harness], use the
+optional argument `emitter: MeanEmitter()`.
 
 - Color output can be switched off by using the option: `--isMonochrome` or `-m`
 when calling the benchmark runner. When executing a single benchmark file the
 corresponding option is `--define=isMonochrome=true`.
 
 - The default colors used to style benchmark reports are best suited
 for a dark terminal background.
-They can, however, be altered by setting the static variables defined by
+They can, however, be altered by setting the *static* variables defined by
 the class [`ColorProfile`][ColorProfile]. In the example below, the styling of
 error messages and the mean value is altered.
   ```Dart
   import 'package:ansi_modifier/ansi_modifier.dart';
   import 'package:benchmark_runner/benchmark_runner.dart';
 
-  void customColorProfile() {
+  void adjustColorProfile() {
     ColorProfile.error = Ansi.red + Ansi.bold;
     ColorProfile.mean = Ansi.green + Ansi.italic;
   }
 
   void main(List<String> args) {
     // Call function to apply the new custom color profile.
-    customColorProfile();
+    adjustColorProfile();
   }
   ```
 
 - When running **asynchronous** benchmarks, the scores are printed in order of
-completion. The print the scores in sequential order (as they are listed in the
+completion. To print the scores in sequential order (as they are listed in the
 benchmark executable) it is required to *await* the completion
 of the async benchmark functions and
 the enclosing group.
 
 ## Score Sampling
 
-In order to calculate benchmark score statistics a sample of scores is
+In order to calculate benchmark score *statistics* a sample of scores is
 required. The question is how to generate the score sample while minimizing
 systematic errors (like overheads) and keeping the
-benchmark run times within acceptable limits.
+total benchmark run times within acceptable limits.
+
+<details> <summary> Click to show details. </summary>
 
-To estimate the benchmark score the functions [`warmup`][warmup]
-or [`warmupAsync`][warmupAsync] are run for 200 milliseconds.
+In a first step, benchmark scores are estimated using the
+functions [`warmup`][warmup]
+or [`warmupAsync`][warmupAsync]. The function [`BenchmarkHelper.sampleSize`][sampleSize]
+uses the score estimate to determine the sampling procedure.
 
 ### 1. Default Sampling Method
 The graph below shows the sample size (orange curve) as calculated by the function
@@ -234,9 +235,10 @@ averaged over (see the cyan curve in the graph above):
 * ticks > 1e5 => No preliminary averaging of sample scores.
 
 ### 2. Custom Sampling Method
-To amend the score sampling process the static function
+To custominze the score sampling process, the static function
 [`BenchmarkHelper.sampleSize`][sampleSize] can be replaced with a custom function:
 ```Dart
+/// Generates a sample containing 100 benchmark scores. 
 BenchmarkHelper.sampleSize = (int clockTicks) {
   return (outer: 100, inner: 1)
 }
@@ -256,6 +258,7 @@ The command above lauches a process and runs a [`gnuplot`][gnuplot] script.
 For this reason, the program [`gnuplot`][gnuplot] must be installed (with
 the `qt` terminal enabled).
 
+</details>
 
 ## Contributions
 

benchmark/custom_emitter_benchmark.dart

@@ -1,8 +1,9 @@
 // ignore_for_file: unused_local_variable
 import 'package:benchmark_runner/benchmark_runner.dart';
 
-class CustomEmitter extends ColorPrintEmitter {
-  void emitMean({required Score score}) {
+class CustomEmitter implements ScoreEmitter {
+  @override
+  void emit({required description, required Score score}) {
     print('Mean               Standard Deviation');
     print('${score.stats.mean}  ${score.stats.stdDev}');
   }
@@ -14,9 +15,6 @@ void main(List<String> args) {
     () {
       var list = <int>[for (var i = 0; i < 1000; ++i) i];
     },
-    emitter: CustomEmitter(),
-    report: (instance, emitter) => emitter.emitMean(
-      score: instance.score(),
-    ),
+    scoreEmitter: CustomEmitter(),
   );
 }

benchmark/example_async_benchmark.dart

@@ -15,7 +15,7 @@ void main(List<String> args) async {
 
     await asyncBenchmark('5ms', () async {
       await later<int>(27, Duration(milliseconds: 5));
-    }, report: asyncReportMean);
+    }, scoreEmitter: MeanEmitter());
   });
 
   group('Set', () async {

benchmark/example_benchmark.dart

@@ -21,6 +21,6 @@ void main(List<String> args) {
 
     benchmark('construct', () {
       var list = <int>[for (var i = 0; i < 1000; ++i) i];
-    }, report: reportMean);
+    }, scoreEmitter: MeanEmitter());
   });
 }

gnuplot/sample_size.dart

@@ -6,6 +6,23 @@ import 'package:ansi_modifier/ansi_modifier.dart';
 import 'package:benchmark_runner/benchmark_runner.dart' show BenchmarkHelper;
 
 final ticksList = [
+  1,
+  2,
+  4,
+  6,
+  8,
+  10,
+  15,
+  25,
+  40,
+  70,
+  100,
+  180,
+  260,
+  380,
+  490,
+  600,
+  800,
   1001,
   1011,
   1070,
@@ -67,24 +84,28 @@ final ticksList = [
   194471590,
 ];
 
+final xAxisInMicroSeconds = ticksList.map(
+  (e) => (e * 1e6) / BenchmarkHelper.frequency,
+);
+
 final gnuplotScript = '''
 reset;
 set samples 1000;
 set term qt size 1000, 500 font "Sans, 14";
 set grid lw 2;
 set logscale x;
 unset label;
-set xlabel "Clock ticks";
-set xrange [ 1000 : 1.9e8 ] noreverse writeback;
+set xlabel "Single Run time [us]";
+set xrange [ 0 : 1e6 ] noreverse writeback;
 set x2range [ * : * ] noreverse writeback;
-set yrange [ 0 : 500 ] noreverse writeback;
+set yrange [ 0 : 600 ] noreverse writeback;
 set y2range [ * : * ] noreverse writeback;
-plot "sample_size.dat" using 1:2 with line lw 3 lt 2 lc "#0000FFFF" title "averaged over" at 0.3, 0.85, \
-     "sample_size.dat" using 1:2 lw 3 lt 6 lc "#0000BBBB" title " " at 0.3, 0.85, \
-     "sample_size.dat" using 1:3 with lines lw 3 lt 2 lc "#00FF8800" title "sample size" at 0.3, 0.77, \
-     "sample_size.dat" using 1:3 lw 3 lt 6 lc "#00991100" title " " at 0.3, 0.77, \
-     "sample_size.dat" using 1:4 with lines lw 3 lt 2 lc "#0000C77E" title "run time [ms]" at 0.3, 0.69, \
-     "sample_size.dat" using 1:4 lw 3 lt 6 lc "#0000974e" title " " at 0.3, 0.69;
+plot "sample_size.dat" using 1:2 with line lw 3 lt 2 lc "#0000FFFF" title "averaged over" at 0.6, 0.85, \
+     "sample_size.dat" using 1:2 lw 3 lt 6 lc "#0000BBBB" title " " at 0.6, 0.85, \
+     "sample_size.dat" using 1:3 with lines lw 3 lt 2 lc "#00FF8800" title "sample size" at 0.6, 0.77, \
+     "sample_size.dat" using 1:3 lw 3 lt 6 lc "#00991100" title " " at 0.6, 0.77, \
+     "sample_size.dat" using 1:4 with lines lw 3 lt 2 lc "#0000C77E" title "total sample generation time [ms]" at 0.6, 0.69, \
+     "sample_size.dat" using 1:4 lw 3 lt 6 lc "#0000974e" title " " at 0.6, 0.69;
 #
 ''';
 
@@ -101,17 +122,22 @@ void main(List<String> args) async {
     return (inner: 10, outer: 10); // This is a stub!
   }
 
+  print(xAxisInMicroSeconds);
+
   // Uncomment the line below to use your custom function:
   // BenchmarkHelper.sampleSize = customSampleSize;
 
   final b = StringBuffer();
   b.writeln(
-      '# Ticks    Inner-Iterations Outer-Iterations      Run-Time [1 ms]');
+    '# Microseconds  Inner-Iterations Outer-Iterations      Run-Time [1 ms]',
+  );
 
   for (final ticks in ticksList) {
     final (inner: inner, outer: outer) = BenchmarkHelper.sampleSize(ticks);
-    b.writeln('$ticks               $inner            '
-        '$outer              ${ticks * inner * outer / 1000000}');
+    b.writeln(
+      '${ticks * 1e6 / BenchmarkHelper.frequency}     $inner            '
+      '$outer              ${ticks * inner * outer * 1000 / BenchmarkHelper.frequency}',
+    );
   }
 
   final file = await File('sample_size.dat').writeAsString(b.toString());
@@ -122,6 +148,8 @@ void main(List<String> args) async {
 
   print(result.stdout);
   print(result.stderr);
-  print('Returning with gnuplot exit code:'
-      ' ${result.exitCode.toString().style(Ansi.green)}');
+  print(
+    'Returning with gnuplot exit code:'
+    ' ${result.exitCode.toString().style(Ansi.green)}',
+  );
 }

lib/benchmark_runner.dart

@@ -1,7 +1,7 @@
 export 'src/base/async_benchmark.dart';
 export 'src/base/benchmark.dart';
 export 'src/base/benchmark_process_result.dart';
-export 'src/emitter/color_print_emitter.dart';
+export 'src/emitter/score_emitter.dart';
 export 'src/base/group.dart';
 export 'src/base/score.dart';
 export 'src/command/benchmark_runner.dart';