Skip to content

Commit 5d575bd

Browse files
committed
Added README.md to folder gnuplot.
1 parent 597d49f commit 5d575bd

File tree

1 file changed

+116
-0
lines changed

1 file changed

+116
-0
lines changed

gnuplot/README.md

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
# Benchmark Runner - Score Sampling
2+
[![Dart](https://github.com/simphotonics/benchmark_runner/actions/workflows/dart.yml/badge.svg)](https://github.com/simphotonics/benchmark_runner/actions/workflows/dart.yml)
3+
4+
5+
## Introduction
6+
7+
When benchmarking a function, one has to take into consideration that
8+
the test is not performed on an isolated system that is only handling the
9+
instructions provided by our piece of code. Instead, the CPUs is very likely
10+
busy performing other tasks before eventually executing the benchmarked code.
11+
This introduces latency that is typically larger for the first few runs
12+
and then has a certain degree of randomness. The initial latency
13+
can be reduced to some extend by introducing a warm-up phase,
14+
where the first few runs are discarded.
15+
16+
A second factor to consider are systematic measuring errors due to the fact
17+
that it takes (a small amount of) time to increment loop counters, perform
18+
loop checks, or access the elapsed time.
19+
20+
The overhead introduced by repeatedly accessing the
21+
elapsed time can be reduced by averaging the
22+
benchmark score over many runs.
23+
24+
The question is how to generate a score sample while minimizing
25+
systematic errors and keeping the
26+
total benchmark run times within acceptable limits.
27+
28+
29+
## Score Estimate
30+
31+
In a first step, benchmark scores are *estimated* using the
32+
functions [`estimate`][estimate]
33+
or [`estimateAsync`][estimateAsync].
34+
The function [`BenchmarkHelper.sampleSize`][sampleSize]
35+
uses the score estimate to determine a suitable sample size and number of inner
36+
iterations (for short run times each sample entry is averaged).
37+
38+
### Default Sampling Method
39+
The graph below shows the sample size (orange curve) as calculated by the function
40+
[`BenchmarkHelper.sampleSize`][sampleSize].
41+
42+
The green curve shows the lower limit of the total microbenchmark duration
43+
in microseconds.
44+
45+
![Sample Size](https://raw.githubusercontent.com/simphotonics/benchmark_runner/main/images/sample_size.png)
46+
47+
For short run times below 1 microsecond each score sample is generated
48+
using the functions [`measure`][measure] or the equivalent
49+
asynchronous method [`measureAsync`][measureAsync]. The cyan curve shows
50+
approx. over how many runs a score entry is averaged.
51+
52+
### Custom Sampling Method
53+
54+
The parameter `sampleSize` of the functions [`benchmark`][benchmark] and
55+
[`asyncBenchmark`][asyncBenchmark] can be used to specify the lenght of the score
56+
sample list and the number of inner iterations used to generate each entry.
57+
58+
To customize the score sampling process, *without* having to specify the parameter
59+
`sampleSize` for each call of [`benchmark`][benchmark] and
60+
[`asyncBenchmark`][asyncBenchmark], the static function
61+
[`BenchmarkHelper.sampleSize`][sampleSize] can be replaced with a custom function:
62+
```Dart
63+
/// Generates a sample containing 100 benchmark scores.
64+
BenchmarkHelper.sampleSize = (int clockTicks) {
65+
return (outer: 100, inner: 1)
66+
}
67+
```
68+
To restore the default score sampling settings use:
69+
```Dart
70+
BenchmarkHelper.sampleSize = BenchmarkHelper.sampleSizeDefault;
71+
```
72+
----
73+
The graph shown above may be re-generated using the custom `sampleSize`
74+
function by copying and amending the file `gnuplot/sample_size.dart`
75+
and using the command:
76+
```Console
77+
dart sample_size.dart
78+
```
79+
The command above lauches a process and runs a [`gnuplot`][gnuplot] script.
80+
For this reason, the program [`gnuplot`][gnuplot] must be installed (with
81+
the `qt` terminal enabled).
82+
83+
</details>
84+
85+
86+
## Features and bugs
87+
88+
Please file feature requests and bugs at the [issue tracker][tracker].
89+
90+
[tracker]: https://github.com/simphotonics/benchmark_runner/issues
91+
92+
[asyncBenchmark]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/asyncBenchmark.html
93+
94+
[asyncGroup]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/asyncGroup.html
95+
96+
[benchmark_harness]: https://pub.dev/packages/benchmark_harness
97+
98+
[benchmark_runner]: https://pub.dev/packages/benchmark_runner
99+
100+
[benchmark]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/benchmark.html
101+
102+
[ColorProfile]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/ColorProfile.html
103+
104+
[gnuplot]: https://sourceforge.net/projects/gnuplot/
105+
106+
[group]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/group.html
107+
108+
[measure]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/BenchmarkHelper/measure.html
109+
110+
[measureAsync]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/BenchmarkHelper/measureAsync.html
111+
112+
[sampleSize]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/BenchmarkHelper/sampleSize.html
113+
114+
[estimate]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/BenchmarkHelper/estimate.html
115+
116+
[estimateAsync]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/BenchmarkHelper/estimateUpAsync.html

0 commit comments

Comments
 (0)