|
| 1 | +# A Guide to Benchmarking with PDL |
| 2 | + |
| 3 | +## Tasks |
| 4 | + |
| 5 | +This is a collection of benchmarking tasks collected from comparison sites, |
| 6 | +such as [plb2](https://github.com/attractivechaos/plb2), |
| 7 | +[FPBench](https://github.com/FPBench/FPBench) and |
| 8 | +the [Benchmarks Game](https://benchmarksgame-team.pages.debian.net/benchmarksgame/index.html). |
| 9 | +Yes, we want PDL to be the fastest dog in the race, but more importantly it lets |
| 10 | +our developers know when their latest commit has degraded performance. |
| 11 | + |
| 12 | +### Matrix multiplication |
| 13 | + |
| 14 | +From [plb2](https://github.com/attractivechaos/plb2/tree/master/src/perl), it creates |
| 15 | +two square matrices and multiplies them together (the inner product, `x`). |
| 16 | + |
| 17 | +This PDL script is more than 65 times faster than their Perl script. |
| 18 | +I've tried a few variations to find a faster version, but most gains are within |
| 19 | +the timing variation. |
| 20 | + |
| 21 | +Initial measurement for PDL 2.095 |
| 22 | +``` |
| 23 | +Benchmark 1: ./matrix_multiplication.pl |
| 24 | + Time (mean ± σ): 3.569 s ± 0.063 s [User: 3.656 s, System: 0.088 s] |
| 25 | + Range (min … max): 3.521 s … 3.710 s 10 runs |
| 26 | +``` |
| 27 | + |
| 28 | +## Benchmarking |
| 29 | + |
| 30 | +You can run a comparison with the Perl script using [hyperfine](https://github.com/sharkdp/hyperfine) |
| 31 | + |
| 32 | +``` |
| 33 | +hyperfine --warmup 1 'YOUR_ENV_VAR=1 path/to/matmul.pl 300' 'path/to/matrix_multiplication.pl 300' |
| 34 | +``` |
| 35 | + |
| 36 | +I recommend low values at first because, using the default (N=1500), |
| 37 | +hyperfine takes over 10 minutes to measure the Perl script because it's default |
| 38 | +is to run each program 10 times (changed with the **[mMr]** options). |
| 39 | +Ideally, you want to run this on a quiet system with few other processes running. |
| 40 | + |
| 41 | +Other benchmarks in pure Perl can be found at [plb2](https://github.com/attractivechaos/plb2/tree/master/src/perl), |
| 42 | + |
| 43 | +### Strategies |
| 44 | + |
| 45 | +If you want to compare two different branches against each other, |
| 46 | +consider using something like `--setup 'git checkout HEAD^'` or perhaps |
| 47 | +just running hyperfine with the single benchmark while searching for |
| 48 | +the offending commit with `git bisect`. Let us know how _you_ do benchmarking. |
| 49 | + |
| 50 | +## Profiling |
| 51 | + |
| 52 | +Use [Devel::NYTProf](https://metacpan.org/pod/Devel::NYTProf) to find out where |
| 53 | +your script is spending its time and how many times a line is run. Running the |
| 54 | +profiler really slows down your code, so I will run both commands on the same line |
| 55 | +and come back later. |
| 56 | +``` |
| 57 | +perl -d:NYTProf matrix_multiplication.pl 500; nytprofhtml --no-flame |
| 58 | +``` |
| 59 | + |
| 60 | +* [profiling](https://github.com/PDLPorters/pdl/issues/451) |
| 61 | +* [perl v java](https://charlesreid1.github.io/perl-vs-java-n-queens-problem.html) |
0 commit comments