-
Notifications
You must be signed in to change notification settings - Fork 22
Replace BenchmarkTools.jl with Chairmarks.jl in Optimizing #138
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 2 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -36,7 +36,7 @@ With this in mind, after you're done with the current page, you should read the | |
|
||
## Measurements | ||
|
||
\tldr{Use BenchmarkTools.jl's `@benchmark` with a setup phase to get the most accurate idea of your code's performance. Use Chairmarks.jl as a faster alternative.} | ||
\tldr{Use Chairmarks.jl's `@be` with a setup phase to get the most accurate idea of your code's performance.} | ||
|
||
The simplest way to measure how fast a piece of code runs is to use the `@time` macro, which returns the result of the code and prints the measured runtime and allocations. | ||
Because code needs to be compiled before it can be run, you should first run a function without timing it so it can be compiled, and then time it: | ||
|
@@ -53,39 +53,42 @@ using BenchmarkTools | |
Using `@time` is quick but it has flaws, because your function is only measured once. | ||
That measurement might have been influenced by other things going on in your computer at the same time. | ||
In general, running the same block of code multiple times is a safer measurement method, because it diminishes the probability of only observing an outlier. | ||
The Chairmarks.jl package provides convenient syntax to do just that. | ||
|
||
### BenchmarkTools | ||
### Chairmarks | ||
|
||
[BenchmarkTools.jl](https://github.com/JuliaCI/BenchmarkTools.jl) is the most popular package for repeated measurements on function executions. | ||
Similarly to `@time`, BenchmarkTools offers `@btime` which can be used in exactly the same way but will run the code multiple times and provide an average. | ||
Additionally, by using `$` to [interpolate external values](https://juliaci.github.io/BenchmarkTools.jl/stable/manual/#Interpolating-values-into-benchmark-expressions), you remove the overhead caused by global variables. | ||
[Chairmarks.jl](https://github.com/LilithHafner/Chairmarks.jl) is the latest and greatest benchmarking suite used to make fast and accurate timing measurements. | ||
Chairmarks offers `@b` (for "benchmark") which can be used in exactly the same way as `@time` but will run the code multiple times and provide a minimum execution time. | ||
nathanrboyer marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Alternatively, Chairmarks also provides `@be` to run the benchmark and output all of its statistics. | ||
nathanrboyer marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
```>$-example | ||
using BenchmarkTools | ||
@btime sum_abs(v); | ||
@btime sum_abs($v); | ||
```>chairmarks-example | ||
using Chairmarks | ||
@b sum_abs(v) | ||
@be sum_abs(v) | ||
``` | ||
|
||
Chairmarks supports a pipeline syntax with optional `init`, `setup`, `teardown`, and `keywords` arguments for more extensive control over the benchmarking process. | ||
The `sum_abs` function could also be benchmarked using pipeline syntax as below. | ||
|
||
```>pipeline-example-simple | ||
@be v sum_abs | ||
``` | ||
|
||
In more complex settings, you might need to construct variables in a [setup phase](https://juliaci.github.io/BenchmarkTools.jl/stable/manual/#Setup-and-teardown-phases) that is run before each sample. | ||
This can be useful to generate a new random input every time, instead of always using the same input. | ||
For a more complicated example, you could write the following to benchmark a matrix multiplication function for one second, excluding the time spent to *setup* the arrays. | ||
|
||
```>setup-example | ||
```>pipeline-example-complex | ||
my_matmul(A, b) = A * b; | ||
@btime my_matmul(A, b) setup=( | ||
A = rand(1000, 1000); # use semi-colons between setup lines | ||
b = rand(1000) | ||
); | ||
@be (A=rand(1000,1000), b=rand(1000)) my_matmul(_.A, _.b) seconds=1 | ||
``` | ||
|
||
For better visualization, the `@benchmark` macro shows performance histograms: | ||
See the [Chairmarks documentation](https://chairmarks.lilithhafner.com/) for more details on benchmarking options. | ||
For better visualization, [PrettyChairmarks.jl](https://github.com/astrozot/PrettyChairmarks.jl) shows performance histograms alongside the numerical results. | ||
|
||
\advanced{ | ||
Certain computations may be [optimized away by the compiler]((https://juliaci.github.io/BenchmarkTools.jl/stable/manual/#Understanding-compiler-optimizations)) before the benchmark takes place. | ||
If you observe suspiciously fast performance, especially below the nanosecond scale, this is very likely to have happened. | ||
} | ||
Comment on lines
87
to
90
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This can now be removed (or updated for Chairmarks @LilithHafner ?). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's always possible to optimize away computation unintentionally julia> @btime 10+20+30+40+50
1.167 ns (0 allocations: 0 bytes)
150
julia> @b 10+20+30+40+50
1.241 ns Using Chairmarks does not mean we no longer need to worry about this. Also, BenchmarkTools.jl's docs are great for this. Chairmarks does not have a better alternative. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think readers might be confused by a link to BenchmarkTools in what is now a section about Chairmarks. If this is crucial information and Chairmarks replaces BenchmarkTools, maybe the Chairmarks docs should mirror the contents? If this is not crucial, maybe we should remove it from MoJuWo entirely. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe just clarify that the explanation from BenchmarkTools applies here too |
||
|
||
[Chairmarks.jl](https://github.com/LilithHafner/Chairmarks.jl) offers an alternative to BenchmarkTools.jl, promising faster benchmarking while attempting to maintain high accuracy and using an alternative syntax based on pipelines. | ||
|
||
### Benchmark suites | ||
|
||
While we previously discussed the importance of documenting breaking changes in packages using [semantic versioning](/sharing/index.md#versions-and-registration), regressions in performance can also be vital to track. | ||
|
@@ -97,10 +100,13 @@ Several packages exist for this purpose: | |
|
||
### Other tools | ||
|
||
BenchmarkTools.jl works fine for relatively short and simple blocks of code (microbenchmarking). | ||
Chairmarks.jl works fine for relatively short and simple blocks of code (microbenchmarking). | ||
To find bottlenecks in a larger program, you should rather use a [profiler](#profiling) or the package [TimerOutputs.jl](https://github.com/KristofferC/TimerOutputs.jl). | ||
It allows you to label different sections of your code, then time them and display a table of grouped by label. | ||
|
||
[BenchmarkTools.jl](https://github.com/JuliaCI/BenchmarkTools.jl) was the previous standard for benchmarking in Julia. It is still widely used today. | ||
nathanrboyer marked this conversation as resolved.
Show resolved
Hide resolved
|
||
However, it is slower than Chairmarks and requires interpolating variables into the benchmarked expressions with `$`. | ||
nathanrboyer marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Finally, if you know a loop is slow and you'll need to wait for it to be done, you can use [ProgressMeter.jl](https://github.com/timholy/ProgressMeter.jl) or [ProgressLogging.jl](https://github.com/JuliaLogging/ProgressLogging.jl) to track its progress. | ||
|
||
## Profiling | ||
|
@@ -141,7 +147,7 @@ To integrate profile visualisations into environments like Jupyter and Pluto, us | |
No matter which tool you use, if your code is too fast to collect samples, you may need to run it multiple times in a loop. | ||
|
||
\advanced{ | ||
To visualize memory allocation profiles, use PProf.jl or VSCode's `@profview_allocs`. | ||
To visualize memory allocation profiles, use PProf.jl or VSCode's `@profview_allocs`. | ||
A known issue with the allocation profiler is that it is not able to determine the type of every object allocated, instead `Profile.Allocs.UnknownType` is shown instead. | ||
Inspecting the call graph can help identify which types are responsible for the allocations. | ||
} | ||
|
@@ -386,7 +392,7 @@ However, in order for all workers to know about a function or module, we have to | |
using Distributed | ||
|
||
# Add additional workers then load code on the workers | ||
addprocs(3) | ||
addprocs(3) | ||
@everywhere using SharedArrays | ||
@everywhere f(x) = 3x^2 | ||
|
||
|
Uh oh!
There was an error while loading. Please reload this page.