1- # Iris Performance Benchmarking
1+ # SciTools Performance Benchmarking
22
3- Iris uses an [ Airspeed Velocity] ( https://github.com/airspeed-velocity/asv )
3+ SciTools uses an [ Airspeed Velocity] ( https://github.com/airspeed-velocity/asv )
44(ASV) setup to benchmark performance. This is primarily designed to check for
55performance shifts between commits using statistical analysis, but can also
66be easily repurposed for manual comparative and scalability analyses.
@@ -21,25 +21,30 @@ by the PR. (This run is managed by
2121[ the aforementioned GitHub Action] ( ../.github/workflows/benchmark.yml ) ).
2222
2323To run locally: the ** benchmark runner** provides conveniences for
24- common benchmark setup and run tasks, including replicating the automated
25- overnight run locally. This is accessed via the Nox ` benchmarks ` session - see
26- ` nox -s benchmarks -- --help ` for detail (_ see also:
27- [ bm_runner.py] ( ./bm_runner.py ) _ ). Alternatively you can directly run ` asv ... `
28- commands from this directory (you will still need Nox installed - see
29- [ Benchmark environments] ( #benchmark-environments ) ).
24+ common benchmark setup and run tasks, including replicating the benchmarking
25+ performed by GitHub Actions workflows. This can be accessed by:
26+
27+ - The Nox ` benchmarks ` session - (use
28+ ` nox -s benchmarks -- --help ` for details).
29+ - ` benchmarks/bm_runner.py ` (use the ` --help ` argument for details).
30+ - Directly running ` asv ` commands from the ` benchmarks/ ` directory (check
31+ whether environment setup has any extra dependencies - see
32+ [ Benchmark environments] ( #benchmark-environments ) ).
33+
34+ ### Reducing run time
3035
3136A significant portion of benchmark run time is environment management. Run-time
32- can be reduced by placing the benchmark environment on the same file system as
33- your
34- [ Conda package cache ] ( https://conda.io/projects/conda/en/latest/user-guide/configuration/use-condarc.html#specify-pkg-directories ) ,
35- if it is not already. You can achieve this by either :
36-
37- - Temporarily reconfiguring ` ENV_PARENT ` in ` delegated_env_commands `
38- in [ asv.conf.json ] ( asv.conf.json ) to reference a location on the same file
39- system as the Conda package cache.
37+ can be reduced by co-locating the benchmark environment and your
38+ [ Conda package cache ] ( https://docs.conda.io/projects/conda/en/latest/user-guide/configuration/custom-env-and-pkg-locations.html )
39+ on the same [ file system ] ( https://en.wikipedia.org/wiki/File_system ) , if they
40+ are not already. This can be done in several ways :
41+
42+ - Temporarily reconfiguring ` env_parent ` in
43+ [ ` _asv_delegated_abc ` ] ( _asv_delegated_abc.py ) to reference a location on the same
44+ file system as the Conda package cache.
4045- Using an alternative Conda package cache location during the benchmark run,
4146 e.g. via the ` $CONDA_PKGS_DIRS ` environment variable.
42- - Moving your Iris repo to the same file system as the Conda package cache.
47+ - Moving your repo checkout to the same file system as the Conda package cache.
4348
4449### Environment variables
4550
@@ -73,7 +78,8 @@ requirements will not be delayed by repeated environment setup - especially
7378relevant given the [ benchmark runner] ( bm_runner.py ) 's use of
7479[ --interleave-rounds] ( https://asv.readthedocs.io/en/stable/commands.html?highlight=interleave-rounds#asv-run ) ,
7580or any time you know you will repeatedly benchmark the same commit. ** NOTE:**
76- Iris environments are large so this option can consume a lot of disk space.
81+ SciTools environments tend to large so this option can consume a lot of disk
82+ space.
7783
7884## Writing benchmarks
7985
@@ -97,6 +103,7 @@ for manual investigations; and consider committing any useful benchmarks as
97103[ on-demand benchmarks] ( #on-demand-benchmarks ) for future developers to use.
98104
99105### Data generation
106+
100107** Important:** be sure not to use the benchmarking environment to generate any
101108test objects/files, as this environment changes with each commit being
102109benchmarked, creating inconsistent benchmark 'conditions'. The
@@ -106,7 +113,7 @@ solution; read more detail there.
106113### ASV re-run behaviour
107114
108115Note that ASV re-runs a benchmark multiple times between its ` setup() ` routine.
109- This is a problem for benchmarking certain Iris operations such as data
116+ This is a problem for benchmarking certain SciTools operations such as data
110117realisation, since the data will no longer be lazy after the first run.
111118Consider writing extra steps to restore objects' original state _ within_ the
112119benchmark itself.
@@ -117,10 +124,13 @@ maintain result accuracy this should be accompanied by increasing the number of
117124repeats _ between_ ` setup() ` calls using the ` repeat ` attribute.
118125` warmup_time = 0 ` is also advisable since ASV performs independent re-runs to
119126estimate run-time, and these will still be subject to the original problem.
127+ The ` @disable_repeat_between_setup ` decorator in
128+ [ ` benchmarks/__init__.py ` ] ( benchmarks/__init__.py ) offers a convenience for
129+ all this.
120130
121131### Custom benchmarks
122132
123- Iris benchmarking implements custom benchmark types, such as a ` tracemalloc `
133+ SciTools benchmarking implements custom benchmark types, such as a ` tracemalloc `
124134benchmark to measure memory growth. See [ custom_bms/] ( ./custom_bms ) for more
125135detail.
126136
@@ -131,10 +141,10 @@ limited available runtime and risk of false-positives. It remains useful for
131141manual investigations).**
132142
133143When comparing performance between commits/file-type/whatever it can be helpful
134- to know if the differences exist in scaling or non-scaling parts of the Iris
135- functionality in question . This can be done using a size parameter, setting
136- one value to be as small as possible (e.g. a scalar ` Cube ` ), and the other to
137- be significantly larger (e.g. a 1000x1000 ` Cube ` ). Performance differences
144+ to know if the differences exist in scaling or non-scaling parts of the
145+ operation under test . This can be done using a size parameter, setting
146+ one value to be as small as possible (e.g. a scalar value ), and the other to
147+ be significantly larger (e.g. a 1000x1000 array ). Performance differences
138148might only be seen for the larger value, or the smaller, or both, getting you
139149closer to the root cause.
140150
@@ -151,13 +161,15 @@ suites for the UK Met Office NG-VAT project.
151161## Benchmark environments
152162
153163We have disabled ASV's standard environment management, instead using an
154- environment built using the same Nox scripts as Iris' test environments. This
155- is done using ASV's plugin architecture - see
156- [ asv_delegated_conda.py] ( asv_delegated_conda.py ) and the extra config items in
157- [ asv.conf.json] ( asv.conf.json ) .
164+ environment built using the same scripts that set up the package test
165+ environments.
166+ This is done using ASV's plugin architecture - see
167+ [ ` asv_delegated.py ` ] ( asv_delegated.py ) and associated
168+ references in [ ` asv.conf.json ` ] ( asv.conf.json ) (` environment_type ` and
169+ ` plugins ` ).
158170
159171(ASV is written to control the environment(s) that benchmarks are run in -
160172minimising external factors and also allowing it to compare between a matrix
161173of dependencies (each in a separate environment). We have chosen to sacrifice
162174these features in favour of testing each commit with its intended dependencies,
163- controlled by Nox + lock-files ).
175+ controlled by the test environment setup script(s) ).
0 commit comments