Prepare release of ReBench 1.3.0 (#284)

Author: smarr

CHANGELOG.md

@@ -2,6 +2,105 @@
 
 ## [Unreleased]
 
+## [1.3.0] Env Vars part of RunId, and Support for Machine-Specific Settings - 2025-02-03
+
+This release has three major changes that will affect how ReBench interprets configurations,
+and a change for `denoise` that requires an update of the `sudoers` file.
+
+### 1. "Run" identity is now based on all details, e.g. also env vars (#271)
+
+This change can affect the number of different runs ReBench identifies.
+Until now, the identity of a "run" was based on the command line generated from the
+configuration. However, this meant, we could not distinguish between runs
+with for instance different environment variables.
+From now on, the identity of runs is based on all details of the configuration,
+which may mean that configuration files may result in more distinct runs
+than they did previously.
+
+### 2. Build commands are combined and run as a single shell script (#269)
+
+The `build:` key takes a list of commands, which are now combined into a single
+shell script. Previously, each command was run separately. 
+Since we avoid executing the same command multiple times,
+treating commands separately led to confusing results and was not very useful.
+By combining the commands into a single script, identifying redundant script
+execution leads to more predictable results.
+
+### 3. Support for machine-specific settings (#272)
+
+A new lowest-priority configuration level is introduced, which is meant to define
+machine-specific settings. This can be useful, for instance for settings like this:
+
+```yaml
+machines:
+  smaller-machine:
+    cores: [1, 2]
+  larger-machine:
+    cores: [1, 2, 4, 8]
+```
+
+These settings are combined into a configuration based on the usual rules.
+So, if a benchmark suite or a benchmark give specific values for `cores:`, these
+have higher priority. However, one would likely use it, avoiding
+benchmark-specific values.
+
+When executing ReBench on the corresponding machine, these settings can be
+selected with the `-m` option. Thus, the larger machine configuration is used
+when running:
+
+```bash
+rebench -m larger-machine rebench.conf
+```
+
+### 4. `rebench` uses `denoise.py` directly as script (#281)
+
+Previously, we used the `rebench-denoise` script that is automatically created
+during the setup process. However, this required a setup where ReBench was
+installed so that the root user had access to it with a proper `PYTHONPATH`.
+To avoid issues with setup, avoid having to pass `PYTHONPATH` through `sudo`,
+and to make the use of denoise more robust, we changed `denoise.py` so that it
+can be used directly as a script by ReBench.
+
+In practice, the means, it's likely that the `sudoers` file will need to be
+updated to point to the `denoise.py` script directly. ReBench will output the
+path to the script when it is not able to use it directly.
+
+
+### Other Minor Changes
+
+#### Features
+ - add support for ReBenchDB API Version 2 (#236)
+ - expand `~` in paths, right before invocation (#240, #283)
+ - add summary for columns where all values are the same (#256)
+ - show `env` in debug output (#280)
+
+#### Changes
+ - make output for currently running benchmark more compact (#237)
+ - invoke denoise with absolute path to avoid need for having it on a secure_path (#238, #273)
+ - rename the machines variable/filters to a tags variable/filter (#264)
+ - support Python 3.13 as latest version, drop support for Python 3.8 (#268)
+ - reduce number of git invocations to get source information (#275)
+ - distinguish `rebench` return codes for different errors (#282)
+
+#### Bug Fixes
+ - added missing machine column heading for summary table (#246)
+ - behave more gracefully on bare-bone setup, e.g. without git (#245)
+ - raise error in `TimeAdapter` to match other adapters (#254)
+ - make profiling with `perf` more robust (#255)
+ - make denoise more robust to absent tools and running as root with a user-level installation (#260)
+ - fix handling of ctrl-c interrupts (#262)
+ - avoid unnecessary warning about data reporting, when no data is to be reported (#277)
+
+#### Development
+ - use black formatting (#267)
+ - use mypy type checking and add some annotations (#270)
+ - use `pip install —editable` in CI to avoid incorrect coverage reporting (#279)
+ - add testing on Rocky Linux with integration test (#282)
+
+Thanks to @antonzhukovin, @vext01, and @martinmcclure for their contributions!
+
+**Full Changelog**: https://github.com/smarr/ReBench/compare/v1.2.0...v1.3.0
+
 ## [1.2.0] Custom Gauge Adapters - 2023-08-06
 
 The main feature of this release is the new support for custom
@@ -20,7 +119,7 @@ Other new features:
  - make the current invocation accessible in the command as `%(invocation)s` (#230)
 
 Other changes:
- - fix bug where `'None'` instead of `null` was reported to ReBenchDB (#232)
+ - make sure `null` is not reported as `'None'` to ReBenchDB (#232)
  - fix handling of environment variables when sudo is used (#210)
  - try `gtime` from MacPorts as alternative `time` command on macOS (#212)
  - update py-cpuinfo to work on macOS with ARM-base CPUs (#212)
@@ -29,6 +128,8 @@ Other changes:
 
 Thanks to @naomiGrew for the contributions!
 
+**Full Changelog**: https://github.com/smarr/ReBench/compare/v1.1.0...v1.2.0
+
 ## [1.1.0] Denoise - 2023-02-21
 
 This release focuses on reducing the noise from the system  (#143, #144).
@@ -68,9 +169,9 @@ Other notable improvements:
  - make gauge adapter names in configurations case-insensitive (#202)
  - improve documentation (#197, #198)
  - use PyTest for unit tests (#192)
- 
+
 **Full Changelog**: https://github.com/smarr/ReBench/compare/v1.0.1...v1.1.0
- 
+
 
 ## [1.0.1] - 2020-06-23
 
@@ -83,7 +184,7 @@ This is a bug fix release.
 
 This is the first official release of ReBench as a "feature-complete" product.
 Feature-complete here means, it is a tried and tested tool for benchmark
-execution. It is highly 
+execution. It is highly
 [configurable](https://rebench.readthedocs.io/en/latest/config/),
 [documented](https://rebench.readthedocs.io/en/latest/),
 and [successfully used](https://github.com/smarr/ReBench#use-in-academia).
@@ -136,7 +237,7 @@ Thank you!
  - added testing of Python 3.7 (#121) and ruamel.yaml (#123)
  - ensure config is YAML 1.2 compliant (#123)
  - added support for ReBenchDB (#129, #130)
- 
+
  - fixed issues with error reporting (#128)
  - fixed handling of input size configuration (#117)
 
@@ -145,7 +246,7 @@ Thank you!
  - added `--setup-only` option, to run one benchmark for each setup (#110, #115)
  - added `ignore_timeout` setting to accept known timeouts without error (#118)
  - added `retries_after_failure` setting (#107, #108)
- 
+
  - fixed data loading, which ignored warmup setting (#111, #116)
  - fixed how settings are inherited for follow documentation (#112, #113)
  - fixed message for consecutive failures (#109)
@@ -219,7 +320,8 @@ Thank you!
  - [0.6.0] - 2014-05-19
  - [0.5.0] - 2014-03-25
 
-[Unreleased]: https://github.com/smarr/ReBench/compare/v1.2.0...HEAD
+[Unreleased]: https://github.com/smarr/ReBench/compare/v1.3.0...HEAD
+[1.3.0]:  https://github.com/smarr/ReBench/compare/v1.2.0...v1.3.0
 [1.2.0]:  https://github.com/smarr/ReBench/compare/v1.1.0...v1.2.0
 [1.1.0]:  https://github.com/smarr/ReBench/compare/v1.0.1...v1.1.0
 [1.0.1]:  https://github.com/smarr/ReBench/compare/v1.0.0...v1.0.1

docs/config.md

@@ -580,14 +580,14 @@ A list of commands/strings to be executed by the system's shell.
 They are intended to set up the system for benchmarking,
 typically to build binaries, compiled archives, etc.
 
-Each command is executed once before any benchmark that depends on it
-is executed. If the `location` of the suite is set, it is used as
+The list of command is combined into a single shell script, which
+is executed once before any benchmark that depends on it
+runs. If the `location` of the suite is set, it is used as
 working directory. Otherwise, it is the current working directory of ReBench.
 
-`build:` is a list of commands to allow multiple suites and executors to depend on the
-same build command without executing it multiple times.
-For this purpose, build commands are considered the same when they have the
-same command and location (based on simple string comparisons).
+When multiple suites and executors depend on lists of commands,
+where the combined script is identical, and to be executed in the same location,
+the script is executed only once.
 
 Commands are executed with an environment managed by ReBench.
 The environment starts empty, but can be added to using the [`env`](#env) directive.

docs/release.md

@@ -3,7 +3,7 @@
 This list is supposed to make sure we do not forget any important steps during
 release.
 
-1. Check and update issue tracker for issues associated with current milestone
+1. Check and update issue tracker for issues that should be included in the release.
 
 2. Make sure all desired changes are merged with `master`.
 

rebench/__init__.py

@@ -1 +1 @@
-__version__ = "1.3.0.dev2"
+__version__ = "1.3.0"