Skip to content

Commit 330aa32

Browse files
authored
Add interleaved benchmark execution for test-suite (#107)
This adds support for separating build and test phases, and running benchmarks from multiple builds in an interleaved fashion to control for environmental factors (ambient temperature, general system load etc). New options: * --build: Build tests without running them * --exec: Run tests from pre-built directory * --build-dir: Specify build directory (used with --exec) * --exec-interleaved-builds: Comma-separated list of builds to interleave Usage: 1. Build two compiler versions: lnt runtest test-suite --build \ --sandbox /tmp/sandbox-a \ --cc /path/to/clang-a \ --test-suite ~/llvm-test-suite \ ... lnt runtest test-suite --build \ --sandbox /tmp/sandbox-b \ --cc /path/to/clang-b \ --test-suite ~/llvm-test-suite \ ... 2. Run with interleaved execution: lnt runtest test-suite \ --sandbox /tmp/results \ --exec-interleaved-builds /tmp/sandbox-a/build,/tmp/sandbox-b/build \ --exec-multisample 3 This runs tests in the pattern: - Sample 0: build-a -> build-b - Sample 1: build-a -> build-b - Sample 2: build-a -> build-b Temporal interleaving controls for environmental changes that could bias results toward one build. Or, test single build: lnt runtest test-suite --exec \ --build-dir /tmp/sandbox-a/build \ --exec-multisample 5 Reports are written to each build directory (report.json, test-results.xunit.xml, test-results.csv).
1 parent 4819b7b commit 330aa32

File tree

4 files changed

+1577
-45
lines changed

4 files changed

+1577
-45
lines changed
107 KB
Binary file not shown.

0 commit comments

Comments
 (0)