You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add interleaved benchmark execution for test-suite (#107)
This adds support for separating build and test phases, and running
benchmarks from multiple builds in an interleaved fashion to control
for environmental factors (ambient temperature, general system load etc).
New options:
* --build: Build tests without running them
* --exec: Run tests from pre-built directory
* --build-dir: Specify build directory (used with --exec)
* --exec-interleaved-builds: Comma-separated list of builds to interleave
Usage:
1. Build two compiler versions:
lnt runtest test-suite --build \
--sandbox /tmp/sandbox-a \
--cc /path/to/clang-a \
--test-suite ~/llvm-test-suite \
...
lnt runtest test-suite --build \
--sandbox /tmp/sandbox-b \
--cc /path/to/clang-b \
--test-suite ~/llvm-test-suite \
...
2. Run with interleaved execution:
lnt runtest test-suite \
--sandbox /tmp/results \
--exec-interleaved-builds /tmp/sandbox-a/build,/tmp/sandbox-b/build \
--exec-multisample 3
This runs tests in the pattern:
- Sample 0: build-a -> build-b
- Sample 1: build-a -> build-b
- Sample 2: build-a -> build-b
Temporal interleaving controls for environmental changes that could
bias results toward one build.
Or, test single build:
lnt runtest test-suite --exec \
--build-dir /tmp/sandbox-a/build \
--exec-multisample 5
Reports are written to each build directory (report.json,
test-results.xunit.xml, test-results.csv).
0 commit comments