Add interleaved benchmark execution for test-suite #107
Conversation
This stack of pull requests is managed by Graphite. Learn more about stacking. |
There was a problem hiding this comment.
I'm not super familiar with the "run the test suite" part of LNT, so this might not be the most sensible question. But have you considered making this into a new subcommand of lnt runtest instead of adding "a new mode" for lnt runtest test-suite? This seems to be doing things quite a bit differently from the regular lnt runtest test-suite, and I (perhaps naively) wonder if it wouldn't be better to acknowledge that and not try to make it fit into lnt runtest test-suite.
Like I said, I don't have a strong opinion, but I wanted to at least ask the question.
Good question. I'm no expert on this project, but from what I can tell the test runners seem to handle most of the actual mechanics of building and testing (and for example, implementing PGO flows). So moving this mechanism into the I don't know of anyone actively using the other test runners, but if there is then maybe we can think about re-working the whole thing to allow the I've addressed your other comments now. |
There was a problem hiding this comment.
I do find it annoying that we're making lnt runtest test-suite do four different things based on command-line flags (current operation, build only, test only, or test interleaved). But I don't know how to make that not be the case without a potentially large amount of work.
The code seems to achieve the desired design, and the tests seem sufficient to me -- thanks for adding them.
You might want to wait for feedback from other people who actually use lnt runtest or have experience with it, since I don't. But from my side, LGTM.
|
I was digging into this a bit more after seeing #108, and I see there's already a We might want to pick names that are a bit less ambiguous than
There is the following equivalence to your current approach:
The benefit I can see of the above is that it's more consistent with I'm not very attached to the above, but I do think it might help reduce the complexity of the command-line interface if we introduce some of the flags as a simple |
|
In principle I agree. Only thing is we have Since "tests" in the LLVM test suite can refer to testing building, not just execution, how about we go with: |
|
Yeah, I'd be fine with that. I do agree the naming is a bit tricky to get right. I'd also be OK with something like Just food for thought. I think I like your initial suggestion best. |
This adds support for separating build and test phases, and running
benchmarks from multiple builds in an interleaved fashion to control
for environmental factors (ambient temperature, general system load etc).
New options:
* --build-only: Build tests without running them
* --test-prebuilt: Run tests from pre-built directory
* --build-dir: Specify build directory (used with --test-prebuilt)
* --exec-interleaved-builds: Comma-separated list of builds to interleave
Usage:
1. Build two compiler versions:
lnt runtest test-suite --build-only \
--sandbox /tmp/sandbox-a \
--cc /path/to/clang-a \
--test-suite ~/llvm-test-suite \
...
lnt runtest test-suite --build-only \
--sandbox /tmp/sandbox-b \
--cc /path/to/clang-b \
--test-suite ~/llvm-test-suite \
...
2. Run with interleaved execution:
lnt runtest test-suite \
--sandbox /tmp/results \
--exec-interleaved-builds /tmp/sandbox-a/build,/tmp/sandbox-b/build \
--exec-multisample 3
This runs tests in the pattern:
- Sample 0: build-a -> build-b
- Sample 1: build-a -> build-b
- Sample 2: build-a -> build-b
Temporal interleaving controls for environmental changes that could
bias results toward one build.
Or, test single build:
lnt runtest test-suite --test-prebuilt \
--build-dir /tmp/sandbox-a/build \
--exec-multisample 5
Reports are written to each build directory (report.json,
test-results.xunit.xml, test-results.csv).
The --build-dir option now works with all build modes, not just --test-prebuilt. It can be used to specify a custom build directory location, overriding the default sandbox/build or timestamped directory.
aemerson
force-pushed
the
users/amara/interleaved-builds
branch
from
064aefe to
17ed608
Compare
This adds support for separating build and test phases, and running
benchmarks from multiple builds in an interleaved fashion to control
for environmental factors (ambient temperature, general system load etc).
New options:
Usage:
Build two compiler versions:
lnt runtest test-suite --build
--sandbox /tmp/sandbox-a
--cc /path/to/clang-a
--test-suite ~/llvm-test-suite
...
lnt runtest test-suite --build
--sandbox /tmp/sandbox-b
--cc /path/to/clang-b
--test-suite ~/llvm-test-suite
...
Run with interleaved execution:
lnt runtest test-suite
--sandbox /tmp/results
--exec-interleaved-builds /tmp/sandbox-a/build,/tmp/sandbox-b/build
--exec-multisample 3
This runs tests in the pattern:
Or, test single build:
lnt runtest test-suite --exec
--build-dir /tmp/sandbox-a/build
--exec-multisample 5
Reports are written to each build directory (report.json, test-results.xunit.xml, test-results.csv).