Skip to content

Commit 01e86be

Browse files
committed
doc: Update A/B-testing documentation
Update the documentation to reflect that the script now takes directories with binaries. The old documentation was wrong anyway, since ab_test.py would fail if it couldn't find binaries in directories named matching the passed revisions after the very specific naming scheme that our buildkite infra uses. Signed-off-by: Patrick Roy <[email protected]>
1 parent 8de9b1b commit 01e86be

File tree

1 file changed

+11
-6
lines changed

1 file changed

+11
-6
lines changed

tests/README.md

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -169,16 +169,21 @@ function to [`.buildkite/pipeline_perf.py`](../.buildkite/pipeline_perf.py). To
169169
manually run an A/B-Test, use
170170

171171
```sh
172-
tools/devtool -y test --ab [optional arguments to ab_test.py] run <revision A> <revision B> --test <test specification>
172+
tools/devtool -y test --ab [optional arguments to ab_test.py] run <dir A> <dir B> --test <test specification>
173173
```
174174

175-
Here, _revision A_ and _revision B_ can be arbitrary git objects, such as commit
176-
SHAs, branches or tags. For example, to compare boottime of microVMs between
177-
Firecracker binaries compiled from the `main` branch and the `HEAD` of your
178-
current branch, run
175+
Here, _dir A_ and _dir B_ are directories containing firecracker and jailer
176+
binaries whose performance characteristics you wish to compare. You can use
177+
`./tools/devtool build --rev <revision> --release` to compile binaries from an
178+
arbitrary git object (commit SHAs, branches, tags etc.). This will create
179+
sub-directories in `build` containing the binaries. For example, to compare
180+
boottime of microVMs between Firecracker binaries compiled from the `main`
181+
branch and the `HEAD` of your current branch, run
179182

180183
```sh
181-
tools/devtool -y test --ab run main HEAD --test integration_tests/performance/test_boottime.py::test_boottime
184+
tools/devtool -y build --rev main --release
185+
tools/devtool -y build --rev HEAD --release
186+
tools/devtool -y test --ab -- run build/main build/HEAD --test integration_tests/performance/test_boottime.py::test_boottime
182187
```
183188

184189
#### How to Write an A/B-Compatible Test and Common Pitfalls

0 commit comments

Comments
 (0)