Skip to content

Commit 9ac6f3a

Browse files
roypatbchalios
authored andcommitted
doc: Add instruction about running functional A/B-tests locally
A quick blurb about which environment variables need to be set to run things like test_vulnerabilities.py locally. Signed-off-by: Patrick Roy <[email protected]>
1 parent f7bceb6 commit 9ac6f3a

File tree

1 file changed

+15
-0
lines changed

1 file changed

+15
-0
lines changed

tests/README.md

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,21 @@ above when run on a PR will fail iff a newly added dependency has a known open
142142
RustSec advisory. If run outside a PR, it will fail if any existing dependency
143143
has an open RustSec advisory).
144144

145+
### Functional A/B-Tests
146+
147+
Firecracker has some functional A/B-tests (for example, in
148+
`test_vulnerabilities.py`), which generally compare the state of the pull
149+
request target branch (e.g. `main`), with the PR head. However, when running
150+
these locally, pytest does not know anything about potential PRs that the commit
151+
the tests are being run on are contained in, and as such cannot do this
152+
A/B-Test. To run functional A/B-Tests locally, you need to create a "fake" PR
153+
environment by setting the `BUILDKITE_PULL_REQUEST` and
154+
`BUILDKITE_PULL_REQUEST_BASE_BRANCH` environment variables:
155+
156+
```
157+
BUILDKITE_PULL_REQUEST=true BUILDKITE_PULL_REQUEST_BASE_BRANCH=main ./tools/devtool test -- integration_tests/security/test_vulnerabilities.py
158+
```
159+
145160
### Performance A/B-Tests
146161

147162
Firecracker has a special framework for orchestrating long-running A/B-tests

0 commit comments

Comments
 (0)