Skip to content

Commit 4a4bacc

Browse files
committed
doc: Add instruction about running functional A/B-tests locally
A quick blurb about which environment variables need to be set to run things like test_vulnerabilities.py locally. Signed-off-by: Patrick Roy <[email protected]>
1 parent 0447a16 commit 4a4bacc

File tree

1 file changed

+16
-0
lines changed

1 file changed

+16
-0
lines changed

tests/README.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,22 @@ above when run on a PR will fail iff a newly added dependency has a known open
142142
RustSec advisory. If run outside a PR, it will fail if any existing dependency
143143
has an open RustSec advisory).
144144

145+
### Functional A/B-Tests
146+
147+
Firecracker has some functional A/B-tests (for example, in
148+
`test_vulnerabilities.py`), which generally compare the state of the pull
149+
request target branch (e.g. `main`), with the PR head. However, when running
150+
these locally, pytest does not know anything about potential PRs that the commit
151+
the tests are being ran on are contained in, and as such cannot do this
152+
A/B-Test. To run functional A/B-Tests locally, you need to create a "fake" PR
153+
environment by setting the `BUILDKITE_PULL_REQUEST` and
154+
`BUILDKITE_PULL_REQUEST_BASE_BRANCH` environment variables:
155+
156+
```
157+
158+
BUILDKITE_PULL_REQUEST=true BUILDKITE_PULL_REQUEST_BASE_BRANCH=main ./tools/devtool test -- integration_tests/security/test_vulnerabilities.py
159+
```
160+
145161
### Performance A/B-Tests
146162

147163
Firecracker has a special framework for orchestrating long-running A/B-tests

0 commit comments

Comments
 (0)