Skip to content

Commit d818888

Browse files
committed
[ot] docs/opentitan: regression: Update regression documentation
When the script was extended to support specifying different execution environments and to collect all the Earlgrey test statuses together in one place, I forgot to update the accompanying documentation. Make sure that is up to date with this commit. Signed-off-by: Alex Jones <[email protected]>
1 parent d19d1c8 commit d818888

File tree

2 files changed

+28
-19
lines changed

2 files changed

+28
-19
lines changed

docs/opentitan/regressions.md

Lines changed: 19 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,20 +7,30 @@ are supported.
77
With a checkout of OpenTitan, the script can be run like this:
88

99
```sh
10-
./scripts/opentitan/run-bazel-tests.sh path/to/opentitan path/to/qemu
10+
./scripts/opentitan/run-bazel-tests.sh path/to/opentitan path/to/qemu \
11+
[execution_environment]
1112
```
1213

1314
The script will execute all QEMU-compatible tests using QEMU as it was built
14-
at the given path. The test results will be compared against two lists checked
15-
into this repository:
15+
at the given path. The test results will be compared against a list checked
16+
into this repository at `tests/opentitan/data/earlgrey-tests.txt`.
1617

17-
* `scripts/opentitan/tests-passing.txt`
18-
* `scripts/opentitan/tests-flaky.txt`
19-
20-
All tests in `tests-passing.txt` are expected to pass. Failures indicate a
18+
Each test occupies its own line, prefixed by the expected status.
19+
Tests with a `pass` status are expected to pass - failures indicate a
2120
regression in either QEMU or Earlgrey. The script will fail if there is a
2221
mismatch between the tests that we expect to pass and the actual results.
2322

2423
Some tests may be flaky and pass or fail on different runs of the same QEMU
25-
and OpenTitan checkouts. These tests can be added to the `tests-flaky.txt` list
26-
to cause the script to ignore them.
24+
and OpenTitan checkouts. These tests can be prefixed with the `flaky` status
25+
to cause the script to ignore them. This will suppress these tests from being
26+
printed as unexpected passes when they do succeed, but will also stop their
27+
failures from causing the script to fail.
28+
29+
After each test, a comment can be provided (starting with a `#`), which can
30+
help log the reasons that tests might have become flaky.
31+
32+
Specifying an execution environment when running the script will restrict to
33+
only running and comparing against tests for that execution environment.
34+
This can be used for more granular testing to break down large test workloads.
35+
If you do not specify an execution environment, _all_ available QEMU tests
36+
will be executed.

scripts/opentitan/run-bazel-tests.sh

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -6,22 +6,21 @@ set -e
66
# This script will run all QEMU tests in the provided OpenTitan repository,
77
# comparing the results with a list of expected passes.
88
#
9-
# USAGE: run-bazel-tests.sh path/to/opentitan/repo path/to/qemu/repo
9+
# USAGE: run-bazel-tests.sh path/to/opentitan/repo path/to/qemu/repo \
10+
# [execution environment]
1011
#
11-
# There are two companion files in this directory read by this script:
12-
#
13-
# * `tests-passing.txt`
14-
# * `tests-flaky.txt`
12+
# There is a companion file `tests/opentitan/data/earlgrey-tests.txt` that
13+
# is read by this script.
1514
#
1615
# The idea is to keep the list of passing tests up to date as QEMU changes.
1716
#
18-
# * If a test starts passing, add it to `tests-passing.txt` to catch
19-
# regressions.
17+
# * If a test starts passing, add it with a `pass` status to the test
18+
# list to catch regressions.
2019
# * When a test starts failing, investigate why and fix it, or remove it
21-
# from the `tests-passing.txt` list if necessary.
20+
# from the test list if necessary.
2221
#
23-
# Tests which are "flaky" and may pass/fail randomly will still be run,
24-
# but can be recorded below to prevent them from being checked.
22+
# Tests which are marked `flaky` and may pass/fail randomly will still be run,
23+
# but can be marked as such to prevent them from causing warnings or failures.
2524

2625
# CI-only job summary feature - write to `/dev/null` when run locally.
2726
GITHUB_STEP_SUMMARY="${GITHUB_STEP_SUMMARY:-/dev/null}"

0 commit comments

Comments
 (0)