[ot] docs/opentitan: regression: Update regression documentation

When the script was extended to support specifying different execution
environments and to collect all the Earlgrey test statuses together in
one place, I forgot to update the accompanying documentation.

Make sure that is up to date with this commit.

Signed-off-by: Alex Jones <[email protected]>

Author: AlexJones0

docs/opentitan/regressions.md

@@ -7,20 +7,30 @@ are supported.
 With a checkout of OpenTitan, the script can be run like this:
 
 ```sh
-./scripts/opentitan/run-bazel-tests.sh path/to/opentitan path/to/qemu
+./scripts/opentitan/run-bazel-tests.sh path/to/opentitan path/to/qemu \
+    [execution_environment]
 ```
 
 The script will execute all QEMU-compatible tests using QEMU as it was built
-at the given path. The test results will be compared against two lists checked
-into this repository:
+at the given path. The test results will be compared against a list checked
+into this repository at `tests/opentitan/data/earlgrey-tests.txt`.
 
-* `scripts/opentitan/tests-passing.txt`
-* `scripts/opentitan/tests-flaky.txt`
-
-All tests in `tests-passing.txt` are expected to pass. Failures indicate a
+Each test occupies its own line, prefixed by the expected status.
+Tests with a `pass` status are expected to pass - failures indicate a
 regression in either QEMU or Earlgrey. The script will fail if there is a
 mismatch between the tests that we expect to pass and the actual results.
 
 Some tests may be flaky and pass or fail on different runs of the same QEMU
-and OpenTitan checkouts. These tests can be added to the `tests-flaky.txt` list
-to cause the script to ignore them.
+and OpenTitan checkouts. These tests can be prefixed with the `flaky` status
+to cause the script to ignore them. This will suppress these tests from being
+printed as unexpected passes when they do succeed, but will also stop their
+failures from causing the script to fail.
+
+After each test, a comment can be provided (starting with a `#`), which can
+help log the reasons that tests might have become flaky.
+
+Specifying an execution environment when running the script will restrict to
+only running and comparing against tests for that execution environment.
+This can be used for more granular testing to break down large test workloads.
+If you do not specify an execution environment, _all_ available QEMU tests
+will be executed.

scripts/opentitan/run-bazel-tests.sh

@@ -6,22 +6,21 @@ set -e
 # This script will run all QEMU tests in the provided OpenTitan repository,
 # comparing the results with a list of expected passes.
 #
-# USAGE: run-bazel-tests.sh path/to/opentitan/repo path/to/qemu/repo
+# USAGE: run-bazel-tests.sh path/to/opentitan/repo path/to/qemu/repo \
+#            [execution environment]
 #
-# There are two companion files in this directory read by this script:
-#
-# * `tests-passing.txt`
-# * `tests-flaky.txt`
+# There is a companion file `tests/opentitan/data/earlgrey-tests.txt` that
+# is read by this script.
 # 
 # The idea is to keep the list of passing tests up to date as QEMU changes.
 #
-# * If a test starts passing, add it to `tests-passing.txt` to catch
-#   regressions.
+# * If a test starts passing, add it with a `pass` status to the test
+#   list to catch regressions.
 # * When a test starts failing, investigate why and fix it, or remove it
-#   from the `tests-passing.txt` list if necessary.
+#   from the test list if necessary.
 #
-# Tests which are "flaky" and may pass/fail randomly will still be run,
-# but can be recorded below to prevent them from being checked.
+# Tests which are marked `flaky` and may pass/fail randomly will still be run,
+# but can be marked as such to prevent them from causing warnings or failures.
 
 # CI-only job summary feature - write to `/dev/null` when run locally.
 GITHUB_STEP_SUMMARY="${GITHUB_STEP_SUMMARY:-/dev/null}"