Skip to content

sembr tests/running.md and improve it a bit #2637

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Nov 10, 2025
Merged

sembr tests/running.md and improve it a bit #2637

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9eaf7f0
sembr tests/running.md
tshepang Nov 10, 2025
5a9e666
sembr: handle text inside div blocks
tshepang Nov 10, 2025
1fe988c
tests/running.md: minor improvements
tshepang Nov 10, 2025
03bcbfa
fix code block language marker
tshepang Nov 10, 2025
021e32e
a more suitable marker
tshepang Nov 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions ci/sembr/src/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,7 @@ fn lengthen_lines(content: &str, limit: usize) -> String {
let mut new_content = content.clone();
let mut new_n = 0;
let mut in_code_block = false;
let mut in_html_div = false;
let mut skip_next = false;
for (n, line) in content.iter().enumerate() {
if skip_next {
Expand All @@ -150,6 +151,17 @@ fn lengthen_lines(content: &str, limit: usize) -> String {
in_code_block = !in_code_block;
continue;
}
if line.trim_start().starts_with("<div") {
in_html_div = true;
continue;
}
if line.trim_start().starts_with("</div") {
in_html_div = false;
continue;
}
if in_html_div {
continue;
}
if ignore(line, in_code_block) || REGEX_SPLIT.is_match(line) {
continue;
}
Expand Down Expand Up @@ -215,9 +227,15 @@ fn test_prettify() {
let original = "\
do not split
short sentences
<div class='warning'>
a bit of text inside
</div>
";
let expected = "\
do not split short sentences
<div class='warning'>
a bit of text inside
</div>
";
assert_eq!(expected, lengthen_lines(original, 50));
}
Expand Down
162 changes: 82 additions & 80 deletions src/tests/running.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
# Running tests

You can run the entire test collection using `x`. But note that running the
*entire* test collection is almost never what you want to do during local
development because it takes a really long time. For local development, see the
subsection after on how to run a subset of tests.
You can run the entire test collection using `x`.
But note that running the *entire* test collection is almost never what you want to do during local
development because it takes a really long time.
For local development, see the subsection after on how to run a subset of tests.

<div class="warning">

Running plain `./x test` will build the stage 1 compiler and then run the whole
test suite. This not only includes `tests/`, but also `library/`, `compiler/`,
Running plain `./x test` will build the stage 1 compiler and then run the whole test suite.
This not only includes `tests/`, but also `library/`, `compiler/`,
`src/tools/` package tests and more.

You usually only want to run a subset of the test suites (or even a smaller set
of tests than that) which you expect will exercise your changes. PR CI exercises
a subset of test collections, and merge queue CI will exercise all of the test
of tests than that) which you expect will exercise your changes.
PR CI exercises a subset of test collections, and merge queue CI will exercise all of the test
collection.

</div>
Expand All @@ -22,8 +22,8 @@ collection.
./x test
```

The test results are cached and previously successful tests are `ignored` during
testing. The stdout/stderr contents as well as a timestamp file for every test
The test results are cached and previously successful tests are `ignored` during testing.
The stdout/stderr contents as well as a timestamp file for every test
can be found under `build/<target-tuple>/test/` for the given
`<target-tuple>`. To force-rerun a test (e.g. in case the test runner fails to
notice a change) you can use the `--force-rerun` CLI option.
Expand All @@ -40,18 +40,17 @@ notice a change) you can use the `--force-rerun` CLI option.

## Running a subset of the test suites

When working on a specific PR, you will usually want to run a smaller set of
tests. For example, a good "smoke test" that can be used after modifying rustc
When working on a specific PR, you will usually want to run a smaller set of tests.
For example, a good "smoke test" that can be used after modifying rustc
to see if things are generally working correctly would be to exercise the `ui`
test suite ([`tests/ui`]):

```text
./x test tests/ui
```

Of course, the choice of test suites is
somewhat arbitrary, and may not suit the task you are doing. For example, if you
are hacking on debuginfo, you may be better off with the debuginfo test suite:
Of course, the choice of test suites is somewhat arbitrary, and may not suit the task you are doing.
For example, if you are hacking on debuginfo, you may be better off with the debuginfo test suite:

```text
./x test tests/debuginfo
Expand All @@ -77,8 +76,8 @@ Likewise, you can test a single file by passing its path:
./x test tests/ui/const-generics/const-test.rs
```

`x` doesn't support running a single tool test by passing its path yet. You'll
have to use the `--test-args` argument as described
`x` doesn't support running a single tool test by passing its path yet.
You'll have to use the `--test-args` argument as described
[below](#running-an-individual-test).

```text
Expand All @@ -97,8 +96,8 @@ have to use the `--test-args` argument as described
./x test --stage 0 library/std
```

Note that this only runs tests on `std`; if you want to test `core` or other
crates, you have to specify those explicitly.
Note that this only runs tests on `std`;
if you want to test `core` or other crates, you have to specify those explicitly.

### Run the tidy script and tests on the standard library

Expand All @@ -117,8 +116,8 @@ you avoid having to run tests for components you did not change at all.

<div class="warning">

Note that bors only runs the tests with the full stage 2 build; therefore, while
the tests **usually** work fine with stage 1, there are some limitations.
Note that bors only runs the tests with the full stage 2 build;
therefore, while the tests **usually** work fine with stage 1, there are some limitations.

</div>

Expand All @@ -129,7 +128,8 @@ the tests **usually** work fine with stage 1, there are some limitations.
```

<div class="warning">
You almost never need to do this; CI will run these tests for you.
You almost never need to do this;
CI will run these tests for you.
</div>

## Run unit tests on the compiler/library
Expand All @@ -140,7 +140,8 @@ You may want to run unit tests on a specific file with following:
./x test compiler/rustc_data_structures/src/thin_vec/tests.rs
```

But unfortunately, it's impossible. You should invoke the following instead:
But unfortunately, it's impossible.
You should invoke the following instead:

```text
./x test compiler/rustc_data_structures/ --test-args thin_vec
Expand All @@ -149,24 +150,23 @@ But unfortunately, it's impossible. You should invoke the following instead:
## Running an individual test

Another common thing that people want to do is to run an **individual test**,
often the test they are trying to fix. As mentioned earlier, you may pass the
full file path to achieve this, or alternatively one may invoke `x` with the
`--test-args` option:
often the test they are trying to fix.
As mentioned earlier, you may pass the
full file path to achieve this, or alternatively one may invoke `x` with the `--test-args` option:

```text
./x test tests/ui --test-args issue-1234
```

Under the hood, the test runner invokes the standard Rust test runner (the same
one you get with `#[test]`), so this command would wind up filtering for tests
that include "issue-1234" in the name. Thus, `--test-args` is a good way to run
a collection of related tests.
that include "issue-1234" in the name.
Thus, `--test-args` is a good way to run a collection of related tests.

## Passing arguments to `rustc` when running tests

It can sometimes be useful to run some tests with specific compiler arguments,
without using `RUSTFLAGS` (during development of unstable features, with `-Z`
flags, for example).
without using `RUSTFLAGS` (during development of unstable features, with `-Z` flags, for example).

This can be done with `./x test`'s `--compiletest-rustc-args` option, to pass
additional arguments to the compiler when building the tests.
Expand All @@ -176,8 +176,7 @@ additional arguments to the compiler when building the tests.
If you have changed the compiler's output intentionally, or you are making a new
test, you can pass `--bless` to the test subcommand.

As an example,
if some tests in `tests/ui` are failing, you can run this command:
As an example, if some tests in `tests/ui` are failing, you can run this command:

```text
./x test tests/ui --bless
Expand All @@ -192,8 +191,9 @@ just like when running the tests without the `--bless` flag.
There are a few options for running tests:

* `bootstrap.toml` has the `rust.verbose-tests` option. If `false`, each test will
print a single dot (the default). If `true`, the name of every test will be
printed. This is equivalent to the `--quiet` option in the [Rust test
print a single dot (the default).
If `true`, the name of every test will be printed.
This is equivalent to the `--quiet` option in the [Rust test
harness](https://doc.rust-lang.org/rustc/tests/).
* The environment variable `RUST_TEST_THREADS` can be set to the number of
concurrent threads to use for testing.
Expand All @@ -202,24 +202,24 @@ There are a few options for running tests:

Pass UI tests now have three modes, `check-pass`, `build-pass` and `run-pass`.
When `--pass $mode` is passed, these tests will be forced to run under the given
`$mode` unless the directive `//@ ignore-pass` exists in the test file. For
example, you can run all the tests in `tests/ui` as `check-pass`:
`$mode` unless the directive `//@ ignore-pass` exists in the test file.
For example, you can run all the tests in `tests/ui` as `check-pass`:

```text
./x test tests/ui --pass check
```

By passing `--pass $mode`, you can reduce the testing time. For each mode,
please see [Controlling pass/fail
By passing `--pass $mode`, you can reduce the testing time.
For each mode, please see [Controlling pass/fail
expectations](ui.md#controlling-passfail-expectations).

## Running tests with different "compare modes"

UI tests may have different output depending on certain "modes" that the
compiler is in. For example, when using the Polonius mode, a test `foo.rs` will
UI tests may have different output depending on certain "modes" that the compiler is in.
For example, when using the Polonius mode, a test `foo.rs` will
first look for expected output in `foo.polonius.stderr`, falling back to the
usual `foo.stderr` if not found. The following will run the UI test suite in
Polonius mode:
usual `foo.stderr` if not found.
The following will run the UI test suite in Polonius mode:

```text
./x test tests/ui --compare-mode=polonius
Expand All @@ -229,66 +229,68 @@ See [Compare modes](compiletest.md#compare-modes) for more details.

## Running tests manually

Sometimes it's easier and faster to just run the test by hand. Most tests are
just `.rs` files, so after [creating a rustup
Sometimes it's easier and faster to just run the test by hand.
Most tests are just `.rs` files, so after [creating a rustup
toolchain](../building/how-to-build-and-run.md#creating-a-rustup-toolchain), you
can do something like:

```text
rustc +stage1 tests/ui/issue-1234.rs
```

This is much faster, but doesn't always work. For example, some tests include
This is much faster, but doesn't always work.
For example, some tests include
directives that specify specific compiler flags, or which rely on other crates,
and they may not run the same without those options.

## Running tests on a remote machine

Tests may be run on a remote machine (e.g. to test builds for a different
architecture). This is done using `remote-test-client` on the build machine to
architecture).
This is done using `remote-test-client` on the build machine to
send test programs to `remote-test-server` running on the remote machine.
`remote-test-server` executes the test programs and sends the results back to
the build machine. `remote-test-server` provides *unauthenticated remote code
`remote-test-server` executes the test programs and sends the results back to the build machine.
`remote-test-server` provides *unauthenticated remote code
execution* so be careful where it is used.

To do this, first build `remote-test-server` for the remote machine, e.g. for
RISC-V
To do this, first build `remote-test-server` for the remote machine
(using RISC-V as an example):

```text
./x build src/tools/remote-test-server --target riscv64gc-unknown-linux-gnu
```

The binary will be created at
`./build/host/stage2-tools/$TARGET_ARCH/release/remote-test-server`. Copy this
over to the remote machine.
The binary will be created at `./build/host/stage2-tools/$TARGET_ARCH/release/remote-test-server`.
Copy this over to the remote machine.

On the remote machine, run the `remote-test-server` with the `--bind
0.0.0.0:12345` flag (and optionally `-v` for verbose output). Output should look
like this:
0.0.0.0:12345` flag (and optionally `--verbose` flag).
Output should look like this:

```text
$ ./remote-test-server -v --bind 0.0.0.0:12345
```console
$ ./remote-test-server --verbose --bind 0.0.0.0:12345
starting test server
listening on 0.0.0.0:12345!
```

Note that binding the server to 0.0.0.0 will allow all hosts able to reach your
machine to execute arbitrary code on your machine. We strongly recommend either
machine to execute arbitrary code on your machine.
We strongly recommend either
setting up a firewall to block external access to port 12345, or to use a more
restrictive IP address when binding.

You can test if the `remote-test-server` is working by connecting to it and
sending `ping\n`. It should reply `pong`:
You can test if the `remote-test-server` is working by connecting to it and sending `ping\n`.
It should reply `pong`:

```text
```console
$ nc $REMOTE_IP 12345
ping
pong
```

To run tests using the remote runner, set the `TEST_DEVICE_ADDR` environment
variable then use `x` as usual. For example, to run `ui` tests for a RISC-V
machine with the IP address `1.2.3.4` use
variable then use `x` as usual.
For example, to run `ui` tests for a RISC-V machine with the IP address `1.2.3.4` use

```text
export TEST_DEVICE_ADDR="1.2.3.4:12345"
Expand All @@ -315,34 +317,34 @@ run "/tmp/work/test1018/a"
[...]
```

Tests are built on the machine running `x` not on the remote machine. Tests
which fail to build unexpectedly (or `ui` tests producing incorrect build
Tests are built on the machine running `x` not on the remote machine.
Tests which fail to build unexpectedly (or `ui` tests producing incorrect build
output) may fail without ever running on the remote machine.

## Testing on emulators

Some platforms are tested via an emulator for architectures that aren't readily
available. For architectures where the standard library is well supported and
Some platforms are tested via an emulator for architectures that aren't readily available.
For architectures where the standard library is well supported and
the host operating system supports TCP/IP networking, see the above instructions
for testing on a remote machine (in this case the remote machine is emulated).

There is also a set of tools for orchestrating running the tests within the
emulator. Platforms such as `arm-android` and `arm-unknown-linux-gnueabihf` are
set up to automatically run the tests under emulation on GitHub Actions. The
following will take a look at how a target's tests are run under emulation.
There is also a set of tools for orchestrating running the tests within the emulator.
Platforms such as `arm-android` and `arm-unknown-linux-gnueabihf` are
set up to automatically run the tests under emulation on GitHub Actions.
The following will take a look at how a target's tests are run under emulation.

The Docker image for [armhf-gnu] includes [QEMU] to emulate the ARM CPU
architecture. Included in the Rust tree are the tools [remote-test-client] and
The Docker image for [armhf-gnu] includes [QEMU] to emulate the ARM CPU architecture.
Included in the Rust tree are the tools [remote-test-client] and
[remote-test-server] which are programs for sending test programs and libraries
to the emulator, and running the tests within the emulator, and reading the
results. The Docker image is set up to launch `remote-test-server` and the
to the emulator, and running the tests within the emulator, and reading the results.
The Docker image is set up to launch `remote-test-server` and the
build tools use `remote-test-client` to communicate with the server to
coordinate running tests (see [src/bootstrap/src/core/build_steps/test.rs]).

To run on the iOS/tvOS/watchOS/visionOS simulator, we can similarly treat it as
a "remote" machine. A curious detail here is that the network is shared between
the simulator instance and the host macOS, so we can use the local loopback
address `127.0.0.1`. Something like the following should work:
To run on the iOS/tvOS/watchOS/visionOS simulator, we can similarly treat it as a "remote" machine.
A curious detail here is that the network is shared between
the simulator instance and the host macOS, so we can use the local loopback address `127.0.0.1`.
Something like the following should work:

```sh
# Build the test server for the iOS simulator:
Expand Down Expand Up @@ -384,7 +386,7 @@ On the [wasm32-wasip1 target support page] a minimum version is specified that y
Some cmake commands that take a while and give a lot of very concerning c++ warnings...
Then, in `bootstrap.toml`, point to the sysroot like so:

```
```toml
[target.wasm32-wasip1]
wasi-root = "<wasi-sdk location>/build/sysroot/install/share/wasi-sysroot"
```
Expand Down
Loading