diff --git a/ci/sembr/src/main.rs b/ci/sembr/src/main.rs index 99e90e908..b50190c78 100644 --- a/ci/sembr/src/main.rs +++ b/ci/sembr/src/main.rs @@ -137,6 +137,7 @@ fn lengthen_lines(content: &str, limit: usize) -> String { let mut new_content = content.clone(); let mut new_n = 0; let mut in_code_block = false; + let mut in_html_div = false; let mut skip_next = false; for (n, line) in content.iter().enumerate() { if skip_next { @@ -150,6 +151,17 @@ fn lengthen_lines(content: &str, limit: usize) -> String { in_code_block = !in_code_block; continue; } + if line.trim_start().starts_with(" -Running plain `./x test` will build the stage 1 compiler and then run the whole -test suite. This not only includes `tests/`, but also `library/`, `compiler/`, +Running plain `./x test` will build the stage 1 compiler and then run the whole test suite. +This not only includes `tests/`, but also `library/`, `compiler/`, `src/tools/` package tests and more. You usually only want to run a subset of the test suites (or even a smaller set -of tests than that) which you expect will exercise your changes. PR CI exercises -a subset of test collections, and merge queue CI will exercise all of the test +of tests than that) which you expect will exercise your changes. +PR CI exercises a subset of test collections, and merge queue CI will exercise all of the test collection. @@ -22,8 +22,8 @@ collection. ./x test ``` -The test results are cached and previously successful tests are `ignored` during -testing. The stdout/stderr contents as well as a timestamp file for every test +The test results are cached and previously successful tests are `ignored` during testing. +The stdout/stderr contents as well as a timestamp file for every test can be found under `build//test/` for the given ``. To force-rerun a test (e.g. in case the test runner fails to notice a change) you can use the `--force-rerun` CLI option. @@ -40,8 +40,8 @@ notice a change) you can use the `--force-rerun` CLI option. ## Running a subset of the test suites -When working on a specific PR, you will usually want to run a smaller set of -tests. For example, a good "smoke test" that can be used after modifying rustc +When working on a specific PR, you will usually want to run a smaller set of tests. +For example, a good "smoke test" that can be used after modifying rustc to see if things are generally working correctly would be to exercise the `ui` test suite ([`tests/ui`]): @@ -49,9 +49,8 @@ test suite ([`tests/ui`]): ./x test tests/ui ``` -Of course, the choice of test suites is -somewhat arbitrary, and may not suit the task you are doing. For example, if you -are hacking on debuginfo, you may be better off with the debuginfo test suite: +Of course, the choice of test suites is somewhat arbitrary, and may not suit the task you are doing. +For example, if you are hacking on debuginfo, you may be better off with the debuginfo test suite: ```text ./x test tests/debuginfo @@ -77,8 +76,8 @@ Likewise, you can test a single file by passing its path: ./x test tests/ui/const-generics/const-test.rs ``` -`x` doesn't support running a single tool test by passing its path yet. You'll -have to use the `--test-args` argument as described +`x` doesn't support running a single tool test by passing its path yet. +You'll have to use the `--test-args` argument as described [below](#running-an-individual-test). ```text @@ -97,8 +96,8 @@ have to use the `--test-args` argument as described ./x test --stage 0 library/std ``` -Note that this only runs tests on `std`; if you want to test `core` or other -crates, you have to specify those explicitly. +Note that this only runs tests on `std`; +if you want to test `core` or other crates, you have to specify those explicitly. ### Run the tidy script and tests on the standard library @@ -117,8 +116,8 @@ you avoid having to run tests for components you did not change at all.
-Note that bors only runs the tests with the full stage 2 build; therefore, while -the tests **usually** work fine with stage 1, there are some limitations. +Note that bors only runs the tests with the full stage 2 build; +therefore, while the tests **usually** work fine with stage 1, there are some limitations.
@@ -129,7 +128,8 @@ the tests **usually** work fine with stage 1, there are some limitations. ```
-You almost never need to do this; CI will run these tests for you. +You almost never need to do this; +CI will run these tests for you.
## Run unit tests on the compiler/library @@ -140,7 +140,8 @@ You may want to run unit tests on a specific file with following: ./x test compiler/rustc_data_structures/src/thin_vec/tests.rs ``` -But unfortunately, it's impossible. You should invoke the following instead: +But unfortunately, it's impossible. +You should invoke the following instead: ```text ./x test compiler/rustc_data_structures/ --test-args thin_vec @@ -149,9 +150,9 @@ But unfortunately, it's impossible. You should invoke the following instead: ## Running an individual test Another common thing that people want to do is to run an **individual test**, -often the test they are trying to fix. As mentioned earlier, you may pass the -full file path to achieve this, or alternatively one may invoke `x` with the -`--test-args` option: +often the test they are trying to fix. +As mentioned earlier, you may pass the +full file path to achieve this, or alternatively one may invoke `x` with the `--test-args` option: ```text ./x test tests/ui --test-args issue-1234 @@ -159,14 +160,13 @@ full file path to achieve this, or alternatively one may invoke `x` with the Under the hood, the test runner invokes the standard Rust test runner (the same one you get with `#[test]`), so this command would wind up filtering for tests -that include "issue-1234" in the name. Thus, `--test-args` is a good way to run -a collection of related tests. +that include "issue-1234" in the name. +Thus, `--test-args` is a good way to run a collection of related tests. ## Passing arguments to `rustc` when running tests It can sometimes be useful to run some tests with specific compiler arguments, -without using `RUSTFLAGS` (during development of unstable features, with `-Z` -flags, for example). +without using `RUSTFLAGS` (during development of unstable features, with `-Z` flags, for example). This can be done with `./x test`'s `--compiletest-rustc-args` option, to pass additional arguments to the compiler when building the tests. @@ -176,8 +176,7 @@ additional arguments to the compiler when building the tests. If you have changed the compiler's output intentionally, or you are making a new test, you can pass `--bless` to the test subcommand. -As an example, -if some tests in `tests/ui` are failing, you can run this command: +As an example, if some tests in `tests/ui` are failing, you can run this command: ```text ./x test tests/ui --bless @@ -192,8 +191,9 @@ just like when running the tests without the `--bless` flag. There are a few options for running tests: * `bootstrap.toml` has the `rust.verbose-tests` option. If `false`, each test will - print a single dot (the default). If `true`, the name of every test will be - printed. This is equivalent to the `--quiet` option in the [Rust test + print a single dot (the default). + If `true`, the name of every test will be printed. + This is equivalent to the `--quiet` option in the [Rust test harness](https://doc.rust-lang.org/rustc/tests/). * The environment variable `RUST_TEST_THREADS` can be set to the number of concurrent threads to use for testing. @@ -202,24 +202,24 @@ There are a few options for running tests: Pass UI tests now have three modes, `check-pass`, `build-pass` and `run-pass`. When `--pass $mode` is passed, these tests will be forced to run under the given -`$mode` unless the directive `//@ ignore-pass` exists in the test file. For -example, you can run all the tests in `tests/ui` as `check-pass`: +`$mode` unless the directive `//@ ignore-pass` exists in the test file. +For example, you can run all the tests in `tests/ui` as `check-pass`: ```text ./x test tests/ui --pass check ``` -By passing `--pass $mode`, you can reduce the testing time. For each mode, -please see [Controlling pass/fail +By passing `--pass $mode`, you can reduce the testing time. +For each mode, please see [Controlling pass/fail expectations](ui.md#controlling-passfail-expectations). ## Running tests with different "compare modes" -UI tests may have different output depending on certain "modes" that the -compiler is in. For example, when using the Polonius mode, a test `foo.rs` will +UI tests may have different output depending on certain "modes" that the compiler is in. +For example, when using the Polonius mode, a test `foo.rs` will first look for expected output in `foo.polonius.stderr`, falling back to the -usual `foo.stderr` if not found. The following will run the UI test suite in -Polonius mode: +usual `foo.stderr` if not found. +The following will run the UI test suite in Polonius mode: ```text ./x test tests/ui --compare-mode=polonius @@ -229,8 +229,8 @@ See [Compare modes](compiletest.md#compare-modes) for more details. ## Running tests manually -Sometimes it's easier and faster to just run the test by hand. Most tests are -just `.rs` files, so after [creating a rustup +Sometimes it's easier and faster to just run the test by hand. +Most tests are just `.rs` files, so after [creating a rustup toolchain](../building/how-to-build-and-run.md#creating-a-rustup-toolchain), you can do something like: @@ -238,57 +238,59 @@ can do something like: rustc +stage1 tests/ui/issue-1234.rs ``` -This is much faster, but doesn't always work. For example, some tests include +This is much faster, but doesn't always work. +For example, some tests include directives that specify specific compiler flags, or which rely on other crates, and they may not run the same without those options. ## Running tests on a remote machine Tests may be run on a remote machine (e.g. to test builds for a different -architecture). This is done using `remote-test-client` on the build machine to +architecture). +This is done using `remote-test-client` on the build machine to send test programs to `remote-test-server` running on the remote machine. -`remote-test-server` executes the test programs and sends the results back to -the build machine. `remote-test-server` provides *unauthenticated remote code +`remote-test-server` executes the test programs and sends the results back to the build machine. +`remote-test-server` provides *unauthenticated remote code execution* so be careful where it is used. -To do this, first build `remote-test-server` for the remote machine, e.g. for -RISC-V +To do this, first build `remote-test-server` for the remote machine +(using RISC-V as an example): ```text ./x build src/tools/remote-test-server --target riscv64gc-unknown-linux-gnu ``` -The binary will be created at -`./build/host/stage2-tools/$TARGET_ARCH/release/remote-test-server`. Copy this -over to the remote machine. +The binary will be created at `./build/host/stage2-tools/$TARGET_ARCH/release/remote-test-server`. +Copy this over to the remote machine. On the remote machine, run the `remote-test-server` with the `--bind -0.0.0.0:12345` flag (and optionally `-v` for verbose output). Output should look -like this: +0.0.0.0:12345` flag (and optionally `--verbose` flag). +Output should look like this: -```text -$ ./remote-test-server -v --bind 0.0.0.0:12345 +```console +$ ./remote-test-server --verbose --bind 0.0.0.0:12345 starting test server listening on 0.0.0.0:12345! ``` Note that binding the server to 0.0.0.0 will allow all hosts able to reach your -machine to execute arbitrary code on your machine. We strongly recommend either +machine to execute arbitrary code on your machine. +We strongly recommend either setting up a firewall to block external access to port 12345, or to use a more restrictive IP address when binding. -You can test if the `remote-test-server` is working by connecting to it and -sending `ping\n`. It should reply `pong`: +You can test if the `remote-test-server` is working by connecting to it and sending `ping\n`. +It should reply `pong`: -```text +```console $ nc $REMOTE_IP 12345 ping pong ``` To run tests using the remote runner, set the `TEST_DEVICE_ADDR` environment -variable then use `x` as usual. For example, to run `ui` tests for a RISC-V -machine with the IP address `1.2.3.4` use +variable then use `x` as usual. +For example, to run `ui` tests for a RISC-V machine with the IP address `1.2.3.4` use ```text export TEST_DEVICE_ADDR="1.2.3.4:12345" @@ -315,34 +317,34 @@ run "/tmp/work/test1018/a" [...] ``` -Tests are built on the machine running `x` not on the remote machine. Tests -which fail to build unexpectedly (or `ui` tests producing incorrect build +Tests are built on the machine running `x` not on the remote machine. +Tests which fail to build unexpectedly (or `ui` tests producing incorrect build output) may fail without ever running on the remote machine. ## Testing on emulators -Some platforms are tested via an emulator for architectures that aren't readily -available. For architectures where the standard library is well supported and +Some platforms are tested via an emulator for architectures that aren't readily available. +For architectures where the standard library is well supported and the host operating system supports TCP/IP networking, see the above instructions for testing on a remote machine (in this case the remote machine is emulated). -There is also a set of tools for orchestrating running the tests within the -emulator. Platforms such as `arm-android` and `arm-unknown-linux-gnueabihf` are -set up to automatically run the tests under emulation on GitHub Actions. The -following will take a look at how a target's tests are run under emulation. +There is also a set of tools for orchestrating running the tests within the emulator. +Platforms such as `arm-android` and `arm-unknown-linux-gnueabihf` are +set up to automatically run the tests under emulation on GitHub Actions. +The following will take a look at how a target's tests are run under emulation. -The Docker image for [armhf-gnu] includes [QEMU] to emulate the ARM CPU -architecture. Included in the Rust tree are the tools [remote-test-client] and +The Docker image for [armhf-gnu] includes [QEMU] to emulate the ARM CPU architecture. +Included in the Rust tree are the tools [remote-test-client] and [remote-test-server] which are programs for sending test programs and libraries -to the emulator, and running the tests within the emulator, and reading the -results. The Docker image is set up to launch `remote-test-server` and the +to the emulator, and running the tests within the emulator, and reading the results. +The Docker image is set up to launch `remote-test-server` and the build tools use `remote-test-client` to communicate with the server to coordinate running tests (see [src/bootstrap/src/core/build_steps/test.rs]). -To run on the iOS/tvOS/watchOS/visionOS simulator, we can similarly treat it as -a "remote" machine. A curious detail here is that the network is shared between -the simulator instance and the host macOS, so we can use the local loopback -address `127.0.0.1`. Something like the following should work: +To run on the iOS/tvOS/watchOS/visionOS simulator, we can similarly treat it as a "remote" machine. +A curious detail here is that the network is shared between +the simulator instance and the host macOS, so we can use the local loopback address `127.0.0.1`. +Something like the following should work: ```sh # Build the test server for the iOS simulator: @@ -384,7 +386,7 @@ On the [wasm32-wasip1 target support page] a minimum version is specified that y Some cmake commands that take a while and give a lot of very concerning c++ warnings... Then, in `bootstrap.toml`, point to the sysroot like so: -``` +```toml [target.wasm32-wasip1] wasi-root = "/build/sysroot/install/share/wasi-sysroot" ```