Rollup merge of rust-lang#142431 - Kobzol:bootstrap-snapshot-tests, r=jieyouxu

Add initial version of snapshot tests to bootstrap

When making any changes to bootstrap (steps), it is very difficult to realize how does it affect various common bootstrap commands, and if everything still works as we expect it to. We are far away from having actual end-to-end tests, but what we could at least do is have a way of testing what steps does bootstrap execute in dry run mode. Now, we already have something like this in `src/bootstrap/src/core/builder/tests.rs`, however that is quite limited, because it only checks executed steps for a specific impl of `Step` and it does not consider step order.

Recently, when working on what I thought was one of the simplest possible step untanglings in bootstrap (rust-lang#142357), I ran into errors in tests that were quite hard to debug. Partly also because the current staging test diffs are multiline and use `Debug` output, so it's quite difficult for me to make sense of them.

In this PR, I introduce `insta`, which allows writing snapshot tests in a very simple way. With it, I want to allow writing tests that will clearly show us what is going on during bootstrap execution, and then write golden tests for `build/check/test` stage `0/1/2` for compiler/std/tools etc., to make sure that we don't regress something, and also to help with [#t-infra/bootstrap > Proposal to cleanup stages and steps after the redesign](https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Proposal.20to.20cleanup.20stages.20and.20steps.20after.20the.20redesign/with/523488806), to help avoid a situation where we would (again) have to make a flurry of staging changes because of unexpected consequences.

In the snapshot tests, we currently render the build of rustc, std and LLVM. Currently I render the executed steps using downcasting, which is not super pretty, but it allows us to make the test rendering localized in one place, and it's IMO enough for now.

I implemented only a single test using the new machinery. Maybe if you take a look at it, you will understand why 😆 Bootstrap currently does some peculiar things, such as running a stage 0 std step (even though stage 0 std no longer exists) and running the Rustc stage 0 -> 1 step twice, once with a single crates, once with all rustc crates. So I think that even with this single step, there will be a bunch of things to fix in the near future...

The way we currently prepare the Config test fixtures is far from ideal, this is something I think ``@Shourya742`` could work on as a part of their GSoC project (remove as much command execution from Config construction as possible, actually run bootstrap on a temporary directory instead of running it on the rustc checkout, create a Builder-like API for creating the Config test fixtures).

r? ``@jieyouxu``

Author: Kobzol

src/bootstrap/Cargo.lock

@@ -44,6 +44,7 @@ dependencies = [
  "fd-lock",
  "home",
  "ignore",
+ "insta",
  "junction",
  "libc",
  "object",
@@ -158,6 +159,18 @@ dependencies = [
  "cc",
 ]
 
+[[package]]
+name = "console"
+version = "0.15.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "054ccb5b10f9f2cbf51eb355ca1d05c2d279ce1804688d0db74b4733a5aeafd8"
+dependencies = [
+ "encode_unicode",
+ "libc",
+ "once_cell",
+ "windows-sys 0.59.0",
+]
+
 [[package]]
 name = "cpufeatures"
 version = "0.2.15"
@@ -218,6 +231,12 @@ dependencies = [
  "crypto-common",
 ]
 
+[[package]]
+name = "encode_unicode"
+version = "1.0.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "34aa73646ffb006b8f5147f3dc182bd4bcb190227ce861fc4a4844bf8e3cb2c0"
+
 [[package]]
 name = "errno"
 version = "0.3.11"
@@ -323,6 +342,17 @@ dependencies = [
  "winapi-util",
 ]
 
+[[package]]
+name = "insta"
+version = "1.43.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "154934ea70c58054b556dd430b99a98c2a7ff5309ac9891597e339b5c28f4371"
+dependencies = [
+ "console",
+ "once_cell",
+ "similar",
+]
+
 [[package]]
 name = "itoa"
 version = "1.0.11"
@@ -675,6 +705,12 @@ version = "1.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64"
 
+[[package]]
+name = "similar"
+version = "2.7.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bbbb5d9659141646ae647b42fe094daf6c6192d1620870b449d9557f748b2daa"
+
 [[package]]
 name = "smallvec"
 version = "1.13.2"

src/bootstrap/Cargo.toml

@@ -84,6 +84,7 @@ features = [
 [dev-dependencies]
 pretty_assertions = "1.4"
 tempfile = "3.15.0"
+insta = "1.43"
 
 # We care a lot about bootstrap's compile times, so don't include debuginfo for
 # dependencies, only bootstrap itself.

src/bootstrap/README.md

@@ -200,6 +200,10 @@ please file issues on the [Rust issue tracker][rust-issue-tracker].
 [rust-bootstrap-zulip]: https://rust-lang.zulipchat.com/#narrow/stream/t-infra.2Fbootstrap
 [rust-issue-tracker]: https://github.com/rust-lang/rust/issues
 
+## Testing
+
+To run bootstrap tests, execute `x test bootstrap`. If you want to bless snapshot tests, then install `cargo-insta` (`cargo install cargo-insta`) and then run `cargo insta review --manifest-path src/bootstrap/Cargo.toml`.
+
 ## Changelog
 
 Because we do not release bootstrap with versions, we also do not maintain CHANGELOG files. To

src/bootstrap/src/core/build_steps/test.rs

@@ -3058,6 +3058,8 @@ impl Step for Bootstrap {
         cargo
             .rustflag("-Cdebuginfo=2")
             .env("CARGO_TARGET_DIR", builder.out.join("bootstrap"))
+            // Needed for insta to correctly write pending snapshots to the right directories.
+            .env("INSTA_WORKSPACE_ROOT", &builder.src)
             .env("RUSTC_BOOTSTRAP", "1");
 
         // bootstrap tests are racy on directory creation so just run them one at a time.

src/bootstrap/src/core/builder/tests.rs

@@ -15,11 +15,12 @@ static TEST_TRIPLE_2: &str = "i686-unknown-hurd-gnu";
 static TEST_TRIPLE_3: &str = "i686-unknown-netbsd";
 
 fn configure(cmd: &str, host: &[&str], target: &[&str]) -> Config {
-    configure_with_args(&[cmd.to_owned()], host, target)
+    configure_with_args(&[cmd], host, target)
 }
 
-fn configure_with_args(cmd: &[String], host: &[&str], target: &[&str]) -> Config {
-    let mut config = Config::parse(Flags::parse(cmd));
+fn configure_with_args(cmd: &[&str], host: &[&str], target: &[&str]) -> Config {
+    let cmd = cmd.iter().copied().map(String::from).collect::<Vec<_>>();
+    let mut config = Config::parse(Flags::parse(&cmd));
     // don't save toolstates
     config.save_toolstates = None;
     config.set_dry_run(DryRun::SelfCheck);
@@ -67,7 +68,7 @@ fn run_build(paths: &[PathBuf], config: Config) -> Cache {
 fn check_cli<const N: usize>(paths: [&str; N]) {
     run_build(
         &paths.map(PathBuf::from),
-        configure_with_args(&paths.map(String::from), &[TEST_TRIPLE_1], &[TEST_TRIPLE_1]),
+        configure_with_args(&paths, &[TEST_TRIPLE_1], &[TEST_TRIPLE_1]),
     );
 }
 
@@ -1000,8 +1001,7 @@ mod sysroot_target_dirs {
 /// cg_gcc tests instead.
 #[test]
 fn test_test_compiler() {
-    let cmd = &["test", "compiler"].map(str::to_owned);
-    let config = configure_with_args(cmd, &[TEST_TRIPLE_1], &[TEST_TRIPLE_1]);
+    let config = configure_with_args(&["test", "compiler"], &[TEST_TRIPLE_1], &[TEST_TRIPLE_1]);
     let cache = run_build(&config.paths.clone(), config);
 
     let compiler = cache.contains::<test::CrateLibrustc>();
@@ -1034,8 +1034,7 @@ fn test_test_coverage() {
         // Print each test case so that if one fails, the most recently printed
         // case is the one that failed.
         println!("Testing case: {cmd:?}");
-        let cmd = cmd.iter().copied().map(str::to_owned).collect::<Vec<_>>();
-        let config = configure_with_args(&cmd, &[TEST_TRIPLE_1], &[TEST_TRIPLE_1]);
+        let config = configure_with_args(cmd, &[TEST_TRIPLE_1], &[TEST_TRIPLE_1]);
         let mut cache = run_build(&config.paths.clone(), config);
 
         let modes =
@@ -1207,8 +1206,7 @@ fn test_get_tool_rustc_compiler() {
 /// of `Any { .. }`.
 #[test]
 fn step_cycle_debug() {
-    let cmd = ["run", "cyclic-step"].map(str::to_owned);
-    let config = configure_with_args(&cmd, &[TEST_TRIPLE_1], &[TEST_TRIPLE_1]);
+    let config = configure_with_args(&["run", "cyclic-step"], &[TEST_TRIPLE_1], &[TEST_TRIPLE_1]);
 
     let err = panic::catch_unwind(|| run_build(&config.paths.clone(), config)).unwrap_err();
     let err = err.downcast_ref::<String>().unwrap().as_str();
@@ -1233,3 +1231,81 @@ fn any_debug() {
     // Downcasting to the underlying type should succeed.
     assert_eq!(x.downcast_ref::<MyStruct>(), Some(&MyStruct { x: 7 }));
 }
+
+/// The staging tests use insta for snapshot testing.
+/// See bootstrap's README on how to bless the snapshots.
+mod staging {
+    use crate::core::builder::tests::{
+        TEST_TRIPLE_1, configure, configure_with_args, render_steps, run_build,
+    };
+
+    #[test]
+    fn build_compiler_stage_1() {
+        let mut cache = run_build(
+            &["compiler".into()],
+            configure_with_args(&["build", "--stage", "1"], &[TEST_TRIPLE_1], &[TEST_TRIPLE_1]),
+        );
+        let steps = cache.into_executed_steps();
+        insta::assert_snapshot!(render_steps(&steps), @r"
+        [build] rustc 0 <target1> -> std 0 <target1>
+        [build] llvm <target1>
+        [build] rustc 0 <target1> -> rustc 1 <target1>
+        [build] rustc 0 <target1> -> rustc 1 <target1>
+        ");
+    }
+}
+
+/// Renders the executed bootstrap steps for usage in snapshot tests with insta.
+/// Only renders certain important steps.
+/// Each value in `steps` should be a tuple of (Step, step output).
+fn render_steps(steps: &[(Box<dyn Any>, Box<dyn Any>)]) -> String {
+    steps
+        .iter()
+        .filter_map(|(step, output)| {
+            // FIXME: implement an optional method on Step to produce metadata for test, instead
+            // of this downcasting
+            if let Some((rustc, output)) = downcast_step::<compile::Rustc>(step, output) {
+                Some(format!(
+                    "[build] {} -> {}",
+                    render_compiler(rustc.build_compiler),
+                    // FIXME: return the correct stage from the `Rustc` step, now it behaves weirdly
+                    render_compiler(Compiler::new(rustc.build_compiler.stage + 1, rustc.target)),
+                ))
+            } else if let Some((std, output)) = downcast_step::<compile::Std>(step, output) {
+                Some(format!(
+                    "[build] {} -> std {} <{}>",
+                    render_compiler(std.compiler),
+                    std.compiler.stage,
+                    std.target
+                ))
+            } else if let Some((llvm, output)) = downcast_step::<llvm::Llvm>(step, output) {
+                Some(format!("[build] llvm <{}>", llvm.target))
+            } else {
+                None
+            }
+        })
+        .map(|line| {
+            line.replace(TEST_TRIPLE_1, "target1")
+                .replace(TEST_TRIPLE_2, "target2")
+                .replace(TEST_TRIPLE_3, "target3")
+        })
+        .collect::<Vec<_>>()
+        .join("\n")
+}
+
+fn downcast_step<'a, S: Step>(
+    step: &'a Box<dyn Any>,
+    output: &'a Box<dyn Any>,
+) -> Option<(&'a S, &'a S::Output)> {
+    let Some(step) = step.downcast_ref::<S>() else {
+        return None;
+    };
+    let Some(output) = output.downcast_ref::<S::Output>() else {
+        return None;
+    };
+    Some((step, output))
+}
+
+fn render_compiler(compiler: Compiler) -> String {
+    format!("rustc {} <{}>", compiler.stage, compiler.host)
+}

src/bootstrap/src/utils/cache.rs

@@ -17,6 +17,7 @@ use std::borrow::Borrow;
 use std::cell::RefCell;
 use std::cmp::Ordering;
 use std::collections::HashMap;
+use std::fmt::Debug;
 use std::hash::{Hash, Hasher};
 use std::marker::PhantomData;
 use std::ops::Deref;
@@ -208,38 +209,51 @@ pub static INTERNER: LazyLock<Interner> = LazyLock::new(Interner::default);
 /// any type in its output. It is a write-once cache; values are never evicted,
 /// which means that references to the value can safely be returned from the
 /// `get()` method.
-#[derive(Debug)]
-pub struct Cache(
-    RefCell<
+#[derive(Debug, Default)]
+pub struct Cache {
+    cache: RefCell<
         HashMap<
             TypeId,
             Box<dyn Any>, // actually a HashMap<Step, Interned<Step::Output>>
         >,
     >,
-);
+    #[cfg(test)]
+    /// Contains steps in the same order in which they were executed
+    /// Useful for tests
+    /// Tuples (step, step output)
+    executed_steps: RefCell<Vec<(Box<dyn Any>, Box<dyn Any>)>>,
+}
 
 impl Cache {
     /// Creates a new empty cache.
     pub fn new() -> Cache {
-        Cache(RefCell::new(HashMap::new()))
+        Cache::default()
     }
 
     /// Stores the result of a computation step in the cache.
     pub fn put<S: Step>(&self, step: S, value: S::Output) {
-        let mut cache = self.0.borrow_mut();
+        let mut cache = self.cache.borrow_mut();
         let type_id = TypeId::of::<S>();
         let stepcache = cache
             .entry(type_id)
             .or_insert_with(|| Box::<HashMap<S, S::Output>>::default())
             .downcast_mut::<HashMap<S, S::Output>>()
             .expect("invalid type mapped");
         assert!(!stepcache.contains_key(&step), "processing {step:?} a second time");
+
+        #[cfg(test)]
+        {
+            let step: Box<dyn Any> = Box::new(step.clone());
+            let output: Box<dyn Any> = Box::new(value.clone());
+            self.executed_steps.borrow_mut().push((step, output));
+        }
+
         stepcache.insert(step, value);
     }
 
     /// Retrieves a cached result for the given step, if available.
     pub fn get<S: Step>(&self, step: &S) -> Option<S::Output> {
-        let mut cache = self.0.borrow_mut();
+        let mut cache = self.cache.borrow_mut();
         let type_id = TypeId::of::<S>();
         let stepcache = cache
             .entry(type_id)
@@ -252,8 +266,8 @@ impl Cache {
 
 #[cfg(test)]
 impl Cache {
-    pub fn all<S: Ord + Clone + Step>(&mut self) -> Vec<(S, S::Output)> {
-        let cache = self.0.get_mut();
+    pub fn all<S: Ord + Step>(&mut self) -> Vec<(S, S::Output)> {
+        let cache = self.cache.get_mut();
         let type_id = TypeId::of::<S>();
         let mut v = cache
             .remove(&type_id)
@@ -265,7 +279,12 @@ impl Cache {
     }
 
     pub fn contains<S: Step>(&self) -> bool {
-        self.0.borrow().contains_key(&TypeId::of::<S>())
+        self.cache.borrow().contains_key(&TypeId::of::<S>())
+    }
+
+    #[cfg(test)]
+    pub fn into_executed_steps(mut self) -> Vec<(Box<dyn Any>, Box<dyn Any>)> {
+        mem::take(&mut self.executed_steps.borrow_mut())
     }
 }