Improve linter UI test runners (and fix them on Windows!)

[BD103]

Sorry for the vague title! This PR is a refactor of our linter UI test runners (ui.rs and ui_cargo.rs) with a number of benefits:

• Progress towards running UI tests on Windows by building bevy and its dependencies with DependencyBuilder (fixes Fix CI on Windows #568 (comment), which prevented Fix CI on Windows #568 from being merged)
• New @exit-code: CODE annotation for Cargo UI tests, used to ensure checks pass
• Support passing arguments to Cargo UI tests (e.g. cargo test --test ui_cargo -- --bless)

Here's a few notes I have on specific changes / design decisions:

Removing test_utils

The test_utils module had two main features:

• Providing a base Config that both ui.rs and ui_cargo.rs could use, avoiding unnecessary duplication
• Building and locating the compiled bevy crate so UI tests could import it

When looking over the base_config(), I realized ui.rs used all of the configuration without modification, while ui_cargo.rs essentially overwrote the entire thing. Because of that, I think it is much clearer to define each Config separately, rather than trying to use a shared base when there is little overlap. I think it worked! Compare the before-and-after for ui_cargo.rs:

bevy_cli/bevy_lint/tests/ui_cargo.rs

Lines 18 to 51 in 569ec9e

	let mut config = base_config("ui-cargo").unwrap();
	let defaults = config.comment_defaults.base();
	// The driver returns a '101' on error.
	// This allows for any status code to be considered a success.
	defaults.exit_status = None.into();
	defaults.require_annotations = None.into();
	// This sets the '--manifest-path' flag
	config.program.input_file_flag = CommandBuilder::cargo().input_file_flag;
	config.program.out_dir_flag = CommandBuilder::cargo().out_dir_flag;
	// Do not print cargo log messages
	config.program.args = vec!["--quiet".into(), "--color".into(), "never".into()];
	let current_exe_path = env::current_exe().unwrap();
	let deps_path = current_exe_path.parent().unwrap();
	let profile_path = deps_path.parent().unwrap();
	// Specify the binary to use when executing tests with this `Config`
	config.program.program = profile_path.join(if cfg!(windows) {
	"bevy_lint_driver.exe"
	} else {
	"bevy_lint_driver"
	});
	config.program.program.set_file_name(if cfg!(windows) {
	"bevy_lint.exe"
	} else {
	"bevy_lint"
	});
	// this clears the default `--edition` flag
	config.comment_defaults.base().custom.clear();

bevy_cli/bevy_lint/tests/ui_cargo.rs

Lines 21 to 40 in d80936b

	let mut config = Config {
	// We need to specify the host tuple manually, because if we don't then `ui_test` will try
	// running `bevy_lint -vV` to discover the host and promptly error because `bevy_lint`
	// doesn't recognize the `-vV` flag.
	host: Some(host_tuple()),
	program: CommandBuilder {
	program: linter_path.into(),
	args: vec!["--color=never".into(), "--quiet".into()],
	out_dir_flag: Some("--target-dir".into()),
	input_file_flag: Some("--manifest-path".into()),
	envs: Vec::new(),
	cfg_flag: None,
	},
	out_dir: PathBuf::from("../target/ui"),
	..Config::cargo(Path::new("tests/ui-cargo"))
	};
	// We haven't found a way to get error annotations like `#~v ERROR: msg` to work, so we disable
	// the requirement for them.
	config.comment_defaults.base().require_annotations = None.into();

The second feature test_utils provided was building and locating libbevy.rlib. Turns out we can replace that with...

Using DependencyBuilder

ui_test provides a DependencyBuilder that makes it easy to build dependencies used by UI tests. DependencyBuilder provides a shift in how we manage dependencies:

test_utils	DependencyBuilder
Dependencies specified in linter's [dev-dependencies]	Dependencies specified by lint-ui-dependencies crate
Dependencies built by default when cargo test runs	Dependencies built only when UI tests run
Dependency .rlibs located by parsing cargo build's JSON output (which was jank enough that I wrote a blog post about an issue I found with it, whose fix never landed because it was in #568)	Dependency .rlibs handled by ui_test
Passing dependencies to bevy_lint_driver with --extern and -L	Dependency flags handled by ui_test
Does not respect cargo::rustc-link-search in build scripts, causing windows dependency to fail to compile on Windows	Handles cargo::rustc-link-search: oli-obk/ui_test#220
Bevy's derive macros successfully detect that bevy::ecs should be imported rather than bevy_ecs	Bevy's derive macros cannot detect that bevy::ecs should be imported

Overall, using DependencyBuilder makes it so we can build and run UI tests on Windows while also making maintaining the UI tests easier. The only downside I've found was that BevyManifest isn't able to find these dependencies, so it isn't able to deduce that bevy::ecs and bevy::reflect should be imported rather than bevy_ecs and bevy_reflect. I hacked around it here, but the solution isn't amazing.

Properly query host tuple

One of the oldest issues (it's there in #32!) I ran into when introducing the UI test harness was ui_test trying to auto-detect the host tuple by calling bevy_lint_driver -vV, which the linter driver skipped because it expected bevy_lint_driver rustc -vV instead. I hacked around the issue by setting the host to an empty string Some(String::new()), but DependencyBuilder doesn't like that because it tries passing the host to cargo build and ends up running cargo build --target=. Because there's no value for --target=, Cargo exits with an error.

Because there was no avoiding it anymore, I chose to instead run rustc --print=host-tuple to get the actual host tuple (which for me returns aarch64-apple-darwin). I duplicated this solution to both ui.rs and ui_cargo.rs rather than re-introducing test_utils, but I'm not set on that decision.

#@exit-status in Cargo UI tests

We previously discarded the exit code of Cargo UI tests because fail tests returned exit code 101, which ui_test mistakenly thought was a compiler ICE rather than a proper error. Instead of blanketly allowing all exit codes, I wrote a small annotation that lets us specify what the exit code of a specific test should be. Now all tests intended to fail should have this at the beginning:

#@exit-status: 101

[package]
# ...

And all tests intending to pass should use the normal #@check-pass:

#@check-pass

[package]
# ...

I couldn't figure out a way to reprogram ui_test to accept 101 as the normal error code, so this is the best thing I came up with.

Argument parsing in ui_cargo.rs

ui_cargo.rs uses ui_test::run_tests_generic() because it needs to configure the runner to work on Cargo.toml files, not *.rs files. run_tests_generic() does not do any argument parsing, however, meaning flags like --bless were completely ignored.

Since --bless is super useful, I copied the argument parsing code from run_tests() to ui_cargo.rs. Cool stuff!

[BD103]

We don't need serde_json anymore, as we're no longer deserializing JSON messages manually. ui_test handles that for us!

[BD103]

This dependency has been moved to bevy_lint/tests/dependencies/Cargo.toml.

[BD103]

This line is important: once we start cargo publish-ing the CLI and linter, this will prevent us from accidentally publishing this crate :)

[BD103]

This #[expect(...)] is because we are using the standard library's assert!() macro. It's fine in this situation because we can't and shouldn't use the linter's version.

[BD103]

I figured it was easiest to add ui-lint-dependencies to the workspace; this way it shares the target directory for caching and Cargo.lock for dependency sharing.