Better cargo-less-support

[TroyKomodo]

Motivation

Related to #842

The idea is to support bazel / other build systems where they do not use cargo to build the project and tests. Currently this is possible but is a bit hacky and would be great if we could get a more integrated way to do so.

Proposal

The alternative below works well and I haven't had any issues with it. I think it would be cool if we could provide nextest with a list of binaries and the cargo metadata json and then have nextest discover the tests without building. Ideally if we could get rid of the cargo metadata requirement too that would be nicer.

Alternatives

I wrote a test-runner binary with the following code

use std::collections::BTreeSet;

use camino::Utf8PathBuf;
use nextest_filtering::ParseContext;
use nextest_metadata::{BuildPlatform, RustBinaryId, RustTestBinaryKind};
use nextest_runner::cargo_config::{CargoConfigs, EnvironmentMap};
use nextest_runner::config::NextestConfig;
use nextest_runner::double_spawn::DoubleSpawnInfo;
use nextest_runner::input::InputHandlerKind;
use nextest_runner::list::{RustBuildMeta, RustTestArtifact, TestExecuteContext};
use nextest_runner::platform::BuildPlatforms;
use nextest_runner::reporter::structured::StructuredReporter;
use nextest_runner::reporter::{ReporterBuilder, ReporterStderr};
use nextest_runner::reuse_build::PathMapper;
use nextest_runner::signal::SignalHandlerKind;
use nextest_runner::target_runner::TargetRunner;
use nextest_runner::test_filter::{FilterBound, RunIgnored, TestFilterBuilder};

fn main() {
    let mut args = std::env::args().skip(1);
    let Some(metadata_path) = args.next() else {
        panic!(
            "invalid usage: {} <metadata.json> <package> <test-binary>",
            std::env::args().next().unwrap_or_default()
        )
    };

    let Some(package) = args.next() else {
        panic!(
            "invalid usage: {} <metadata.json> <package> <test-binary>",
            std::env::args().next().unwrap_or_default()
        )
    };

    let Some(test_binary_path) = args.next() else {
        panic!(
            "invalid usage: {} <metadata.json> <package> <test-binary>",
            std::env::args().next().unwrap_or_default()
        )
    };

    let metadata_path = Utf8PathBuf::from(metadata_path);
    let test_binary_path = Utf8PathBuf::from(test_binary_path);

    let metadata = std::fs::read_to_string(metadata_path).unwrap();
    let metadata = guppy::CargoMetadata::parse_json(metadata).unwrap();
    let graph = metadata.build_graph().unwrap();

    let package = graph.packages().find(|p| p.name() == package).unwrap();

    let double_spawn = DoubleSpawnInfo::disabled();
    let target_runner = TargetRunner::empty();

    let ctx = TestExecuteContext {
        double_spawn: &double_spawn,
        profile_name: "test",
        target_runner: &target_runner,
    };

    let artifact = RustTestArtifact {
        binary_id: RustBinaryId::new("test-id"),
        binary_name: test_binary_path.file_name().unwrap().to_string(),
        binary_path: test_binary_path,
        cwd: Utf8PathBuf::from(std::env::current_dir().unwrap().to_string_lossy().as_ref()),
        build_platform: BuildPlatform::Target,
        kind: RustTestBinaryKind::LIB,
        non_test_binaries: BTreeSet::new(),
        package: package,
    };

    let build_platforms = BuildPlatforms::new_with_no_target().unwrap();
    let config =
        NextestConfig::from_sources(Utf8PathBuf::new(), &ParseContext::new(&graph), None, [], &BTreeSet::new()).unwrap();
    let profile = config.profile("default").unwrap().apply_build_platforms(&build_platforms);
    let meta = RustBuildMeta::new("", build_platforms).map_paths(&PathMapper::noop());
    let filter = TestFilterBuilder::default_set(RunIgnored::Default);
    let env = EnvironmentMap::new(&CargoConfigs::new([] as [&str; 0]).unwrap());

    let list = nextest_runner::list::TestList::new(
        &ctx,
        Some(artifact),
        meta,
        &filter,
        Utf8PathBuf::new(),
        env,
        &profile,
        FilterBound::DefaultSet,
        1,
    )
    .unwrap();

    let runner = nextest_runner::runner::TestRunnerBuilder::default()
        .build(
            &list,
            &profile,
            Vec::new(),
            SignalHandlerKind::Standard,
            InputHandlerKind::Standard,
            double_spawn,
            target_runner,
        )
        .unwrap();

    let mut reporter = ReporterBuilder::default()
        .set_colorize(false)
        .set_hide_progress_bar(true)
        .build(&list, &profile, ReporterStderr::Terminal, StructuredReporter::new());

    let r = runner
        .execute(|event| {
            reporter.report_event(event).unwrap();
        })
        .unwrap();

    if r.has_failures() {
        std::process::exit(1)
    }
}

Then used it like such

Made a nicer macro wrapper around it so i dont have to do any

rust_test(
    name = "__rust_test",
    crate = ":example",
    aliases = aliases,
    compile_data = data,
    data = test_data,
    env = test_env,
    tags = ["manual"],
    rustc_flags = [
        "--cfg=bazel_test",
    ],
    visibility = ["//visibility:private"],
)

sh_test(
    name = "test",
    srcs = ["//dev-tools/test-runner:script"],
    args = ["$(location //dev-tools/test-runner)", "$(location //:cargo_metadata)", crate_name, "$(location :__rust_test)"],
    data = [":__rust_test", "//dev-tools/test-runner", "//:cargo_metadata"] + test_data,
    tags = test_tags,
    env = test_env,
)

This works well and I havent had any issues with it. I think it would be cool if we could provide nextest with a list of binaries and the cargo metadata json asnd then have nextest discover the tests without building. Ideally if we could get rid of the cargo metadata requirement too that would be nice.

Additional context

No response

[TroyKomodo]

i am not an expert on nextest's internals so I am unsure if my usage of nextest_runner is correct. I the way I came to the above code was basically finding the function i wanted to call runner.execute and then looking at how I could get there providing the minimal amount of setup.

[sunshowers]

I think what you have is generally fine, though it isn't quite complete. (But it doesn't need to be, since you're tackling a more limited use case than nextest.)

Note that nextest-runner is not a stable interface and each nextest release generally has some breaking changes to it. If an older version keeps working for you, you can stay there of course.

[sunshowers]

Thanks for the issue. You can provide the list of binaries and cargo metadata JSON over the command line via --binaries-metadata and --cargo-metadata respectively, though the binary list is a custom JSON blob (see https://docs.rs/nextest-metadata/latest/nextest_metadata/struct.BinaryListSummary.html).

Agreed it would be nice to have an integrated solution. Unfortunately there are a lot of little details here to get right, such as paths to add to LD_LIBRARY_PATH, so a simple list of binaries isn't enough in many cases. I'd definitely welcome if someone did the work, though my impression is there's quite a bit of it.

[TroyKomodo]

I see, I tried using this but it seems that when using binary ntr --binaries-metadata <path> --cargo-metadata <path> it seems to effect the env variables / working directory which changes the way the build behaves. it seems to run the tests from the workspace root directory as apposed to the directory which the binary was invoked from.

[sunshowers]

Yeah that's true -- you can use --workspace-remap and --target-dir-remap to get around that, I think.

You're in a little bit of uncharted territory here, fair warning. I'd look at how running tests out of an archive works as a guide.