CI: can filter jobs to run depending on modified files

[mockersf]

Objective

• Allow Bevy CI jobs to be filtered depending on what needs to be tested
• But not in a way that is too dependent on GitHub actions
• But still be compatible with branch protection rules

Solution

• Add a command cargo run -p ci -- what-to-run --trigger pull_request --head main --rust-version stable that list commands to be run
• Add a CI job that runs the above command and outputs the results
• Add a CI matrix job that runs all the jobs from the output
• Add a CI job that has a single status for all matrix jobs, and can be used for branch protection rules

For now, this is just the shape of it to open discussion, no filtering is done

[mockersf]

@NicoZweifel @janhohenheim you were interested in this

[NicoZweifel]

Nice, it's apparent that you put a lot of thought into this already, I think this looks pretty good for a PoC (maybe it could be converted to a draft for now). I do have a few thoughts about it:

A minor maintenance issue in ci.rs is that adding a Command requires editing multiple match statements and locations, which either don't compile or fail silently, but that overhead might be a trade-off that has to be made somewhere.

A job runner is always provisioned and the tools are always built but I think that is an okay trade-off to make.
I noticed that the ci crate is very lean, which is nice. Is it faster to fully re-compile instead of using a cache in the prepare stage as well or is there another reason?

Maybe you already thought about this too, but the WhatToRunCommand will become a huge function with lots of responsibilities, so personally I would improve the WhatToRunCommand to work with workflows/strategies, e.g.:

pub struct Context {
    pub trigger: Trigger,
    pub changed_files: Vec<String>,
}

pub trait Workflow {
    fn suite(&self, context: &Context) -> Vec<Commands>;
}

This also simplifies some of the existing code for running everything since you can just return a Full strategy that runs everything (reduces the maintenance issue touched earlier), e.g. here:

    fn prepare<'a>(&self, sh: &'a xshell::Shell) -> Vec<PreparedCommand<'a>> {
        let args = self.into();
        match &self.command {
            Some(command) => command.prepare(sh, args),
            None => {
                // Note that we are running the subcommands directly rather than using any aliases
                let mut cmds = vec![];
                cmds.append(&mut commands::FormatCommand::default().prepare(sh, args));
                cmds.append(&mut commands::ClippyCommand::default().prepare(sh, args));
                cmds.append(&mut commands::TestCommand::default().prepare(sh, args));
                cmds.append(&mut commands::TestCheckCommand::default().prepare(sh, args));
                cmds.append(&mut commands::IntegrationTestCommand::default().prepare(sh, args));
                cmds.append(
                    &mut commands::IntegrationTestCheckCommand::default().prepare(sh, args),
                );
                cmds.append(
                    &mut commands::IntegrationTestCleanCommand::default().prepare(sh, args),
                );
                cmds.append(&mut commands::DocCheckCommand::default().prepare(sh, args));
                cmds.append(&mut commands::DocTestCommand::default().prepare(sh, args));
                cmds.append(&mut commands::CompileCheckCommand::default().prepare(sh, args));
                cmds.append(&mut commands::CompileFailCommand::default().prepare(sh, args));
                cmds.append(&mut commands::BenchCheckCommand::default().prepare(sh, args));
                cmds.append(&mut commands::ExampleCheckCommand::default().prepare(sh, args));

                cmds
            }
        }
    }

[NicoZweifel]

This seems a bit odd, maybe it's just temporary?

[NicoZweifel]

Maybe we could differentiate between shell and rust commands?

[NicoZweifel]

This is a bit fragile. I assume this is so that there is no dependency on github ($GITHUB_OUTPUT) here?

[NicoZweifel]

I think I would prefer to pass an env variable as argument that the script should set, following the same logic/pattern (prepare/run) that the rest of the commands use.

The command could prepare: set $ENV_VARIABLE = [workflow] , which then resolves to set $GITHUB_OUTPUT = [cargo test, etc...].

I would also recommend something like the Workflow trait I mentioned earlier to help with organizing the workflows. Happy to help with any of this if you'd like.

[NicoZweifel]

I'm gonna step back from this PR and related discussions because I don't understand the vision or reasoning behind it.

I think there are major contradictions here, and I don't believe this direction will actually be better than the dependence on github until they are resolved. I'd be happy to help if it clears up, but as it stands it's not logical to me and I feel like my attempts at improving or discussing this have been dismissed.

Good luck with the implementation!