Allow setup and clean commands

[sgpthomas]

If you want to run a makefile to generate the file to do diff testing on, you are forced to make the command look like:

cmd = "make {} 1>/dev/null 2>/dev/null; cat {}; rm {}"

It would be nice if you could instead do:

setup = "make {}"
cmd = "cat {}"
cleanup = "rm {}"

Eventually, you could imagine automating the cleanup to remove any files generated during the setup command.

[rachitnigam]

There are two problems here:

1. It gets harder to read complicated scripts in runt descriptions.
2. Runt captures both STDOUT and STDERR making it more annoying to redirect everything.

For 1, I recommend using TOML multiline strings.

For 2, I think perhaps runt should not capture STDERR by default. I can implement that.

[TheZoq2]

This is also something I'd like and I'd argue against point 1. For a runt script which needs setup and cleanup, I'd say it is easier to read

setup = "make {}"
cmd = "cat {}"
cleanup = "rm {}"

than

cmd = "make {} 1>/dev/null 2>/dev/null; cat {}; rm {}"

because it is made clear what is being tested vs what is just there for setting and cleaning up

I'd also be interested in global setup commands in addition to setup commands local to the tests

For example, I now have to remember run instead of just runt which has caught me off guard several times

cargo build && runt

[rachitnigam]

Yeah, another question I have with this is should the setup be considered atomic? If so, we'd need some way of enforcing that for each test suite. Right now, runt can just collect all test specifications and fire them off in parallel without worrying about which test suite they come from. If this is needed then we'd need some way of tracking that dependency in the test executor and make sure that each test suite is correctly setup before firing them off.

[TheZoq2]

That's fair. Perhaps it could internally be merged into setup && cmd && cleanup

[rachitnigam]

We could but that won't fix the whole cargo build before runt problem because you don't actually want cargo build before each test. Another option is just a universal setup step that runs before any test executes which would make it at least easier to do the whole cargo build thing

[TheZoq2]

Yeah that's what i had in mind. One global setup that is run sequentially, then a bunch of independent setup cmd cleanup and then perhaps a final global cleanup

[rachitnigam]

Ah okay, that seems like a reasonable thing to start with in that case. Let me know if you're interested in implementing this and I can help get started with it!