Tracking issue for benchmark support

[sunshowers]

• Initial implementation: [cargo-nextest] add a bench command #2417
• Documentation: https://nexte.st/docs/features/benchmarks/

TODO:

• Add a machine-readable way to output benchmark results, so nextest can collect them, publish them in JSON output, etc
• Solicit feedback

[eduardorittner]

Hi! I'd like to take a stab at adding the machine-readable output, if that's ok. From what I can tell after a quick look this would entail:

1. Add a --message-format option for the bench command (updating BenchReporterOpts accordingly)
2. Add a new benchmark event to ReporterEvent
3. Add benchmark support to StructuredReporter
4. Add benchmark support to JUnit reporting?

Does this look right? Or should the benchmark reporting infrastructure be totally separate from test reporting?

[sunshowers]

Ah I'm actually looking for the opposite, i.e. benchmark harnesses like Criterion reporting results to nextest so they can be aggregated and such. Right now benchmark output is passed through, so there's no way for nextest to know anything about benchmarks results.

[sunshowers]

But what you're describing is also valuable, I think -- I'd spend some time investigating what libtest does for benchmark results, so --message-format libtest-json can work.

[eduardorittner]

Sorry for the misunderstanding, I'd rather help out on what you meant the first time. So just to clarify:

Currently, cargo nextest bench runs benchmarks and prints their output, but it doesn't capture output (CaptureStrategy::None). This means that nextest has no way of knowing benchmark numbers, performance metrics, etc. Also of note is that nextest bench works with any benchmark harness (standard benchmark harness, criterion, etc.)

The feature you want is for nextest to be able to parse the output from these harnesses so that it can collect, record, display and do whatever else it wants with the data. From a quick look both cargo bench and criterion bench have a --message-format=json option which outputs structured benchmark data, but it looks like criterion's format has all of cargo bench's fields and some additional ones¹. We can then parse that "manually" (define the types and use serde) or using cargo_metadata like the cargo bench docs suggest to do whatever we want with the info in the future.

So an initial implementation would entail running benchmarks with CaptureStrategy::Split (or Combined?) and then parsing the output from those benchmarks. What we do with the benchmark data (in my opinion) could be resolved in a later PR. Is my AI-assisted understanding correct?

Footnotes

1. I haven't looked to closely, it might be that they have different schemas altogether, in which case we'd need to have separate parsing code for each harness? ↩