Input on a json schema for bench reporting

[epage]

In a theoretical future where test/bench binaries are encouraged to report back results to cargo as json which then formats and reports them, what would callper see as needed for the json schema?

I've been considering how we can improve testing in Rust and one area I've found is the limited contract between cargo and test harnesses is a source of friction. I'm also hoping that this lens for looking at json reporting can help get the schema stabilized.

For reference, this is the current schema for benches.

[osiewicz]

Hi,
Thank you for reaching out.
I'd consider the following items to be important for calliper's use case:

1. It should be possible to report multiple metrics per benchmark run; calliper (and I guess iai too to an extent) reports multiple metrics (total memory reads, cache hit ratio, instruction counts etc.) that are not necessarily of the same unit.
2. Contrary to benchmarking based on repeated measurements, we are benchmarking in a controlled environment. Therefore it does not make sense for us to report minimum/maximum values out of the benchmark harness.
3. Since we can provide variety of results for a single benchmark run, it would be great to have good support for "custom" units. At the very least, I think that having support for float/int values would be the minimum; it should cover cases where one would like to report a ratio (e.g. cache hit ratio, that might be a float), time (not relevant for calliper) or scalars like instructions count.

I think that what you're going for is great; I will spend some more time on coming up with use cases for calliper and provide additional feedback if it's welcome. =)

[epage]

Thanks for that input!

And yes, more feedback is welcome!