Generated benchmark threshold files may not be Windows compatible

[slashmo]

Context

We received an issue in Swift Distributed Tracing (apple/swift-distributed-tracing#165) about not being able to check out the library on Windows. Upon inspection, we found the issue to be the file names of some of our Benchmark thresholds. In our case, the Benchmark names contained a : which isn't valid in file paths on Windows.

Solution

For now, we'll simply fix this problem by replacing the colons in our Benchmark names. I was wondering though if longterm this could be prevented by package-benchmark in the first place to avoid folks from running into similar issues. One option that came to mind is to hash the Benchmark name and use that hash as file name. This would come at the cost of not being able to easily navigate around threshold files but I'm not sure if that's a valid concern. What do you think?

[hassila]

Yeah, this is related to #193 really - we'd definitely want to do that, but perhaps what you suggest with using a hash of the file name would be ok in practice (modulo hash collisions, but one could easily construct collisions with any name mangling algorithm too) - just a simple check that there are no collisions and otherwise warn would be more than ok in practice I think.

[slashmo]

just a simple check that there are no collisions and otherwise warn would be more than ok in practice I think.

Yep, that would work too.