perf: Improve comprehensive benchmark to be less finicky #51
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've noticed that the current implementation, while a commendable goal, was too delicate in the input size.
For example, the example of a bad PR ran the "bad" version of the code for 6h and then got timed out by GitHub.
Even playing locally with the max operations constant shows me that sometimes a 20% change in the number is the difference between faster than the other benchmark to "infinite" (infinite defined as more time than I had patience to wait).
This is an attempt to make it more linearly dependent on the max size, as well as calibrating the constants, so the total runtime is more manageable.
I will test this by rebasing the "bad PR example" once merged.
That all being said I am very open to suggestions on other alternatives for good benchmarks, this was my idea to try to simulate real life behaviours with concurrent modifications and such, but it has become slightly Frankenstein's monster-esque.