Redesign the Ractor/Knucleotide benchmark #451
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
eightbitraptor
force-pushed
the
mvh-knuecleotide-refactor
branch
from
de6ca3f to
b4f216d
Compare
The previous ractor knucleotide benchmark ran the whole benchmark in parallel inside multiple ractors. This is a good test that the benchmark can be parallelised but isn't directly comparable to the original Knucleotide implementation which uses processes to partition the work done by the benchmark itself. This implementation uses ractors to parallelise the work in the same way that the original uses Process.fork so it's more directly comparable. This needs to be run with the regular benchmark harness instead of the ractor harness, otherwise the ractor harness will attempt to wrap this benchmark run in multiple ractors too.
I don't think there is value in keeping this around now that we have a Ractor enabled version that has feature parity with the old Process based one
For benchmarks like the knucleotide ractor benchmark, which use Ractors internally so need to be included in the ractor benchmarks, but which must not use the ractor harness New category selection is as follows: - No category: Runs all benchmarks except those with `ractor_only: true`. Uses each benchmark's `default_harness`, falling back to the `default` harness. - `--category=ractor`: Runs benchmarks with ractor: true or `ractor_only: true`. Uses each benchmark's `default_harness`, falling back to `harness-ractor`. - `--category=ractor-only`: Runs only benchmarks with `ractor_only: true`. Uses each benchmark's `default_harness`, falling back to `harness-ractor`.
eightbitraptor
force-pushed
the
mvh-knuecleotide-refactor
branch
from
b4f216d to
06d34e7
Compare
The API has changed between 3.4.7 and HEAD. Until 4.0 is realeased we shouldn't test 3.4.7
eightbitraptor
force-pushed
the
mvh-knuecleotide-refactor
branch
from
5fd05ad to
dc13f62
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR rebuilds the knucleotide Ractor benchmark to use Ractors internally, rather than doing the internal work sequentially and running the whole workload in multiple Ractors using the Ractor harness.
This makes it directly comparable to the process-based implementation, so we can measure the forking approach directly against the Ractor approach, instead of measuring ractor overhead on a single threaded workload, which we have other benchmarks to do.
The old
benchmarks-ractor/knucleotidehas been removed.To support this properly, we've rethought how benchmark categories work:
All benchmarks have been consolidated back into the single
benchmarksdirectory andbenchmarks-ractorhas been removed.Benchmarks that were previously in the
benchmarks-ractordirectory no haveractor_only: trueinbenchmarks.ymland theractor_onlycategory will run only these benchmarks.Additionally, the new key
default_harnesshas been introduced intobenchmarks.yml. This lets individual benchmarks specify their own harness, independent of category. This has been done to facilitate running benchmarks like the newknucleotide-ractorintroduced in this commit - which needs to run when--category=ractoris specified, but should use thedefaultharness, unlike otherractorcategory benchmarks, which generally require the ractor harness.The category filtering now works as follows:
ractor_only: true. Uses each benchmark'sdefault_harness, falling back to thedefaultharness.--category=ractor: Runs benchmarks with ractor: true orractor_only: true. Uses each benchmark'sdefault_harness, falling back toharness-ractor.--category=ractor-only: Runs only benchmarks withractor_only: true. Uses each benchmark'sdefault_harness, falling back toharness-ractor.