perf: speedup ci by splitting tests into parallel default/all-features jobs - #456
Open
samlaf wants to merge 1 commit into
Open
perf: speedup ci by splitting tests into parallel default/all-features jobs#456samlaf wants to merge 1 commit into
samlaf wants to merge 1 commit into
Conversation
…s jobs
The test job ran `cargo test` and `cargo test --all-features` back to back:
7.8m + 7.7m, about 16m of wall clock. A two-entry matrix runs them side by
side and cuts that to about 8m. Each entry gets its own rust-cache shared-key,
so the two jobs no longer overwrite each other's cache. `fail-fast: false`
keeps a failure in one config from cancelling the other.
Timings from run 32185252971:
cargo test 7.8m (2m34s compile, ~5m10s tests)
cargo test --all-features 7.7m (1m52s compile, ~5m50s tests)
Compilation is only about 30% of the cost. Almost all the rest is one binary:
the `summit` lib test target (94 tests in 307s, 96 tests in 287s). Every other
crate finishes in under a second, except `summit-syncer` under all-features
(27 tests, 56s).
Both configs earn their place. `--all-features` runs a strict superset of the
default test names (613 -> 644, and no test is default-only). But three
features change behavior in place instead of adding tests:
- `bench` swaps `start_building_block` for a variant that takes a height and
drops the state root (application/src/actor.rs)
- `bad-blocks` swaps in `BadBlockEngineClient` (node/src/args.rs)
- `permissioned` changes the RPC auth path (rpc/src/server.rs)
`bench` is on under `--all-features`, so that run never exercises the
production block-building path. The default run is what covers it.
This also refreshes the stale test counts in CLAUDE.md. The documented 153/170
is now 613/644, with a per-crate breakdown for both configs.
Follow-ups, not done here:
- `--all-features` turns on `bench` and `bad-blocks` together. No deployment
runs that combination. A named feature set, such as
`--features prom,permissioned,e2e`, would test a real configuration
instead of a frankenconfig.
- To get below ~8m, shard the `summit` lib binary with
`cargo nextest run --partition count:N/M` over a matrix. The slowest tests
are multi-node integration tests. Local timings: 207s for
test_checkpoint_verification_fixed_committee, 175s for
test_process_time_invalid_new_validator_refund_does_not_merge_with_reused_pubkey_withdrawal,
158s for test_deposit_request_top_up.
Contributor
Author
|
TODO: in order to merge will have to replace the now outdated required CI check "test" and make the 2 new ones requires instead. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The test job ran
cargo testandcargo test --all-featuresback to back: 7.8m + 7.7m, about 16m of wall clock. A two-entry matrix runs them side by side and cuts that to about 8m. Each entry gets its own rust-cache shared-key, so the two jobs no longer overwrite each other's cache.fail-fast: falsekeeps a failure in one config from cancelling the other.Timings from run 32185252971:
Compilation is only about 30% of the cost. Almost all the rest is one binary: the
summitlib test target (94 tests in 307s, 96 tests in 287s). Every other crate finishes in under a second, exceptsummit-syncerunder all-features (27 tests, 56s).Both configs earn their place.
--all-featuresruns a strict superset of the default test names (613 -> 644, and no test is default-only). But three features change behavior in place instead of adding tests:benchswapsstart_building_blockfor a variant that takes a height and drops the state root (application/src/actor.rs)bad-blocksswaps inBadBlockEngineClient(node/src/args.rs)permissionedchanges the RPC auth path (rpc/src/server.rs)benchis on under--all-features, so that run never exercises the production block-building path. The default run is what covers it.Possible Follow-ups (not done here)
--all-featuresturns onbenchandbad-blockstogether. No deployment runs that combination. A named feature set, such as--features prom,permissioned,e2e, would test a real configuration instead of a frankenconfig.summitlib binary withcargo nextest run --partition count:N/Mover a matrix. The slowest tests are multi-node integration tests. Local timings: 207s for test_checkpoint_verification_fixed_committee, 175s for test_process_time_invalid_new_validator_refund_does_not_merge_with_reused_pubkey_withdrawal, 158s for test_deposit_request_top_up.