Skip to content

perf: speedup ci by splitting tests into parallel default/all-features jobs - #456

Open
samlaf wants to merge 1 commit into
mainfrom
sl/speedup-ci-test-split
Open

perf: speedup ci by splitting tests into parallel default/all-features jobs#456
samlaf wants to merge 1 commit into
mainfrom
sl/speedup-ci-test-split

Conversation

@samlaf

@samlaf samlaf commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

The test job ran cargo test and cargo test --all-features back to back: 7.8m + 7.7m, about 16m of wall clock. A two-entry matrix runs them side by side and cuts that to about 8m. Each entry gets its own rust-cache shared-key, so the two jobs no longer overwrite each other's cache. fail-fast: false keeps a failure in one config from cancelling the other.

Timings from run 32185252971:

  cargo test                 7.8m   (2m34s compile, ~5m10s tests)
  cargo test --all-features  7.7m   (1m52s compile, ~5m50s tests)

Compilation is only about 30% of the cost. Almost all the rest is one binary: the summit lib test target (94 tests in 307s, 96 tests in 287s). Every other crate finishes in under a second, except summit-syncer under all-features (27 tests, 56s).

Both configs earn their place. --all-features runs a strict superset of the default test names (613 -> 644, and no test is default-only). But three features change behavior in place instead of adding tests:

  • bench swaps start_building_block for a variant that takes a height and drops the state root (application/src/actor.rs)
  • bad-blocks swaps in BadBlockEngineClient (node/src/args.rs)
  • permissioned changes the RPC auth path (rpc/src/server.rs)

bench is on under --all-features, so that run never exercises the production block-building path. The default run is what covers it.

Possible Follow-ups (not done here)

  • --all-features turns on bench and bad-blocks together. No deployment runs that combination. A named feature set, such as --features prom,permissioned,e2e, would test a real configuration instead of a frankenconfig.
  • To get below ~8m, shard the summit lib binary with cargo nextest run --partition count:N/M over a matrix. The slowest tests are multi-node integration tests. Local timings: 207s for test_checkpoint_verification_fixed_committee, 175s for test_process_time_invalid_new_validator_refund_does_not_merge_with_reused_pubkey_withdrawal, 158s for test_deposit_request_top_up.

…s jobs

The test job ran `cargo test` and `cargo test --all-features` back to back:
7.8m + 7.7m, about 16m of wall clock. A two-entry matrix runs them side by
side and cuts that to about 8m. Each entry gets its own rust-cache shared-key,
so the two jobs no longer overwrite each other's cache. `fail-fast: false`
keeps a failure in one config from cancelling the other.

Timings from run 32185252971:

  cargo test                 7.8m   (2m34s compile, ~5m10s tests)
  cargo test --all-features  7.7m   (1m52s compile, ~5m50s tests)

Compilation is only about 30% of the cost. Almost all the rest is one binary:
the `summit` lib test target (94 tests in 307s, 96 tests in 287s). Every other
crate finishes in under a second, except `summit-syncer` under all-features
(27 tests, 56s).

Both configs earn their place. `--all-features` runs a strict superset of the
default test names (613 -> 644, and no test is default-only). But three
features change behavior in place instead of adding tests:

  - `bench` swaps `start_building_block` for a variant that takes a height and
    drops the state root (application/src/actor.rs)
  - `bad-blocks` swaps in `BadBlockEngineClient` (node/src/args.rs)
  - `permissioned` changes the RPC auth path (rpc/src/server.rs)

`bench` is on under `--all-features`, so that run never exercises the
production block-building path. The default run is what covers it.

This also refreshes the stale test counts in CLAUDE.md. The documented 153/170
is now 613/644, with a per-crate breakdown for both configs.

Follow-ups, not done here:

  - `--all-features` turns on `bench` and `bad-blocks` together. No deployment
    runs that combination. A named feature set, such as
    `--features prom,permissioned,e2e`, would test a real configuration
    instead of a frankenconfig.
  - To get below ~8m, shard the `summit` lib binary with
    `cargo nextest run --partition count:N/M` over a matrix. The slowest tests
    are multi-node integration tests. Local timings: 207s for
    test_checkpoint_verification_fixed_committee, 175s for
    test_process_time_invalid_new_validator_refund_does_not_merge_with_reused_pubkey_withdrawal,
    158s for test_deposit_request_top_up.
@samlaf

samlaf commented Aug 18, 2026

Copy link
Copy Markdown
Contributor Author

TODO: in order to merge will have to replace the now outdated required CI check "test" and make the 2 new ones requires instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant