-
Notifications
You must be signed in to change notification settings - Fork 237
feat: Support test sharding for CI parallelization with test_optimizer #1538
Description
Problem
very_good test --optimization consolidates all tests into a single .test_optimizer.dart file, which is great for reducing startup overhead. However, this single-file approach prevents CI-level sharding — splitting the test suite across multiple parallel CI runners.
For large Flutter projects, the test phase is often the CI bottleneck. In our case, PR checks take 13-23 minutes with the test phase accounting for 8-15 minutes, even with -j 4 concurrency within a single runner. Our NFR target is under 10 minutes for the full PR pipeline.
Proposal
Add --shard and --total-shards parameters (or similar) to very_good test --optimization that would:
- Partition the consolidated test imports across N shards deterministically (e.g., by hash or alphabetical split)
- Generate shard-specific optimizer files (e.g.,
.test_optimizer_1_of_3.dart) each containing a subset of test imports - Allow CI workflows to use a
strategy.matrixto run shards in parallel:
strategy:
matrix:
shard: [1, 2, 3]
steps:
- run: very_good test --optimization --shard ${{ matrix.shard }} --total-shards 3Alternatives Considered
-j Nconcurrency: Already in use, but limited to single-runner parallelism. Diminishing returns beyond core count.--exclude-tags: Helps reduce scope but doesn't address the fundamental single-runner bottleneck.- Manual test file splitting: Defeats the purpose of test_optimizer's consolidation.
flutter test --shard-index: Flutter's native sharding works on individual test files, but conflicts with test_optimizer's single-file approach.
Context
This would complement the existing --optimization flag by extending its benefits to multi-runner CI setups. The test_optimizer already knows the full list of test files — partitioning them into N balanced groups seems like a natural extension.
Would also pair well with very_good_workflows if the reusable workflow added a shards input parameter.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status