Skip to content

feat: Support test sharding for CI parallelization with test_optimizer #1538

@TonyDowney

Description

@TonyDowney

Problem

very_good test --optimization consolidates all tests into a single .test_optimizer.dart file, which is great for reducing startup overhead. However, this single-file approach prevents CI-level sharding — splitting the test suite across multiple parallel CI runners.

For large Flutter projects, the test phase is often the CI bottleneck. In our case, PR checks take 13-23 minutes with the test phase accounting for 8-15 minutes, even with -j 4 concurrency within a single runner. Our NFR target is under 10 minutes for the full PR pipeline.

Proposal

Add --shard and --total-shards parameters (or similar) to very_good test --optimization that would:

  1. Partition the consolidated test imports across N shards deterministically (e.g., by hash or alphabetical split)
  2. Generate shard-specific optimizer files (e.g., .test_optimizer_1_of_3.dart) each containing a subset of test imports
  3. Allow CI workflows to use a strategy.matrix to run shards in parallel:
strategy:
  matrix:
    shard: [1, 2, 3]
steps:
  - run: very_good test --optimization --shard ${{ matrix.shard }} --total-shards 3

Alternatives Considered

  • -j N concurrency: Already in use, but limited to single-runner parallelism. Diminishing returns beyond core count.
  • --exclude-tags: Helps reduce scope but doesn't address the fundamental single-runner bottleneck.
  • Manual test file splitting: Defeats the purpose of test_optimizer's consolidation.
  • flutter test --shard-index: Flutter's native sharding works on individual test files, but conflicts with test_optimizer's single-file approach.

Context

This would complement the existing --optimization flag by extending its benefits to multi-runner CI setups. The test_optimizer already knows the full list of test files — partitioning them into N balanced groups seems like a natural extension.

Would also pair well with very_good_workflows if the reusable workflow added a shards input parameter.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    Status

    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions