Skip to content

CI tests reorg & coverage drop#7897

Open
maxtropets wants to merge 2 commits into
microsoft:mainfrom
maxtropets:f/ci-op-run
Open

CI tests reorg & coverage drop#7897
maxtropets wants to merge 2 commits into
microsoft:mainfrom
maxtropets:f/ci-op-run

Conversation

@maxtropets
Copy link
Copy Markdown
Collaborator

@maxtropets maxtropets commented May 16, 2026

This drops 10 minutes in all run concurrently by

  • dropping coverage (will make another job for it and run in a bencher-manner)
  • test reshuffling
Today VMSS-C runs a coverage build that re-does what VMSS-A (unit) and
VMSS-B (e2e) already do, and ends up being the long pole at ~30m wall-
clock. This PR drops the coverage duplicate and splits the e2e suite
across the three VMSS runners so they finish in roughly equal time.

Reshuffle (timings from latest successful main run, 2026-05-14):

  runner   before                          after                          time before -> after
  -------  ------------------------------  -----------------------------  --------------------
  VMSS-A   checks + clang-tidy build +     unchanged                      20.8m -> 20.8m
           python pkg + doc + unit +
           partitions
  VMSS-B   debug build + full e2e          debug build + e2e HEAVY        29.1m -> 19.0m
           (-LE benchmark|suite|unit)      (schema_test, recovery_test,
                                           nodes_test)
  VMSS-C   coverage build + unit + e2e +   debug build + e2e REST         30.3m -> 16.4m
           coverage report                 (-LE benchmark|suite|unit
                                            minus the 3 heavy ones)
  -------  ------------------------------  -----------------------------  --------------------
  wall                                                                    30.3m -> 20.7m

  ~32% faster wall-clock; ~28 min of runner-time saved per CI run.
  Floor is the 10.4m clang-tidy build on VMSS-A.

Test coverage verified with `ctest -N`:
  old B+C       = 18 e2e tests
  new B (3) + new C (15) = 18 e2e tests, disjoint, identical union.
  No test gained, no test lost. Only thing removed is the coverage report
  upload, which was the stated goal.

Heavy bucket chosen as schema_test (6.9m), recovery_test (2.7m),
nodes_test (2.4m) - together ~56% of the e2e suite, the minimum split
that brings B and C within a couple of minutes of each other while
keeping the YAML to a single regex per runner.

@maxtropets maxtropets self-assigned this May 16, 2026
@maxtropets maxtropets changed the title WIP WIP - CI reorg May 16, 2026
@maxtropets maxtropets changed the title WIP - CI reorg CI tests reorg & coverage drop May 16, 2026
@maxtropets maxtropets marked this pull request as ready for review May 16, 2026 12:19
@maxtropets maxtropets requested a review from a team as a code owner May 16, 2026 12:19
Copilot AI review requested due to automatic review settings May 16, 2026 12:19
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR restructures CI by removing the duplicate coverage build (VMSS-C) and rebalancing end-to-end test execution across VMSS-B (heavy bucket: schema_test, recovery_test, nodes_test) and VMSS-C (remaining e2e tests excluding benchmark/suite/unit/partitions). Wall-clock improves from ~30m to ~21m without losing any test coverage; coverage reporting is planned for a separate dedicated job.

Changes:

  • Replace VMSS-C coverage build with a plain Debug build that runs the "rest" bucket of e2e tests.
  • Split full e2e run on VMSS-B into a small "heavy" bucket via ctest -R.
  • Drop unit-test rerun, coverage script invocation, and coverage HTML upload artifact.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants