Skip to content

Commit f793699

Browse files
ludamadludamad
authored
feat: merge-train/spartan (#20472)
2 parents f125ea6 + dd5024f commit f793699

File tree

9 files changed

+297
-42
lines changed

9 files changed

+297
-42
lines changed

‎.claude/skills/merge-train-infra/SKILL.md‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
---
2+
name: merge-train-infra
3+
description: Reference for merge-train automation internals -- workflows, scripts, CI integration, and configuration. Use when modifying or debugging merge-train infrastructure.
4+
---
5+
6+
# Merge-Train Infrastructure
7+
8+
This skill covers the automation internals of the merge-train system. For contributor-facing guidance (creating PRs, labels, handling failures), see the `merge-trains` skill.
9+
10+
## Automation Lifecycle
11+
12+
The merge-train system is fully automated via GitHub Actions in `.github/workflows/merge-train-*.yml`:
13+
14+
1. **PR Creation** (`merge-train-create-pr.yml`): Triggered on push to `merge-train/*` branches. Creates a PR targeting `next` with the `ci-no-squash` label (and `ci-full-no-test-cache` for spartan). Skips merge commits and commits already in `next`.
15+
16+
2. **Body Updates** (`merge-train-update-pr-body.yml`): Triggered on push to `merge-train/**`. Updates the PR body with meaningful commits (those containing PR references like `(#1234)`). The body uses `BEGIN_COMMIT_OVERRIDE` / `END_COMMIT_OVERRIDE` markers for release-please.
17+
18+
3. **Next Integration** (`merge-train-next-to-branches.yml`): Triggered on push to `next`. Merges `next` into each active merge-train branch via `scripts/merge-train/merge-next.sh`. Uses `continue-on-error: true` so a conflict in one branch does not block others. Skips branches whose PR already has auto-merge enabled.
19+
20+
4. **Auto-Merge** (`merge-train-auto-merge.yml`): Runs hourly via cron (`0 * * * *`). Calls `scripts/merge-train/auto-merge.sh` for both merge-train (4-hour inactivity) and backport-train (8-hour inactivity) branches. Uses separate GitHub tokens: `AZTEC_BOT_GITHUB_TOKEN` for API calls and `MERGE_TRAIN_GITHUB_TOKEN` for approvals. Will not auto-merge if the last merge-queue CI run failed or was cancelled.
21+
22+
5. **Recreation & Wakeup** (`merge-train-recreate.yml`): Triggered when a PR is closed (merged). If the merged PR's head branch starts with `merge-train/`, recreates the branch from the base branch (usually `next`). Then runs `scripts/merge-train/wakeup-prs.sh` to add the `ci-wakeup-pr-after-merge` label to all open PRs targeting the branch that have passed CI and have automerge enabled. This triggers a CI re-run (typically a no-op via tree-hash cache) so those PRs can proceed through the merge queue. The label is immediately removed by a step in `ci3.yml` so it can be re-applied on subsequent merges.
23+
24+
6. **Failure Notification** (`merge-queue-dequeue-notify.yml`): Triggered when a PR is dequeued from the merge queue. If the PR's head branch starts with `merge-train/` and the PR was NOT merged, sends a Slack notification via `ci3/merge_train_failure_slack_notify`.
25+
26+
## CI Integration Details
27+
28+
### CI Mode Selection (`.github/ci3_labels_to_env.sh`)
29+
30+
Merge-train branches influence CI mode:
31+
- `merge_group` events or `ci-merge-queue` label → `merge-queue` mode
32+
- If the merge-group event is for `merge-train/spartan` → upgraded to `merge-queue-heavy` mode (10 parallel grind runs instead of 4)
33+
- Target branch `merge-train/docs` → `ci-docs` mode
34+
- Target branch `merge-train/barretenberg` → `ci-barretenberg` mode
35+
36+
### CI Concurrency (`.github/workflows/ci3.yml`)
37+
38+
```yaml
39+
group: ci3-${{ (startsWith(github.event.pull_request.head.ref, 'merge-train/') && github.run_id) || ... }}
40+
```
41+
42+
Merge-train PRs get **full concurrency** (each run has its own unique group via `github.run_id`), while non-merge-train PRs share a group by branch name with cancel-in-progress.
43+
44+
### Instance Postfix (`.github/ci3.sh`)
45+
46+
```bash
47+
if [[ "${PR_HEAD_REF:-}" == merge-train/* ]]; then
48+
export INSTANCE_POSTFIX=${PR_COMMITS:-}
49+
fi
50+
```
51+
52+
Merge-train PRs get a unique instance postfix (commit count) to allow parallel EC2 instances.
53+
54+
### CI Modes in bootstrap.sh
55+
56+
- `ci-docs`: Only builds and tests documentation
57+
- `ci-barretenberg`: Only builds and tests barretenberg (AVM disabled)
58+
- `ci-barretenberg-full`: Full barretenberg CI including acir_tests
59+
- `merge-queue`: 4x AMD64 full + 1x ARM64 fast in parallel
60+
- `merge-queue-heavy`: 10x AMD64 full + 1x ARM64 fast in parallel (used for `merge-train/spartan`)
61+
62+
### Test History Tracking (`ci3/run_test_cmd`)
63+
64+
```bash
65+
if [[ "$is_merge_queue" -eq 1 || ("${TARGET_BRANCH:-}" =~ ^v[0-9]) || ("${TARGET_BRANCH:-}" == merge-train/*) ]]; then
66+
track_test_history=1
67+
fi
68+
```
69+
70+
### Failure Notification (`ci3/bootstrap_ec2`)
71+
72+
When a CI run fails on an EC2 instance, it calls `merge_train_failure_slack_notify` to send failure notifications to the appropriate Slack channel based on the branch name.
73+
74+
## Creating a New Merge Train
75+
76+
1. Create a branch from `next` with naming pattern `merge-train/{team}`
77+
2. Add the branch to the matrix in `.github/workflows/merge-train-next-to-branches.yml`
78+
3. Add the branch-to-Slack-channel mapping in `ci3/merge_train_failure_slack_notify`
79+
4. Optionally add CI mode overrides in `.github/ci3_labels_to_env.sh` and `bootstrap.sh`
80+
5. Push code to the branch -- automation handles PR creation from there
81+
82+
## Key Files Reference
83+
84+
### Workflows
85+
86+
| File | Purpose |
87+
|---|---|
88+
| `.github/workflows/merge-train-readme.md` | User-facing documentation |
89+
| `.github/workflows/merge-train-create-pr.yml` | Auto-creates PRs for train branches |
90+
| `.github/workflows/merge-train-auto-merge.yml` | Hourly cron to auto-merge inactive trains |
91+
| `.github/workflows/merge-train-next-to-branches.yml` | Syncs `next` into all train branches; defines active branches |
92+
| `.github/workflows/merge-train-recreate.yml` | Recreates branch after merge |
93+
| `.github/workflows/merge-train-update-pr-body.yml` | Updates PR body with commit list |
94+
| `.github/workflows/merge-queue-dequeue-notify.yml` | Slack notification on merge-queue dequeue |
95+
| `.github/workflows/squashed-pr-check.yml` | Squash enforcement (skipped for `ci-no-squash`) |
96+
97+
### Scripts
98+
99+
| File | Purpose |
100+
|---|---|
101+
| `scripts/merge-train/auto-merge.sh` | Auto-merge logic -- checks inactivity, last CI status, approves and merges |
102+
| `scripts/merge-train/merge-next.sh` | Merges `next` into a train branch, handles conflicts, cancels stale CI runs |
103+
| `scripts/merge-train/update-pr-body.sh` | Updates PR body with meaningful commits |
104+
| `scripts/merge-train/squash-pr.sh` | Squashes PR commits (used by `ci-squash-and-merge` label) |
105+
| `scripts/merge-train/wakeup-prs.sh` | Adds `ci-wakeup-pr-after-merge` label to qualifying PRs after branch recreation |
106+
107+
### CI Configuration
108+
109+
| File | Purpose |
110+
|---|---|
111+
| `.github/ci3_labels_to_env.sh` | CI mode selection based on labels and target branches |
112+
| `.github/ci3.sh` | Instance postfix for merge-train parallelism |
113+
| `ci3/merge_train_failure_slack_notify` | Slack failure notification with branch-to-channel mapping |
114+
| `ci3/run_test_cmd` | Test history tracking for merge-train branches |
115+
| `ci3/bootstrap_ec2` | EC2 failure notification trigger |
116+
| `bootstrap.sh` | CI mode definitions (`ci-docs`, `ci-barretenberg`, etc.) |
117+
118+
### Other Scripts
119+
120+
| File | Purpose |
121+
|---|---|
122+
| `scripts/auto_close_issues.py` | Auto-closes issues referenced in merged merge-train PRs (GitHub's native auto-close doesn't work for intermediate branches) |
123+
| `scripts/find_orphaned_issues_in_prs.py` | Finds PRs in merge-train commits that reference still-open issues |
124+
| `scripts/dedupe_release_notes.py` | Deduplicates release notes from merge-train merges |
125+
| `scripts/commits` | Pretty git log that groups merge-train children by subsystem |
126+
| `scripts/filter_history` | Filters git history, identifying merge-train merge commits as "containers" |

‎.claude/skills/merge-trains/SKILL.md‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
---
2+
name: merge-trains
3+
description: Guide for working with merge-train branches -- creating PRs, choosing the right base branch, understanding labels, handling failures, and bypassing checks.
4+
---
5+
6+
# Working with Merge Trains
7+
8+
## What Is a Merge Train?
9+
10+
A merge train is an automated batching system (inspired by [Rust rollups](https://forge.rust-lang.org/release/rollups.html)) that groups multiple PRs together for coordinated integration into the `next` branch. Instead of each PR going through the merge queue individually, teams push their PRs into a shared `merge-train/*` branch. Periodically, that branch is merged as a single unit into `next`.
11+
12+
## Active Merge-Train Branches
13+
14+
| Branch | Team / Domain | Slack Channel |
15+
|---|---|---|
16+
| `merge-train/avm` | AVM, barretenberg vm2 folder | `#team-bonobos` |
17+
| `merge-train/barretenberg` | Barretenberg folder, but not vm2 folder | `#honk-team` |
18+
| `merge-train/ci` | CI infrastructure / ci3 | `#help-ci` |
19+
| `merge-train/docs` | Documentation | `#dev-rels` |
20+
| `merge-train/fairies` | aztec-nr | `#team-fairies` |
21+
| `merge-train/spartan` | Spartan / infra / yarn-project sequencer and prover orchestration | `#team-alpha` |
22+
23+
## How to Use a Merge Train
24+
25+
### Targeting a Merge Train with Your PR
26+
27+
1. Create your feature branch **off the appropriate merge-train branch** (not our default branch `next`).
28+
2. Open your PR targeting that merge-train branch (e.g., base: `merge-train/barretenberg`).
29+
3. When your PR is approved and merged, it gets squashed into the merge-train branch.
30+
4. The merge-train PR (which targets `next`) automatically accumulates your commit.
31+
32+
### Key Rules for Contributors
33+
34+
- **Base branch matters**: Always branch from the branch specified in the CI_BASE_BRANCH environment variable. If it is not set, then ask the user their intent and offer to set CI_BASE_BRANCH in their shell's RC file.
35+
- **Your PR is squashed into the train**: Individual PRs targeting a merge-train branch are squash-merged as usual. You should not use the merge commit merge method, but the squash method.
36+
- **The train itself is NOT squashed**: The merge-train PR (e.g., `merge-train/barretenberg` -> `next`) is merged with a **merge commit**, preserving the individual squashed commits. This is why the `ci-no-squash` label is automatically applied.
37+
- **You generally don't need to worry about the train PR itself** -- it is fully automated (creation, body updates, approval, merge, and recreation). You only need to pay attention to it if an alert is sent to your team channel.
38+
39+
## CI Behavior for Merge Trains
40+
41+
- **Specialized CI modes**: PRs targeting `merge-train/docs` run docs-only CI. PRs targeting `merge-train/barretenberg` run barretenberg-only CI. This avoids running the full test suite for domain-specific changes.
42+
- **Merge-queue mode**: When the merge-train PR enters GitHub's merge queue, it runs the full `merge-queue` CI mode (4 parallel grind runs on AMD64 + 1 ARM64). `merge-train/spartan` uses the heavier `merge-queue-heavy` mode (10 grind runs).
43+
- **Full concurrency**: Merge-train PRs get unique CI concurrency groups (using `github.run_id`), so multiple CI runs can proceed in parallel without cancelling each other.
44+
- **Test history tracking**: Test results are tracked for merge-train PRs, same as merge-queue runs.
45+
46+
## Handling Merge-Train Failures
47+
48+
### When CI Fails on the Merge-Train PR
49+
50+
Two options from the [merge-train-readme.md](https://github.com/AztecProtocol/aztec-packages/blob/next/.github/workflows/merge-train-readme.md):
51+
52+
**Option 1: Direct Fix** -- Push a fix directly to the merge-train branch. Use bypass merge to expedite (all users have this permission). You can use the ci-skip label to no-op CI if really necessary.
53+
54+
**Option 2: Fix in Next** -- Merge a revert or workaround into `next`. The fix will auto-propagate to the merge-train via the `merge-train-next-to-branches` workflow. Best when the root cause is in `next` or multiple trains are affected.
55+
56+
### When Auto-Merge Is Blocked
57+
58+
The auto-merge script will **not** enable auto-merge if the last merge-queue CI run for the PR concluded with `failure` or `cancelled`. Someone needs to either fix the issue and push, or force-merge.
59+
60+
### Merge Conflicts from Next
61+
62+
When merging `next` into a train branch causes conflicts, the `merge-next.sh` script:
63+
- Aborts the merge
64+
- Posts a comment on the latest `next` commit listing the conflicted files
65+
- The team must manually resolve conflicts on their train branch
66+
67+
## Bypassing Checks / Force-Merging
68+
69+
If the user needs to bypass CI checks for their merge-train PR (e.g., a known flaky failure, an urgent merge, or CI infrastructure issues):
70+
71+
1. **Confirm intent**: Always confirm with the user that they want to skip CI, since this merges untested code into `next`.
72+
2. **Add the `ci-skip` label**: Apply the `ci-skip` label to the merge-train PR. This causes CI to skip entirely. Use: `gh pr edit <PR_NUMBER> --add-label ci-skip`
73+
3. **Force merge in the UI**: The user can then use GitHub's "Merge without waiting for requirements to be met" button (bypass merge) in the PR UI. All users have this permission.
74+
75+
**Important**: Only do this when the user explicitly asks to bypass checks. Always confirm first since it skips all CI validation.
76+
77+
## Backport Trains
78+
79+
A related system exists for backport branches (`backport-to-*`). These use the same auto-merge mechanism but with different settings:
80+
- Branch pattern: `backport-to-`
81+
- Inactivity threshold: 8 hours (instead of 4)
82+
- Merge strategy: merge commit

‎.github/workflows/ci3.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,12 @@ jobs:
4444
if: github.event.pull_request.head.repo.fork != true && (github.event.pull_request.draft == false || contains(github.event.pull_request.labels.*.name, 'ci-draft'))
4545
environment: ${{ startsWith(github.ref, 'refs/tags/v') && 'master' || '' }}
4646
steps:
47+
- name: Remove wakeup label
48+
if: contains(github.event.pull_request.labels.*.name, 'ci-wakeup-pr-after-merge')
49+
env:
50+
GH_TOKEN: ${{ secrets.AZTEC_BOT_GITHUB_TOKEN }}
51+
run: gh pr edit ${{ github.event.pull_request.number }} --remove-label ci-wakeup-pr-after-merge --repo ${{ github.repository }} || true
52+
4753
- name: Checkout
4854
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
4955
with:

‎.github/workflows/merge-train-next-to-branches.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ jobs:
1616
- merge-train/barretenberg
1717
- merge-train/ci
1818
- merge-train/docs
19+
- merge-train/fairies
1920
- merge-train/spartan
2021
steps:
2122
- name: Checkout

‎.github/workflows/merge-train-readme.md‎

Lines changed: 13 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -1,49 +1,20 @@
11
# Merge Train Documentation
22

3-
## Overview
3+
## Skills
44

5-
The merge train is an automated system for managing pull requests in a coordinated queue. It helps prevent merge conflicts and ensures smooth integration of multiple PRs targeting the same base branch.
6-
Inspired by [rust rollups](https://forge.rust-lang.org/release/rollups.html), but that's a non-ideal name for our domain.
5+
The merge-train system is documented via Claude Code skills in `.claude/skills/`. These are structured markdown files designed to be consumed by LLMs (such as Claude) to answer interactive questions about the system.
76

8-
## How It Works
7+
| Skill | Description |
8+
|---|---|
9+
| [`merge-trains`](../../.claude/skills/merge-trains/SKILL.md) | Contributor-facing guide -- creating PRs, choosing the right base branch, understanding labels, CI behavior, handling failures, and bypassing checks. |
10+
| [`merge-train-infra`](../../.claude/skills/merge-train-infra/SKILL.md) | Infrastructure reference -- workflows, scripts, CI integration, configuration, and how to create a new merge train. |
911

10-
1. **Merge Train Branches**: Special branches prefixed with `merge-train/` (e.g., `merge-train/barretenberg`, `merge-train/docs`)
11-
2. **Automatic Recreation**: When a merge train PR is merged, the branch is automatically recreated with an empty commit
12-
3. **PR Body Updates**: The merge train PR body is automatically updated with a list of commits whenever changes are pushed
13-
4. **Auto-merge**: PRs that have been inactive for 4+ hours are automatically approved and merged (if they contain meaningful commits)
14-
5. **Next Branch Integration**: Changes from the `next` branch are automatically merged into active merge train branches
12+
### Using the Skills
1513

16-
## Using the Merge Train
14+
Ask Claude Code questions like:
15+
- "How do I add my PR to a merge train?"
16+
- "What happens when a merge-train CI run fails?"
17+
- "How do I create a new merge-train branch?"
18+
- "How does the auto-merge system work?"
1719

18-
### Creating a New Merge-Train
19-
20-
1. **Create Branch**: Fork from `next` with naming pattern `merge-train/{team}`
21-
2. **Enable Auto-Sync**: Add your branch to `.github/workflows/merge-train-next-to-branches.yml`
22-
3. **Open PR**: Create a PR from your merge-train branch to `next` - automation handles the rest
23-
24-
### Adding Your PR to the Merge Train
25-
26-
1. Create your feature branch and make your changes
27-
2. Instead of targeting `master` or `next` directly, target the appropriate merge train branch (e.g., `merge-train/barretenberg`)
28-
3. When your PR merges the changes will be included in the next merge train cycle
29-
30-
### Merge Train Lifecycle
31-
32-
1. **Creation**: A merge train PR is created automatically when changes are pushed to the branch and labeled with `ci-no-squash`.
33-
2. **Accumulation**: Feature PRs are merged into the merge train branch, squashed
34-
3. **Auto-merge**: After 4 hours of inactivity (with meaningful commits), the train is automatically merged with a merge-commit (not a squash)
35-
4. **Recreation**: The cycle starts again with a new merge train
36-
37-
## Handling Merge Failures
38-
39-
When a merge-train fails due to issues from `next`:
40-
41-
### Option 1: Direct Fix
42-
- Push a fix directly to the merge-train branch
43-
- Use bypass merge to expedite (all users have this permission)
44-
- Best for small, quick fixes
45-
46-
### Option 2: Fix in Next
47-
- Merge a revert or workaround into `next`
48-
- The fix will auto-propagate to merge-train via automation
49-
- Best when key assumptions are broken or multiple trains affected
20+
Claude will load the relevant skill and provide contextual answers based on the actual automation code.

‎.github/workflows/merge-train-recreate.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,3 +36,10 @@ jobs:
3636
# This will generally only succeed if the branch does not exist, as a fallback if the branch is unprotected and gets deleted.
3737
git push origin "$MERGE_TRAIN_BRANCH"
3838
39+
- name: Wake up PRs targeting this branch
40+
env:
41+
GH_TOKEN: ${{ secrets.AZTEC_BOT_GITHUB_TOKEN }}
42+
run: |
43+
MERGE_TRAIN_BRANCH="${{ github.event.pull_request.head.ref }}"
44+
./scripts/merge-train/wakeup-prs.sh "$MERGE_TRAIN_BRANCH"
45+

‎ci3/merge_train_failure_slack_notify‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,8 @@ elif [[ "$REF_NAME" == "merge-train/ci" ]]; then
2727
channel="#help-ci"
2828
elif [[ "$REF_NAME" == "merge-train/docs" ]]; then
2929
channel="#dev-rels"
30+
elif [[ "$REF_NAME" == "merge-train/fairies" ]]; then
31+
channel="#team-fairies"
3032
elif [[ "$REF_NAME" == "merge-train/spartan" ]]; then
3133
channel="#team-alpha"
3234
else

0 commit comments

Comments
 (0)