|
| 1 | +--- |
| 2 | +name: merge-train-infra |
| 3 | +description: Reference for merge-train automation internals -- workflows, scripts, CI integration, and configuration. Use when modifying or debugging merge-train infrastructure. |
| 4 | +--- |
| 5 | + |
| 6 | +# Merge-Train Infrastructure |
| 7 | + |
| 8 | +This skill covers the automation internals of the merge-train system. For contributor-facing guidance (creating PRs, labels, handling failures), see the `merge-trains` skill. |
| 9 | + |
| 10 | +## Automation Lifecycle |
| 11 | + |
| 12 | +The merge-train system is fully automated via GitHub Actions in `.github/workflows/merge-train-*.yml`: |
| 13 | + |
| 14 | +1. **PR Creation** (`merge-train-create-pr.yml`): Triggered on push to `merge-train/*` branches. Creates a PR targeting `next` with the `ci-no-squash` label (and `ci-full-no-test-cache` for spartan). Skips merge commits and commits already in `next`. |
| 15 | + |
| 16 | +2. **Body Updates** (`merge-train-update-pr-body.yml`): Triggered on push to `merge-train/**`. Updates the PR body with meaningful commits (those containing PR references like `(#1234)`). The body uses `BEGIN_COMMIT_OVERRIDE` / `END_COMMIT_OVERRIDE` markers for release-please. |
| 17 | + |
| 18 | +3. **Next Integration** (`merge-train-next-to-branches.yml`): Triggered on push to `next`. Merges `next` into each active merge-train branch via `scripts/merge-train/merge-next.sh`. Uses `continue-on-error: true` so a conflict in one branch does not block others. Skips branches whose PR already has auto-merge enabled. |
| 19 | + |
| 20 | +4. **Auto-Merge** (`merge-train-auto-merge.yml`): Runs hourly via cron (`0 * * * *`). Calls `scripts/merge-train/auto-merge.sh` for both merge-train (4-hour inactivity) and backport-train (8-hour inactivity) branches. Uses separate GitHub tokens: `AZTEC_BOT_GITHUB_TOKEN` for API calls and `MERGE_TRAIN_GITHUB_TOKEN` for approvals. Will not auto-merge if the last merge-queue CI run failed or was cancelled. |
| 21 | + |
| 22 | +5. **Recreation & Wakeup** (`merge-train-recreate.yml`): Triggered when a PR is closed (merged). If the merged PR's head branch starts with `merge-train/`, recreates the branch from the base branch (usually `next`). Then runs `scripts/merge-train/wakeup-prs.sh` to add the `ci-wakeup-pr-after-merge` label to all open PRs targeting the branch that have passed CI and have automerge enabled. This triggers a CI re-run (typically a no-op via tree-hash cache) so those PRs can proceed through the merge queue. The label is immediately removed by a step in `ci3.yml` so it can be re-applied on subsequent merges. |
| 23 | + |
| 24 | +6. **Failure Notification** (`merge-queue-dequeue-notify.yml`): Triggered when a PR is dequeued from the merge queue. If the PR's head branch starts with `merge-train/` and the PR was NOT merged, sends a Slack notification via `ci3/merge_train_failure_slack_notify`. |
| 25 | + |
| 26 | +## CI Integration Details |
| 27 | + |
| 28 | +### CI Mode Selection (`.github/ci3_labels_to_env.sh`) |
| 29 | + |
| 30 | +Merge-train branches influence CI mode: |
| 31 | +- `merge_group` events or `ci-merge-queue` label → `merge-queue` mode |
| 32 | +- If the merge-group event is for `merge-train/spartan` → upgraded to `merge-queue-heavy` mode (10 parallel grind runs instead of 4) |
| 33 | +- Target branch `merge-train/docs` → `ci-docs` mode |
| 34 | +- Target branch `merge-train/barretenberg` → `ci-barretenberg` mode |
| 35 | + |
| 36 | +### CI Concurrency (`.github/workflows/ci3.yml`) |
| 37 | + |
| 38 | +```yaml |
| 39 | +group: ci3-${{ (startsWith(github.event.pull_request.head.ref, 'merge-train/') && github.run_id) || ... }} |
| 40 | +``` |
| 41 | +
|
| 42 | +Merge-train PRs get **full concurrency** (each run has its own unique group via `github.run_id`), while non-merge-train PRs share a group by branch name with cancel-in-progress. |
| 43 | + |
| 44 | +### Instance Postfix (`.github/ci3.sh`) |
| 45 | + |
| 46 | +```bash |
| 47 | +if [[ "${PR_HEAD_REF:-}" == merge-train/* ]]; then |
| 48 | + export INSTANCE_POSTFIX=${PR_COMMITS:-} |
| 49 | +fi |
| 50 | +``` |
| 51 | + |
| 52 | +Merge-train PRs get a unique instance postfix (commit count) to allow parallel EC2 instances. |
| 53 | + |
| 54 | +### CI Modes in bootstrap.sh |
| 55 | + |
| 56 | +- `ci-docs`: Only builds and tests documentation |
| 57 | +- `ci-barretenberg`: Only builds and tests barretenberg (AVM disabled) |
| 58 | +- `ci-barretenberg-full`: Full barretenberg CI including acir_tests |
| 59 | +- `merge-queue`: 4x AMD64 full + 1x ARM64 fast in parallel |
| 60 | +- `merge-queue-heavy`: 10x AMD64 full + 1x ARM64 fast in parallel (used for `merge-train/spartan`) |
| 61 | + |
| 62 | +### Test History Tracking (`ci3/run_test_cmd`) |
| 63 | + |
| 64 | +```bash |
| 65 | +if [[ "$is_merge_queue" -eq 1 || ("${TARGET_BRANCH:-}" =~ ^v[0-9]) || ("${TARGET_BRANCH:-}" == merge-train/*) ]]; then |
| 66 | + track_test_history=1 |
| 67 | +fi |
| 68 | +``` |
| 69 | + |
| 70 | +### Failure Notification (`ci3/bootstrap_ec2`) |
| 71 | + |
| 72 | +When a CI run fails on an EC2 instance, it calls `merge_train_failure_slack_notify` to send failure notifications to the appropriate Slack channel based on the branch name. |
| 73 | + |
| 74 | +## Creating a New Merge Train |
| 75 | + |
| 76 | +1. Create a branch from `next` with naming pattern `merge-train/{team}` |
| 77 | +2. Add the branch to the matrix in `.github/workflows/merge-train-next-to-branches.yml` |
| 78 | +3. Add the branch-to-Slack-channel mapping in `ci3/merge_train_failure_slack_notify` |
| 79 | +4. Optionally add CI mode overrides in `.github/ci3_labels_to_env.sh` and `bootstrap.sh` |
| 80 | +5. Push code to the branch -- automation handles PR creation from there |
| 81 | + |
| 82 | +## Key Files Reference |
| 83 | + |
| 84 | +### Workflows |
| 85 | + |
| 86 | +| File | Purpose | |
| 87 | +|---|---| |
| 88 | +| `.github/workflows/merge-train-readme.md` | User-facing documentation | |
| 89 | +| `.github/workflows/merge-train-create-pr.yml` | Auto-creates PRs for train branches | |
| 90 | +| `.github/workflows/merge-train-auto-merge.yml` | Hourly cron to auto-merge inactive trains | |
| 91 | +| `.github/workflows/merge-train-next-to-branches.yml` | Syncs `next` into all train branches; defines active branches | |
| 92 | +| `.github/workflows/merge-train-recreate.yml` | Recreates branch after merge | |
| 93 | +| `.github/workflows/merge-train-update-pr-body.yml` | Updates PR body with commit list | |
| 94 | +| `.github/workflows/merge-queue-dequeue-notify.yml` | Slack notification on merge-queue dequeue | |
| 95 | +| `.github/workflows/squashed-pr-check.yml` | Squash enforcement (skipped for `ci-no-squash`) | |
| 96 | + |
| 97 | +### Scripts |
| 98 | + |
| 99 | +| File | Purpose | |
| 100 | +|---|---| |
| 101 | +| `scripts/merge-train/auto-merge.sh` | Auto-merge logic -- checks inactivity, last CI status, approves and merges | |
| 102 | +| `scripts/merge-train/merge-next.sh` | Merges `next` into a train branch, handles conflicts, cancels stale CI runs | |
| 103 | +| `scripts/merge-train/update-pr-body.sh` | Updates PR body with meaningful commits | |
| 104 | +| `scripts/merge-train/squash-pr.sh` | Squashes PR commits (used by `ci-squash-and-merge` label) | |
| 105 | +| `scripts/merge-train/wakeup-prs.sh` | Adds `ci-wakeup-pr-after-merge` label to qualifying PRs after branch recreation | |
| 106 | + |
| 107 | +### CI Configuration |
| 108 | + |
| 109 | +| File | Purpose | |
| 110 | +|---|---| |
| 111 | +| `.github/ci3_labels_to_env.sh` | CI mode selection based on labels and target branches | |
| 112 | +| `.github/ci3.sh` | Instance postfix for merge-train parallelism | |
| 113 | +| `ci3/merge_train_failure_slack_notify` | Slack failure notification with branch-to-channel mapping | |
| 114 | +| `ci3/run_test_cmd` | Test history tracking for merge-train branches | |
| 115 | +| `ci3/bootstrap_ec2` | EC2 failure notification trigger | |
| 116 | +| `bootstrap.sh` | CI mode definitions (`ci-docs`, `ci-barretenberg`, etc.) | |
| 117 | + |
| 118 | +### Other Scripts |
| 119 | + |
| 120 | +| File | Purpose | |
| 121 | +|---|---| |
| 122 | +| `scripts/auto_close_issues.py` | Auto-closes issues referenced in merged merge-train PRs (GitHub's native auto-close doesn't work for intermediate branches) | |
| 123 | +| `scripts/find_orphaned_issues_in_prs.py` | Finds PRs in merge-train commits that reference still-open issues | |
| 124 | +| `scripts/dedupe_release_notes.py` | Deduplicates release notes from merge-train merges | |
| 125 | +| `scripts/commits` | Pretty git log that groups merge-train children by subsystem | |
| 126 | +| `scripts/filter_history` | Filters git history, identifying merge-train merge commits as "containers" | |
0 commit comments