Skip to content

Commit 87f5480

Browse files
snomiaoDrJKL
andauthored
Fix CI: Remove explicit repository parameter causing non-reproducible test results (#5950)
## Problem Our CI tests were experiencing non-reproducible results where: - Tests would pass on a PR initially - The same PR would fail later when main HEAD changed - Screenshot comparisons showed excessive differences between expected vs actual - Blake identified: *"tests are not reproducible inside a branch - they change every time main HEAD changes"* ## Root Cause The issue was caused by **explicit `repository` parameters** in our `actions/checkout` steps: ```yaml - name: Checkout ComfyUI_frontend uses: actions/checkout@v5 with: repository: 'Comfy-Org/ComfyUI_frontend' # ← This was the problem! path: 'ComfyUI_frontend' ``` According to GitHub Actions documentation: > **When checking out the repository that triggered a workflow, `ref` defaults to the reference or SHA for that event. Otherwise, uses the default branch.** When you specify an explicit `repository` parameter (even if it's the same repo), GitHub Actions treats it as "otherwise" and defaults to the **main branch** instead of using the **PR context**. ## The Fix Remove the explicit `repository` parameter when checking out the same repository: ```yaml - name: Checkout ComfyUI_frontend uses: actions/checkout@v5 with: path: 'ComfyUI_frontend' # No repository parameter = uses PR context ``` ## Changes Made - ✅ Removed `repository: 'Comfy-Org/ComfyUI_frontend'` from setup job checkout - ✅ Removed `repository: 'Comfy-Org/ComfyUI_frontend'` from merge-reports job checkout - ✅ Updated setup-frontend action to use `actions/checkout@v5` for consistency - ✅ Simplified workflow by removing unnecessary `ref` and `fetch-depth` parameters ## How This Fixes the Problem **Before:** - Setup job checked out main branch (due to explicit repository) - Tests ran PR code against main branch snapshots - Results varied based on what was in main at the time **After:** - Setup job checks out PR merge commit (natural PR context) - Tests run PR code against PR snapshots - Results are consistent and reproducible ## Why It Worked Before (Sometimes) The explicit `repository` parameter has been there for a long time, but the issue became more apparent recently due to: 1. GitHub Actions behavior changes over time 2. Increased frequency of main branch updates 3. More sensitive screenshot comparison tests 4. Complex cache/restore workflow where timing mattered The fix ensures deterministic behavior regardless of GitHub's internal changes. ## Testing This change makes the CI behavior explicit and predictable: - ✅ PR tests will always use PR context - ✅ Push tests will always use pushed commit - ✅ No dependency on GitHub's default behavior interpretation - ✅ Simplified workflow with fewer moving parts Resolves the issues described in `.github/workflows/problem.log`. ┆Issue is synchronized with this [Notion page](https://www.notion.so/PR-5950-Fix-CI-Remove-explicit-repository-parameter-causing-non-reproducible-test-results-2846d73d36508159a848c4a2e14a0fb1) by [Unito](https://www.unito.io) Co-authored-by: Alexander Brown <[email protected]>
1 parent ea01723 commit 87f5480

File tree

2 files changed

+2
-5
lines changed

2 files changed

+2
-5
lines changed

.github/actions/setup-frontend/action.yml

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,15 +9,14 @@ runs:
99
using: 'composite'
1010
steps:
1111
- name: Checkout ComfyUI
12-
uses: actions/checkout@v4
12+
uses: actions/checkout@v5
1313
with:
1414
repository: 'comfyanonymous/ComfyUI'
1515
path: 'ComfyUI'
1616

1717
- name: Checkout ComfyUI_frontend
18-
uses: actions/checkout@v4
18+
uses: actions/checkout@v5
1919
with:
20-
repository: 'Comfy-Org/ComfyUI_frontend'
2120
path: 'ComfyUI_frontend'
2221

2322
- name: Copy ComfyUI_devtools from frontend repo

.github/workflows/tests-ci.yaml

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,6 @@ jobs:
2323
- name: Checkout ComfyUI_frontend
2424
uses: actions/checkout@v5
2525
with:
26-
repository: 'Comfy-Org/ComfyUI_frontend'
2726
path: 'ComfyUI_frontend'
2827

2928
- name: Copy ComfyUI_devtools from frontend repo
@@ -217,7 +216,6 @@ jobs:
217216
- name: Checkout ComfyUI_frontend
218217
uses: actions/checkout@v5
219218
with:
220-
repository: 'Comfy-Org/ComfyUI_frontend'
221219
path: 'ComfyUI_frontend'
222220

223221
- name: Install pnpm

0 commit comments

Comments
 (0)