Fix CI: Remove explicit repository parameter causing non-reproducible test results

[snomiao]

Problem

Our CI tests were experiencing non-reproducible results where:

• Tests would pass on a PR initially
• The same PR would fail later when main HEAD changed
• Screenshot comparisons showed excessive differences between expected vs actual
• Blake identified: "tests are not reproducible inside a branch - they change every time main HEAD changes"

Root Cause

The issue was caused by explicit repository parameters in our actions/checkout steps:

- name: Checkout ComfyUI_frontend
  uses: actions/checkout@v5
  with:
    repository: 'Comfy-Org/ComfyUI_frontend'  # ← This was the problem!
    path: 'ComfyUI_frontend'

According to GitHub Actions documentation:

When checking out the repository that triggered a workflow, ref defaults to the reference or SHA for that event. Otherwise, uses the default branch.

When you specify an explicit repository parameter (even if it's the same repo), GitHub Actions treats it as "otherwise" and defaults to the main branch instead of using the PR context.

The Fix

Remove the explicit repository parameter when checking out the same repository:

- name: Checkout ComfyUI_frontend
  uses: actions/checkout@v5
  with:
    path: 'ComfyUI_frontend'  # No repository parameter = uses PR context

Changes Made

• ✅ Removed repository: 'Comfy-Org/ComfyUI_frontend' from setup job checkout
• ✅ Removed repository: 'Comfy-Org/ComfyUI_frontend' from merge-reports job checkout
• ✅ Updated setup-frontend action to use actions/checkout@v5 for consistency
• ✅ Simplified workflow by removing unnecessary ref and fetch-depth parameters

How This Fixes the Problem

Before:

• Setup job checked out main branch (due to explicit repository)
• Tests ran PR code against main branch snapshots
• Results varied based on what was in main at the time

After:

• Setup job checks out PR merge commit (natural PR context)
• Tests run PR code against PR snapshots
• Results are consistent and reproducible

Why It Worked Before (Sometimes)

The explicit repository parameter has been there for a long time, but the issue became more apparent recently due to:

1. GitHub Actions behavior changes over time
2. Increased frequency of main branch updates
3. More sensitive screenshot comparison tests
4. Complex cache/restore workflow where timing mattered

The fix ensures deterministic behavior regardless of GitHub's internal changes.

Testing

This change makes the CI behavior explicit and predictable:

• ✅ PR tests will always use PR context
• ✅ Push tests will always use pushed commit
• ✅ No dependency on GitHub's default behavior interpretation
• ✅ Simplified workflow with fewer moving parts

Resolves the issues described in .github/workflows/problem.log.

┆Issue is synchronized with this Notion page by Unito

[github-actions]

🎨 Storybook Build Status

✅ Build completed successfully!

⏰ Completed at: 10/07/2025, 07:55:14 PM UTC

🔗 Links

• 📊 View Workflow Run
• 🎨 View Storybook

---

🎉 Your Storybook is ready for review!

[github-actions]

🎭 Playwright Test Results

⚠️ Tests passed with flaky tests

⏰ Completed at: 10/07/2025, 08:03:52 PM UTC

📈 Summary

• Total Tests: 489
• Passed: 457 ✅
• Failed: 0
• Flaky: 2 ⚠️
• Skipped: 30 ⏭️

📊 Test Reports by Browser

• ✅ chromium: View Report • ✅ 448 / ❌ 0 / ⚠️ 2 / ⏭️ 30
• ✅ chromium-2x: View Report • ✅ 2 / ❌ 0 / ⚠️ 0 / ⏭️ 0
• ✅ chromium-0.5x: View Report • ✅ 1 / ❌ 0 / ⚠️ 0 / ⏭️ 0
• ✅ mobile-chrome: View Report • ✅ 6 / ❌ 0 / ⚠️ 0 / ⏭️ 0

---

🎉 Click on the links above to view detailed test results for each browser configuration.