fix: fix perf CI pipeline — z-score baselines, force-push staleness, baseline storage#9886
Merged
christian-byrne merged 3 commits intomainfrom Mar 15, 2026
Merged
Conversation
…baseline storage - Fix z-score '0/5 runs': add perf-data branch storage + historical baseline loading in pr-perf-report.yaml (data was being downloaded in wrong workflow) - Fix force-push comment staleness: change cancel-in-progress to false so last push always completes, add SHA validation to skip posting stale results - Add perf-data orphan branch mechanism: on push to main, save perf-metrics.json to rolling 20-baseline window in perf-data branch - Load last 5 baselines into temp/perf-history/ for z-score computation - Add actions: read permission for baseline loading
🎨 Storybook: ✅ Built — View Storybook |
🎭 Playwright: ✅ 551 passed, 0 failed · 9 flaky📊 Browser Reports
|
Contributor
📝 WalkthroughWalkthroughSaves CI perf baselines to an orphan Changes
Sequence Diagram(s)sequenceDiagram
participant CI as Workflow Runner
participant GH as GitHub API
participant Git as Git Remote (origin / perf-data)
participant AR as Artifact Store
CI->>GH: Trigger on push to main -> perf job completes
CI->>Git: Create/fetch orphan `perf-data` branch (worktree)
CI->>Git: Write timestamped SHA baselines, prune to last 20
CI->>Git: Commit & push `perf-data` using GH_TOKEN
CI->>GH: On PR workflow start -> query PR head SHA
GH-->>CI: Return PR head SHA
CI->>CI: Compare run head_sha vs PR head SHA -> set `stale`
alt not stale
CI->>Git: Fetch `perf-data`, load up to 5 historical baselines
CI->>AR: Download perf artifacts
CI->>CI: Generate report using baselines
CI->>GH: Post perf report/comment to PR
else stale
CI->>CI: Skip artifact download / report generation
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Comment |
📦 Bundle: 4.99 MB gzip 🔴 +28 BDetailsSummary
Category Glance App Entry Points — 21.7 kB (baseline 21.7 kB) • ⚪ 0 BMain entry bundles and manifests
Status: 1 added / 1 removed Graph Workspace — 1.08 MB (baseline 1.08 MB) • ⚪ 0 BGraph editor runtime, canvas, workflow orchestration
Status: 1 added / 1 removed Views & Navigation — 75.3 kB (baseline 75.3 kB) • ⚪ 0 BTop-level views, pages, and routed surfaces
Status: 9 added / 9 removed Panels & Settings — 460 kB (baseline 460 kB) • ⚪ 0 BConfiguration panels, inspectors, and settings screens
Status: 10 added / 10 removed User & Accounts — 16.6 kB (baseline 16.6 kB) • ⚪ 0 BAuthentication, profile, and account management bundles
Status: 5 added / 5 removed Editors & Dialogs — 81.8 kB (baseline 81.8 kB) • ⚪ 0 BModals, dialogs, drawers, and in-app editors
Status: 2 added / 2 removed UI Components — 59 kB (baseline 59 kB) • ⚪ 0 BReusable component library chunks
Status: 5 added / 5 removed Data & Services — 3.17 MB (baseline 3.17 MB) • ⚪ 0 BStores, services, APIs, and repositories
Status: 15 added / 15 removed Utilities & Hooks — 68.9 kB (baseline 68.9 kB) • ⚪ 0 BHelpers, composables, and utility bundles
Status: 12 added / 12 removed Vendor & Third-Party — 9.78 MB (baseline 9.78 MB) • ⚪ 0 BExternal libraries and shared vendor chunks
Other — 8.21 MB (baseline 8.21 MB) • ⚪ 0 BBundles that do not match a named category
Status: 52 added / 52 removed |
⚡ Performance Report
Raw data{
"timestamp": "2026-03-15T04:55:12.630Z",
"gitSha": "760cfdde3603c031d3435bf385dbb9748eeb91d8",
"branch": "fix/perf-ci-pipeline-fixes",
"measurements": [
{
"name": "canvas-idle",
"durationMs": 2051.9949999999767,
"styleRecalcs": 10,
"styleRecalcDurationMs": 9.845999999999997,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 416.74999999999994,
"heapDeltaBytes": 823648
},
{
"name": "canvas-idle",
"durationMs": 2047.6230000000442,
"styleRecalcs": 10,
"styleRecalcDurationMs": 9.229000000000001,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 397.444,
"heapDeltaBytes": -5736564
},
{
"name": "canvas-idle",
"durationMs": 2025.4999999999654,
"styleRecalcs": 12,
"styleRecalcDurationMs": 11.450999999999999,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 394.57599999999996,
"heapDeltaBytes": 756028
},
{
"name": "canvas-mouse-sweep",
"durationMs": 1866.517999999985,
"styleRecalcs": 78,
"styleRecalcDurationMs": 47.568,
"layouts": 12,
"layoutDurationMs": 3.7039999999999997,
"taskDurationMs": 832.4090000000001,
"heapDeltaBytes": 17464444
},
{
"name": "canvas-mouse-sweep",
"durationMs": 1799.4939999999815,
"styleRecalcs": 74,
"styleRecalcDurationMs": 39.716,
"layouts": 12,
"layoutDurationMs": 3.6290000000000004,
"taskDurationMs": 773.514,
"heapDeltaBytes": -3384044
},
{
"name": "canvas-mouse-sweep",
"durationMs": 2036.371000000031,
"styleRecalcs": 84,
"styleRecalcDurationMs": 44.678000000000004,
"layouts": 12,
"layoutDurationMs": 3.3420000000000005,
"taskDurationMs": 988.0439999999999,
"heapDeltaBytes": 17804868
},
{
"name": "dom-widget-clipping",
"durationMs": 600.3379999999652,
"styleRecalcs": 13,
"styleRecalcDurationMs": 9.687999999999999,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 360.81,
"heapDeltaBytes": 12852280
},
{
"name": "dom-widget-clipping",
"durationMs": 566.080999999997,
"styleRecalcs": 14,
"styleRecalcDurationMs": 10.399999999999999,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 350.955,
"heapDeltaBytes": 13328840
},
{
"name": "dom-widget-clipping",
"durationMs": 616.2689999999884,
"styleRecalcs": 16,
"styleRecalcDurationMs": 15.236,
"layouts": 1,
"layoutDurationMs": 0.347,
"taskDurationMs": 375.474,
"heapDeltaBytes": 13616432
},
{
"name": "subgraph-dom-widget-clipping",
"durationMs": 590.6990000000292,
"styleRecalcs": 49,
"styleRecalcDurationMs": 13.494000000000003,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 375.342,
"heapDeltaBytes": 12656784
},
{
"name": "subgraph-dom-widget-clipping",
"durationMs": 565.3169999999932,
"styleRecalcs": 49,
"styleRecalcDurationMs": 12.645999999999999,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 357.23900000000003,
"heapDeltaBytes": 12608512
},
{
"name": "subgraph-dom-widget-clipping",
"durationMs": 589.6450000000186,
"styleRecalcs": 48,
"styleRecalcDurationMs": 12.353000000000002,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 372.77,
"heapDeltaBytes": 12741712
},
{
"name": "subgraph-idle",
"durationMs": 2004.3419999999514,
"styleRecalcs": 13,
"styleRecalcDurationMs": 13.097000000000001,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 399.10400000000004,
"heapDeltaBytes": 575092
},
{
"name": "subgraph-idle",
"durationMs": 2054.2580000000044,
"styleRecalcs": 12,
"styleRecalcDurationMs": 12.424,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 392.84799999999996,
"heapDeltaBytes": 303532
},
{
"name": "subgraph-idle",
"durationMs": 2021.9079999999963,
"styleRecalcs": 12,
"styleRecalcDurationMs": 12.054999999999998,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 389.361,
"heapDeltaBytes": 652488
},
{
"name": "subgraph-mouse-sweep",
"durationMs": 1991.6279999999915,
"styleRecalcs": 88,
"styleRecalcDurationMs": 50.339999999999996,
"layouts": 16,
"layoutDurationMs": 5.156,
"taskDurationMs": 948.6199999999999,
"heapDeltaBytes": -7671800
},
{
"name": "subgraph-mouse-sweep",
"durationMs": 2005.8669999999665,
"styleRecalcs": 88,
"styleRecalcDurationMs": 49.663,
"layouts": 16,
"layoutDurationMs": 4.469,
"taskDurationMs": 945.5179999999999,
"heapDeltaBytes": -6936084
},
{
"name": "subgraph-mouse-sweep",
"durationMs": 1985.5019999999968,
"styleRecalcs": 86,
"styleRecalcDurationMs": 46.745,
"layouts": 16,
"layoutDurationMs": 4.361999999999999,
"taskDurationMs": 925.749,
"heapDeltaBytes": -7658876
}
]
} |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/ci-perf-report.yaml:
- Around line 82-93: The workflow creates an orphan perf-data branch and pushes
it (git checkout --orphan perf-data / git push origin perf-data) but does not
fetch the new remote ref before using it, so the subsequent git worktree add
/tmp/perf-data origin/perf-data fails on first-run; fix by adding a git fetch
origin perf-data (or a full git fetch) immediately after the git push origin
perf-data so the origin/perf-data remote-tracking ref exists locally before
running git worktree add.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 5c81435f-4644-49aa-9f43-a576b9a39663
📒 Files selected for processing (2)
.github/workflows/ci-perf-report.yaml.github/workflows/pr-perf-report.yaml
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
.github/workflows/ci-perf-report.yaml (1)
73-111: Solid implementation of the baseline storage pattern.The worktree-based approach correctly isolates operations on the
perf-databranch without disrupting the main checkout. Key observations:
- ✅ Past review feedback addressed:
git fetch origin perf-data(line 91) now follows the initial push- ✅ Graceful degradation with
continue-on-error: true- ✅ Pruning logic correctly retains latest 20 baselines
- ✅ Smart to copy metrics to
/tmpbefore branch operations (line 80)Minor: The
GH_TOKENenv var (line 111) appears unused sincegit pushuses the authentication configured byactions/checkout. It's harmless but could be removed for clarity.,
♻️ Optional: Remove unused env var
git worktree remove /tmp/perf-data --force 2>/dev/null || true - env: - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/ci-perf-report.yaml around lines 73 - 111, The GH_TOKEN environment variable is defined but never used in the "Save perf baseline to perf-data branch" step; remove the unused GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} entry from the step's env block to avoid confusion and keep the workflow minimal (look for the step named "Save perf baseline to perf-data branch" and the GH_TOKEN env variable).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In @.github/workflows/ci-perf-report.yaml:
- Around line 73-111: The GH_TOKEN environment variable is defined but never
used in the "Save perf baseline to perf-data branch" step; remove the unused
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} entry from the step's env block to avoid
confusion and keep the workflow minimal (look for the step named "Save perf
baseline to perf-data branch" and the GH_TOKEN env variable).
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: d1e4d062-b924-4e46-b638-50eb3d03b784
📒 Files selected for processing (1)
.github/workflows/ci-perf-report.yaml
dosubot
bot
added
the
size:M
| password: ${{ secrets.GITHUB_TOKEN }} | ||
| permissions: | ||
| contents: read | ||
| contents: write |
Member
There was a problem hiding this comment.
do we need write here?. This job checks out PR-controlled code and then runs repo-controlled actions/scripts from that checkout, and actions/checkout persists authenticated git credentials by default.
Contributor
Author
There was a problem hiding this comment.
good catch - fixed in c634daa. added persist-credentials: false to the checkout step so PR-triggered runs never have write-capable git credentials available to PR-controlled code. contents: write is still needed at the job level because the baseline push step (push-to-main only) does git push origin HEAD:perf-data, but credentials are now set up explicitly only in that step via git config url."https://x-access-token:...".insteadOf, so write access is scoped to exactly where it's needed.
Add persist-credentials: false to actions/checkout so PR-triggered runs never have write-capable git credentials available to PR-controlled code. Credentials are set up explicitly only in the baseline push step (which only runs on push to main). Addresses review feedback: #9886 (comment)
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
.github/workflows/ci-perf-report.yaml (1)
99-102: Baseline filename approach is clear and functional.The timestamped naming with truncated SHA provides good traceability.
Minor observation: bash parameter expansion (
SHA="${GITHUB_SHA:0:8}") could replace theecho | cutpipeline, but the current approach is readable and works correctly.♻️ Optional: Use parameter expansion
TIMESTAMP=$(date -u +%Y%m%dT%H%M%SZ) -SHA=$(echo "${{ github.sha }}" | cut -c1-8) +SHA="${GITHUB_SHA:0:8}"Note:
GITHUB_SHAis automatically available as an environment variable in GitHub Actions.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/ci-perf-report.yaml around lines 99 - 102, Replace the pipeline that computes SHA with bash parameter expansion to simplify and avoid spawning a subshell: change the SHA assignment that currently uses SHA=$(echo "${{ github.sha }}" | cut -c1-8) to use the GITHUB_SHA environment variable and parameter expansion (refer to the TIMESTAMP and SHA variables in the workflow snippet) so SHA is set from GITHUB_SHA truncated to 8 characters; keep the rest of the timestamped filename logic (mkdir and cp to /tmp/perf-data/baselines/perf-${TIMESTAMP}-${SHA}.json) unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In @.github/workflows/ci-perf-report.yaml:
- Around line 99-102: Replace the pipeline that computes SHA with bash parameter
expansion to simplify and avoid spawning a subshell: change the SHA assignment
that currently uses SHA=$(echo "${{ github.sha }}" | cut -c1-8) to use the
GITHUB_SHA environment variable and parameter expansion (refer to the TIMESTAMP
and SHA variables in the workflow snippet) so SHA is set from GITHUB_SHA
truncated to 8 characters; keep the rest of the timestamped filename logic
(mkdir and cp to /tmp/perf-data/baselines/perf-${TIMESTAMP}-${SHA}.json)
unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: e7a86775-3cc0-432c-bc0a-215ce8cf96dd
📒 Files selected for processing (1)
.github/workflows/ci-perf-report.yaml
christian-byrne
added a commit
that referenced
this pull request
Mar 15, 2026
## Summary Adds Total Blocking Time (TBT) and frame duration metrics to the performance testing infrastructure, plus three new test scenarios covering zoom, pan, and many-nodes-idle. ## Changes ### New Metrics - **`totalBlockingTimeMs`** — Computed from PerformanceObserver `longtask` entries: `sum(duration - 50ms)` for tasks >50ms. Measures main thread blocking. - **`frameDurationMs`** — Average frame duration via rAF timing (16.67ms = 60fps target). Measures rendering smoothness. ### New Test Scenarios | Scenario | Description | |---|---| | `canvas-zoom-sweep` | 10 zoom-in + 10 zoom-out cycles on default workflow | | `canvas-pan-many-nodes` | 10 pan sweeps over 100-node workflow | | `canvas-many-nodes-idle` | 2-second idle measurement with 100 nodes rendered | ### Infrastructure - `PerformanceHelper.ts`: Installs PerformanceObserver for longtask, collects TBT, measures frame duration via rAF - `perf-report.ts`: Reports TBT and frame duration in PR comment tables - `browser_tests/assets/perf/many_nodes_100.json`: 100-node (10×10 grid) test fixture ## Review Focus - TBT collection clears entries at `startMeasuring()` and reads at `stopMeasuring()` — ensure no race with observer buffering - Frame duration sampling uses 10 frames — enough for signal without slowing tests Depends on: #9886, #9887 ┆Issue is synchronized with this [Notion page](https://www.notion.so/PR-9910-feat-add-TBT-frameDuration-metrics-and-new-perf-test-scenarios-3236d73d365081488ae3c594a8bf7cff) by [Unito](https://www.unito.io)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes three critical issues with the CI performance reporting pipeline that made perf reports useless on PRs (demonstrated by PR #9248 — deep watcher removal merged without useful perf signal).
Changes
1. Fix z-score baseline variance collection (
0/5 runs)Root cause: PR #9305 added z-score statistical analysis code to
perf-report.ts, but the historical data download step was placed in the wrong workflow file. The report is generated inpr-perf-report.yaml(aworkflow_run-triggered job), but the historical download was inci-perf-report.yaml(the test runner) — different runners, different filesystems.Fix: Implement
perf-dataorphan branch storage:perf-metrics.jsontoperf-databranch with timestamped filenameperf-databranch intotemp/perf-history/github-action-benchmark(33.7k repos)2. Fix force-push comment staleness
Root cause:
cancel-in-progress: truekills the perf test run before it uploads artifacts. The downstream report workflow only triggers onconclusion == 'success'— cancelled runs are ignored, so the comment from the first successful run goes stale.Fix:
cancel-in-progress: false— with GitHub's queue depth of 1, rapid pushes (A,B,C,D) run A and D, skipping B and Cpr-perf-report.yaml— before posting, check if the workflow_run's head SHA still matches the PR's current head. Skip posting stale results.3. Add permissions for baseline operations
contents: writeon CI job (needed for pushing to perf-data branch)actions: readon both workflows (needed for artifact/baseline access)One-time setup required
After merging, create the
perf-dataorphan branch:The first 2 pushes to main after setup will build up variance data, and z-scores will start appearing in PR reports (threshold is
historical.length >= 2).Testing
yaml.safe_load()perf-report.tsloadHistoricalReports()already reads fromtemp/perf-history/<index>/perf-metrics.json— no code changes neededcontinue-on-error: truefor graceful degradation┆Issue is synchronized with this Notion page by Unito