Skip to content

Bug 2036968: Replaced fast-kde with fftkde and used bootstrap-ci to get CI summary #1034

Open
kala-moz wants to merge 8 commits into
mozilla:mainfrom
kala-moz:js-integration-analysis
Open

Bug 2036968: Replaced fast-kde with fftkde and used bootstrap-ci to get CI summary #1034
kala-moz wants to merge 8 commits into
mozilla:mainfrom
kala-moz:js-integration-analysis

Conversation

@kala-moz
Copy link
Copy Markdown
Contributor

@kala-moz kala-moz commented May 5, 2026

This PR (Bug 2036968) integrates the js version of @padenot's standalone-stats into perfcompare. Here are the main changes for the integration:

Deploy Link

1.) Added standalone-stats/kde.js and kde.d.ts; bootstrap-ci.js and bootstrap-ci.ts to src/utils
2.) In src/components/CompareResults/CommonGraph.tsx - Removed the the fast-kde import and replaced with the fftkde. I removed approximateSJBandwidth because the ISJ method below auto-selects the bandwidth per dataset. It also has an internal grid and quantile logic so didn't need the quantileSorted method.

  const bKde =
    baseValues.length >= 2 ? fftkde(baseValues, 'ISJ', undefined, 1024) : null;
  const nKde =
    newValues.length >= 2 ? fftkde(newValues, 'ISJ', undefined, 1024) : null;
  const baseRunsDensity = bKde
    ? bKde.x.map((x, i) => ({ x, y: bKde.y[i] }))
    : [];
  const newRunsDensity = nKde
    ? nKde.x.map((x, i) => ({ x, y: nKde.y[i] }))
    : [];

3.) Use the bootstrapMedianDiffCI from bootstrap-ci to get the confidence interval summary and confidence interval statement.

   const baseRuns = mwResult.base_runs ?? [];
    const newRuns = mwResult.new_runs ?? [];
    const ci =
      baseRuns.length > 0 && newRuns.length > 0
        ? bootstrapMedianDiffCI(baseRuns, newRuns)
        : null;
    const fmt = (n: number) => (n >= 0 ? '+' : '') + n.toFixed(1);
    const summary = ci ? (
      <span>
        <strong>Δ median</strong> = {fmt(ci.medianDiff)} ms 95% CI [
        {fmt(ci.ciLow)}, {fmt(ci.ciHigh)}]
        {ci.significant ? '' : '  (not significant)'}
      </span>
    ) : null;
    const confidenceInterval = ci && (
      <span>
        <strong>Confidence Interval</strong>: We are 95% confident the median
        difference is between <strong>{fmt(ci.ciLow)}</strong> and{' '}
        <strong>{fmt(ci.ciHigh)}</strong>
      </span>
    );

Note: please check that the comments are relevant as I may have missed to update or add them where necessary.
Still need to add coverage.

Screenshot 2026-05-07 at 4 55 36 PM

Updates:

  • Hook up CommonGraph to echarts, replacing chartjs

@netlify
Copy link
Copy Markdown

netlify Bot commented May 5, 2026

Deploy Preview for mozilla-perfcompare ready!

Name Link
🔨 Latest commit 0b2fb6a
🔍 Latest deploy log https://app.netlify.com/projects/mozilla-perfcompare/deploys/6a0625ccf86db600089346b7
😎 Deploy Preview https://deploy-preview-1034--mozilla-perfcompare.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 5, 2026

Codecov Report

❌ Patch coverage is 95.65217% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.37%. Comparing base (ec289b8) to head (0b2fb6a).

Files with missing lines Patch % Lines
src/components/CompareResults/CommonGraph.tsx 94.23% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1034      +/-   ##
==========================================
+ Coverage   96.32%   96.37%   +0.05%     
==========================================
  Files         113      113              
  Lines        3262     3254       -8     
  Branches      739      741       +2     
==========================================
- Hits         3142     3136       -6     
+ Misses        118      116       -2     
  Partials        2        2              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@kala-moz kala-moz changed the title Integrate js stats analysis Bug 2036968: Integrate js stats analysis May 5, 2026
@kala-moz kala-moz requested a review from padenot May 6, 2026 20:30
@padenot
Copy link
Copy Markdown
Collaborator

padenot commented May 7, 2026

Good start! As discussed, let's switch to echarts for feature, performance and accessibility reasons. I've updated the examples in the other repo so that it is even more drop in, and added a "gallery" with multiple examples. The closest this PR looks like the examples the best it is.

In particular, it's going to be a lot less visually noisy, with lots of useless numbers removed. This is a complaint I hear from perf people. lmk if we should do a quick 1:1 one off or pair to get this done!

@kala-moz kala-moz changed the title Bug 2036968: Integrate js stats analysis Bug 2036968: Replaced fast-kde with fftkde and used bootstrap-ci to get CI summary May 7, 2026
@kala-moz kala-moz marked this pull request as draft May 7, 2026 16:22
@kala-moz
Copy link
Copy Markdown
Contributor Author

kala-moz commented May 8, 2026

Good start! As discussed, let's switch to echarts for feature, performance and accessibility reasons. I've updated the examples in the other repo so that it is even more drop in, and added a "gallery" with multiple examples. The closest this PR looks like the examples the best it is.

In particular, it's going to be a lot less visually noisy, with lots of useless numbers removed. This is a complaint I hear from perf people. lmk if we should do a quick 1:1 one off or pair to get this done!

Thanks, @padenot. I went ahead and switched to echarts. Should I continue with the rest of the updates or should we break them into separate PRs? I feel this PR is pretty full already.

@kala-moz kala-moz marked this pull request as ready for review May 8, 2026 00:45
@kala-moz kala-moz requested review from canova and esanuandra as code owners May 8, 2026 00:45
Copy link
Copy Markdown
Collaborator

@padenot padenot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, those are only some changes to do now, and we'll do the rest in a second PR. Looks nice, most comments are cosmetics or just comments (not linked to any action).

Comment thread src/components/CompareResults/CommonGraph.tsx
Comment thread src/components/CompareResults/CommonGraph.tsx Outdated
Comment thread src/components/CompareResults/CommonGraph.tsx Outdated
Comment thread src/components/CompareResults/CommonGraph.tsx Outdated
Comment thread src/components/CompareResults/CommonGraph.tsx Outdated
],
tooltip: {
trigger: 'axis',
axisPointer: { type: 'cross', crossStyle: { color: '#999' } },
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's nice but I'm having issues with having the tooltip lock onto a data series, it shows the base or new but not both. Can we make it show both and lock onto the data series? E.g. a single tooltip always when overing over the KDE part, that shows base and new value at this position in x ?

@kala-moz
Copy link
Copy Markdown
Contributor Author

kala-moz commented May 13, 2026

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants