investigate what caused the latest regression via the bump to `proof-systems`

[Trivo25]

[bleepbloopsify]

timebox until Dec 12th, for handoff of the SPIKE so we don't spin wheels

[Shigoto-dev19]

The investigation into the performance regression caused by bumping proof-systems began in
PR #2686 — develop-3.0 performance regression tests.

This PR branched from develop-3.0 and was used solely to trigger CI and evaluate performance regression behaviour.

The key finding from the regression results was that CI consistently failed at the ECDSA verify step, as seen in: https://github.com/o1-labs/o1js/actions/runs/20134667611/job/57785071325?pr=2686

Across multiple CI runs, the regression appeared reliably at ~17–18%, which is above the allowed threshold.

To check whether the regression affected proof verification more broadly, the ECDSA example was temporarily ignored.

• All ZkProgram examples showed no regression in compile/prove/verify time.
• zkApps showed expected compile-time regression due to structural changes introduced by the Mesa Hardfork (details: Collect benchmarks for o1js-3.0 periodically until we release #2661 (comment))

These structural changes also affect the digest of zkApp methods.
A digest dump was added in commit 92f213f to explicitly separate protocol-level changes from regressions introduced by the proof-systems bump.

Thus, the only consistently failing case remained ECDSA verify, pointing toward a deeper issue.

@Trivo25 suggested bumping proof-systems directly on main to isolate the regression from protocol changes and opened #2691

To test independently, I opened a new PR was forked from that branch: #2692

However, CI behaviour on the new PR was noticeably different:

• Sometimes, no regression failures appeared(example: https://github.com/o1-labs/o1js/actions/runs/20172229813/job/57911308727) but sometimes fails at the compile step of different ZkProgram examples.

• Multiple runs showed similar “non-regression” behaviour

• This inconsistency makes it harder to pinpoint the regression source

A relevant Slack discussion:
https://o1-labs.slack.com/archives/C095BADBQ11/p1765466999157959

Some candidates for the regression included:

• PR #3335 — o1-utils: remove reliance on ToBigInt
• PR #3330 — Deps: bump num-bigint to 0.4.6

Both were reverted and tested independently, but neither reversion removed the ECDSA regression.

Notes

Performance regression results on GitHub Actions are inherently noisy due to:

• Sometimes GitHub runners can be unstable, causing variations in timing results
• Queueing delays when multiple PRs trigger CI simultaneously
• Increased elapsed time for jobs under load, e.g.:

Despite this noise, all statements above were validated across multiple CI retriggers to minimize the impact of runner instability.

[Trivo25]

However, CI behaviour on the new PR was noticeably different:
No regression failures appeared

Hmm I don't think I've obsered the same behaviour, to be honest. On the PR I opened which bumped proof-systems on main I've at least encountered CI failures 3 out of 5 times 🤔 I will trigger CI again and split it up into individual runs instead of re-running the same job

[Shigoto-dev19]

However, CI behaviour on the new PR was noticeably different:
No regression failures appeared

Hmm I don't think I've obsered the same behaviour, to be honest. On the PR I opened which bumped proof-systems on main I've at least encountered CI failures 3 out of 5 times 🤔 I will trigger CI again and split it up into individual runs instead of re-running the same job

oh yeah, that section was; it indeed passes sometimes and fails at compile step sometimes(for different examples)

I will fix this point in the report!

[Trivo25]

However, CI behaviour on the new PR was noticeably different:
No regression failures appeared

Hmm I don't think I've obsered the same behaviour, to be honest. On the PR I opened which bumped proof-systems on main I've at least encountered CI failures 3 out of 5 times 🤔 I will trigger CI again and split it up into individual runs instead of re-running the same job

oh yeah, that section was; it indeed passes sometimes and fails at compile step sometimes(for different examples)

I will fix this point in the report!

Thanks! And thanks for the summary, super helpful!

I would recommend that, once we pick this back up, we investigate the behaviour in o1js/main since main is our base branch and baseline for any regression. The regression may not seem like a big deal but it's still a change in behaviour compared to an earlier version of proof-systems so we should understand what's going on. Once we have a better understanding of what causes the problem in o1js/main we can then see if the problem persists on develop-3.0 once we've identified the core issue.

[Trivo25]

my latest results on o1js/main incl. proof systems bump

Run 1: failed. 
[Perf][ecdsa-ethers] compile: baseline=39.120676s, actual=43.144325s, regression=+10.29% (allowed +8%)
—
Run 2: failed.
[Perf][ecdsa-ethers] compile: baseline=39.120676s, actual=42.977926s, regression=+9.86% (allowed +8%)
—
Run 3: success.
—
Run 4: failed.
[Perf][ecdsa-ethers] compile: baseline=39.120676s, actual=42.629566s, regression=+8.97% (allowed +8%)
—
Run 5: success.