Skip to content

Conversation

@xlc
Copy link
Member

@xlc xlc commented Sep 8, 2025

Summary of Changes:

  • Introduces a new benchmark suite for core JAM components using the ordo-one/package-benchmark library.
  • Adds a new JAMBenchmarks executable target to the JAMTests package to house the benchmarks.
  • Implements benchmarks for several key areas based on W3F test vectors:
    • Erasure Coding (encode + reconstruct)
    • Shuffling
    • JamTestnet State Transition Function (STF) application
    • RecentHistory STF updates
  • Creates a new BenchSupport.swift helper to expose test data from the JAMTests module to the external benchmark executable.
  • Updates VS Code launch.json with configurations for running and debugging the benchmark suite.
  • Adjusts visibility and concurrency conformance (public, Sendable) in JamTestnet.swift to allow its use in the benchmark target.

Highlights of Changed Code:

  • JAMTests/Benchmarks/JAMBenchmarks/JAMBenchmarks.swift: New file containing all benchmark definitions.
  • JAMTests/Package.swift: Addition of the package-benchmark dependency and the new JAMBenchmarks executable target.
  • JAMTests/Sources/JAMTests/BenchSupport.swift: New public helper JAMBenchSupport for loading test data.
  • JAMTests/Sources/JAMTests/JamTestnet.swift: runSTF is now public, and JamTestnetTestcase and related types are Sendable.
  • .vscode/launch.json: Added "Debug JAMBenchmarks" and "Release JAMBenchmarks" configurations.
  • Blockchain/Sources/Blockchain/RuntimeProtocols/Accumulation.swift: Minor stylistic refactor from closure-based flatMap to key-path expressions.

Additional Information:

  • Testing: The new benchmarks can be executed using the Swift Package Manager (swift benchmark) or via the newly added VS Code launch configurations.

Adds a `JAMBenchmarks` executable using `swift-benchmark` to test the performance of erasure coding, shuffling, and STF application. This change includes necessary helpers, public API adjustments, and VS Code launch configurations.
Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes introduce a benchmark suite, which is a great addition. However, the implementation of the asynchronous benchmarks uses a custom, unsafe helper to block on async operations. This helper is unnecessary as the benchmarking library natively supports async/await, and its use should be replaced to improve code safety and simplicity.

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes introduce a new benchmark suite, which is a great addition. The implementation is mostly solid, but there's a potential for crashes in the benchmark executable due to force-unwrapping (try!) during test data decoding. This should be addressed to make the benchmarks more robust.

@qiweiii qiweiii changed the title feat: Add benchmark suite for core JAM components feat: add benchmark with ci Sep 10, 2025
@codecov
Copy link

codecov bot commented Sep 10, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.75%. Comparing base (cfe98d5) to head (e9b7423).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #365      +/-   ##
==========================================
+ Coverage   81.64%   81.75%   +0.10%     
==========================================
  Files         381      381              
  Lines       33572    33572              
==========================================
+ Hits        27410    27446      +36     
+ Misses       6162     6126      -36     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes introduce a new benchmark suite, which is a great addition. However, there are a few issues to address. The primary issue is that one of the new benchmarks incorrectly includes data decoding time in its measurement, which will lead to inaccurate results. Additionally, there's a minor improvement to be made in the Swift package dependency declaration and a best-practice adjustment for a GitHub Action version.

@xlc xlc merged commit d3baee2 into master Sep 11, 2025
7 checks passed
@xlc xlc deleted the benchmark branch September 11, 2025 05:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants