Skip to content

Conversation

@feliciien
Copy link

@feliciien feliciien commented Apr 23, 2025

This PR implements a per-commit benchmarking system for NativeLink inspired by the Lucene Nightly Benchmarks project.

Features:

  • Benchmarks for both Rust (rustlings) and C++ (googletest) projects
  • Tests remote cache only and remote cache + execution scenarios
  • Measures incremental build performance
  • Generates visualizations and reports
  • Detects performance regressions
  • Uses NativeLink cloud service (app.nativelink.com)

Fixes #1700

Type of change

  • New feature (non-breaking change which adds functionality)

How Has This Been Tested?

The benchmarking system has been tested locally with the following configuration:

  • Rust project (rustlings) for testing remote cache, remote execution, and incremental builds
  • C++ project (googletest) for testing remote cache performance
  • Integration with NativeLink cloud service (app.nativelink.com)

Checklist

  • Updated documentation if needed
  • Tests added/amended
  • bazel test //... passes locally

/claim #1700


This change is Reviewable

@CLAassistant
Copy link

CLAassistant commented Apr 23, 2025

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ feliciien
❌ diouf


diouf seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

- Implement GitHub Actions workflow for automated benchmarking
- Add dashboard for visualizing build times across commits
- Include Slack notifications for regressions
- Inspired by Lucene Nightly Benchmarks architecture
return jsonify(projects)

if __name__ == '__main__':
app.run(debug=True, host='0.0.0.0') No newline at end of file

Check failure

Code scanning / CodeQL

Flask app is run in debug mode High test

A Flask app appears to be run in debug mode. This may allow an attacker to run arbitrary code through the debugger.
Introduce a suite of scripts to automate benchmarking, result storage, regression detection, and report generation for NativeLink. This includes:
- `run_benchmarks.sh`: Orchestrates benchmarking across multiple commits.
- `benchmark_runner.py`: Executes benchmarks for specific configurations.
- `db_uploader.py`: Uploads benchmark results to MongoDB.
- `regression_detector.py`: Detects performance regressions.
- `report_generator.py`: Generates visual and HTML reports from benchmark data.

These tools enable continuous performance monitoring and regression detection for NativeLink.
@MarcusSorealheis MarcusSorealheis marked this pull request as draft April 23, 2025 23:25
@feliciien feliciien closed this Apr 24, 2025
@feliciien feliciien deleted the feature/per-commit-benchmarks branch April 28, 2025 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Benchmarking on Per-Commit Basis

2 participants