Skip to content

NativeLink Benchmark Dashboard PR #1714

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
feliciien wants to merge 26 commits into from
Closed

NativeLink Benchmark Dashboard PR #1714

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
d0541f8
Trigger benchmark workflow
Apr 22, 2025
e3628c9
Enhanced benchmarking workflow
Apr 22, 2025
2d5f94f
Trigger benchmark workflow
Apr 22, 2025
34bf941
Update artifact upload to v4 in benchmarks workflow
Apr 22, 2025
1511204
Fix Bazel setup in benchmark workflow
Apr 22, 2025
7aed1d1
Fix Bazel installation in benchmarking workflow
Apr 22, 2025
102e344
Fix Bazel version parameter in workflow
Apr 22, 2025
fdf2fe8
Update Bazelisk to v2 with compatible version
Apr 22, 2025
ce073e4
Update Bazelisk to v1.19.0 for cache compatibility
Apr 22, 2025
8195fde
Add benchmarking workflow
Apr 23, 2025
0299d19
Add enhanced benchmark workflow and documentation
Apr 23, 2025
d6e1420
Add enhanced benchmark workflow and documentation
Apr 23, 2025
aac4231
Add enhanced benchmark workflow and documentation
Apr 23, 2025
23bc1d4
Add enhanced benchmark workflow and documentation
Apr 23, 2025
7e372ee
Add enhanced benchmark workflow and documentation
Apr 23, 2025
2b9a01c
Add enhanced benchmark workflow and documentation
Apr 23, 2025
4a7de03
Fix benchmark workflow by switching to bazel-contrib/setup-bazel
Apr 23, 2025
d7343be
Update workflow trigger to include benchmark-test branch
Apr 23, 2025
3be5d9e
Verification commit
Apr 23, 2025
c97fdbc
Fix Bazel setup and Python syntax errors in benchmark workflow
Apr 23, 2025
efdde29
Force workflow refresh
Apr 23, 2025
e873591
Final workflow verification
Apr 23, 2025
8c8973c
Final workflow verification
Apr 23, 2025
2dd096d
Final cache resolution
Apr 23, 2025
f7536aa
Final cache configuration verification
Apr 23, 2025
3c52ec8
ci: Add per-commit benchmarking automation with visualization
feliciien Apr 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 123 additions & 0 deletions .github/workflows/benchmark.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
---
name: NativeLink Benchmarks

on:
push:
branches: [main]
pull_request:
branches: [main]
paths-ignore:
- 'docs/**'

permissions: read-all

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}

jobs:
benchmark:
name: Run NativeLink Benchmarks
runs-on: ubuntu-24.04
environment: production
timeout-minutes: 60
steps:
- name: Checkout
uses: >- # v4.2.2
actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683

- name: Prepare Worker
uses: ./.github/actions/prepare-nix

- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: '3.10'

- name: Setup Bazel
uses: >- # v0.13.0
bazel-contrib/setup-bazel@663f88d97adf17db2523a5b385d9407a562e5551
with:
bazelisk-cache: true
repository-cache: true

- name: Checkout Test Project (Rust)
uses: actions/checkout@v4
with:
repository: 'rust-lang/rustlings'
path: 'benchmark-projects/rustlings'

- name: Checkout Test Project (C++)
uses: actions/checkout@v4
with:
repository: 'google/googletest'
path: 'benchmark-projects/googletest'

- name: Get Previous Commit
id: get-prev-commit
run: |
if [[ "${{ github.event_name }}" == "pull_request" ]]; then
echo "prev_commit=${{ github.event.pull_request.base.sha }}" >> $GITHUB_OUTPUT
else
echo "prev_commit=$(git rev-parse HEAD~1)" >> $GITHUB_OUTPUT
fi

- name: Run Benchmarks (Rust Project)
run: |
python3 tools/benchmark/benchmark.py \
--project=benchmark-projects/rustlings \
--commit=${{ github.sha }} \
--compare-to=${{ steps.get-prev-commit.outputs.prev_commit }} \
--nativelink-url=https://app.nativelink.com \
--api-key=${{ secrets.NATIVELINK_COM_API_HEADER }} \
--output-dir=./benchmark_results

- name: Run Benchmarks (C++ Project)
run: |
python3 tools/benchmark/benchmark.py \
--project=benchmark-projects/googletest \
--commit=${{ github.sha }} \
--compare-to=${{ steps.get-prev-commit.outputs.prev_commit }} \
--nativelink-url=https://app.nativelink.com \
--api-key=${{ secrets.NATIVELINK_COM_API_HEADER }} \
--output-dir=./benchmark_results

- name: Upload Benchmark Results
uses: actions/upload-artifact@v4
with:
name: benchmark-results
path: ./benchmark_results
retention-days: 90

- name: Generate Benchmark Report
run: |
echo "## NativeLink Benchmark Results" > benchmark_report.md
echo "" >> benchmark_report.md
echo "### Commit: ${{ github.sha }}" >> benchmark_report.md
echo "" >> benchmark_report.md
echo "#### Rust Project (rustlings)" >> benchmark_report.md
echo "" >> benchmark_report.md
echo '```' >> benchmark_report.md
cat ./benchmark_results/rustlings_${{ github.sha }}_*.json >> benchmark_report.md
echo '```' >> benchmark_report.md
echo "" >> benchmark_report.md
echo "#### C++ Project (googletest)" >> benchmark_report.md
echo "" >> benchmark_report.md
echo '```' >> benchmark_report.md
cat ./benchmark_results/googletest_${{ github.sha }}_*.json >> benchmark_report.md
echo '```' >> benchmark_report.md

- name: Comment PR
uses: actions/github-script@v7
if: github.event_name == 'pull_request'
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const fs = require('fs');
const report = fs.readFileSync('benchmark_report.md', 'utf8');
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: report
});
159 changes: 159 additions & 0 deletions .github/workflows/benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
---
name: NativeLink Benchmarks

on:
push:
branches: [main, benchmark-test] # Add your current branch here
pull_request:
branches: [main]
paths-ignore:
- 'docs/**'

permissions: read-all

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}

jobs:
benchmark:
name: Run NativeLink Benchmarks
runs-on: ubuntu-24.04
environment: production
timeout-minutes: 60
steps:
- name: Checkout
uses: >- # v4.2.2
actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683

- name: Prepare Worker
uses: ./.github/actions/prepare-nix

- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: '3.10'

- name: Install Dependencies
run: |
pip install -r tools/benchmark/requirements.txt

- name: Clean GitHub Actions cache
uses: actions/cache/restore@v3
with:
path: |
~/.cache/bazel
~/.bazelisk
key: ${{ runner.os }}-bazel-cache-${{ hashFiles('.github/workflows/benchmark.yml') }}
restore-keys: |
${{ runner.os }}-bazel-cache-

- name: Setup Bazel
uses: bazel-contrib/setup-bazel@663f88d97adf17db2523a5b385d9407a562e5551
with:
bazelisk-version: 'v1.19.0'

- name: Checkout Test Project (Rust)
uses: actions/checkout@v4
with:
repository: 'rust-lang/rustlings'
path: 'benchmark-projects/rustlings'

# 新增Bazel初始化步骤
- name: Initialize Bazel workspace
run: |
cd benchmark-projects/rustlings
touch WORKSPACE.bazel

- name: Checkout Test Project (C++)
uses: actions/checkout@v4
with:
repository: 'google/googletest'
path: 'benchmark-projects/googletest'

- name: Get Previous Commit
id: get-prev-commit
run: |
if [[ "${{ github.event_name }}" == "pull_request" ]]; then
echo "prev_commit=${{ github.event.pull_request.base.sha }}" >> $GITHUB_OUTPUT
else
echo "prev_commit=$(git rev-parse HEAD~1)" >> $GITHUB_OUTPUT
fi

- name: Run Benchmarks (Rust Project)
run: |
python3 tools/benchmark/benchmark.py \
--project=benchmark-projects/rustlings \
--commit=${{ github.sha }} \
--compare-to=${{ steps.get-prev-commit.outputs.prev_commit }} \
--nativelink-url=https://app.nativelink.com \
--api-key=${{ secrets.NATIVELINK_COM_API_HEADER }} \
--output-dir=./benchmark_results \
--runs=3

- name: Run Benchmarks (C++ Project)
run: |
python3 tools/benchmark/benchmark.py \
--project=benchmark-projects/googletest \
--commit=${{ github.sha }} \
--compare-to=${{ steps.get-prev-commit.outputs.prev_commit }} \
--nativelink-url=https://app.nativelink.com \
--api-key=${{ secrets.NATIVELINK_COM_API_HEADER }} \
--output-dir=./benchmark_results \
--runs=3

- name: Upload Benchmark Results
uses: actions/upload-artifact@v4
with:
name: benchmark-results
path: ./benchmark_results
retention-days: 90

- name: Generate Benchmark Report
run: |
echo "## NativeLink Benchmark Results" > benchmark_report.md
echo "" >> benchmark_report.md
echo "### Commit: ${{ github.sha }}" >> benchmark_report.md
echo "" >> benchmark_report.md
echo "#### Rust Project (rustlings)" >> benchmark_report.md
echo "" >> benchmark_report.md
echo '```' >> benchmark_report.md
cat ./benchmark_results/rustlings_${{ github.sha }}_*.json >> benchmark_report.md
echo '```' >> benchmark_report.md
echo "" >> benchmark_report.md
echo "#### C++ Project (googletest)" >> benchmark_report.md
echo "" >> benchmark_report.md
echo '```' >> benchmark_report.md
cat ./benchmark_results/googletest_${{ github.sha }}_*.json >> benchmark_report.md
echo '```' >> benchmark_report.md

- name: Comment PR
if: github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const fs = require('fs');
const report = fs.readFileSync('benchmark_report.md', 'utf8');
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: report
});

# Setup GitHub Pages deployment
- name: Setup Pages
if: github.ref == 'refs/heads/main'
uses: actions/configure-pages@v4

- name: Upload Pages Artifact
if: github.ref == 'refs/heads/main'
uses: actions/upload-pages-artifact@v3
with:
path: './benchmark_viz'

- name: Deploy to GitHub Pages
if: github.ref == 'refs/heads/main'
id: deployment
uses: actions/deploy-pages@v4
58 changes: 58 additions & 0 deletions .github/workflows/benchmarks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
name: NativeLink Benchmarks
on: [push]

jobs:
benchmark:
runs-on: ubuntu-latest
# container: nativelink-benchmark-env # Comment out until image exists
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Setup Bazel
uses: bazelbuild/setup-bazelisk@v2
with:
bazelisk-version: "1.19.0" # Correct parameter name and use quotes

- name: Install Python dependencies
run: |
python3 -m pip install -r tools/benchmark/requirements.txt

- name: Run Benchmarks
env:
NATIVELINK_API_KEY: ${{ secrets.NATIVELINK_API_KEY }}
run: |
echo "Benchmarking commit: $GITHUB_SHA"
echo "Current directory: $(pwd)"
ls -la ./tools/benchmark/

./tools/benchmark/run_benchmarks.sh \
--project=./integration_tests/test_project \
--commit=$GITHUB_SHA \
--output-dir=./benchmark_results \
--nativelink-url=https://app.nativelink.com \
--iterations=5 \
--cold-cache \
--warm-cache \
--incremental-build

- name: Analyze Results
run: |
python3 tools/benchmark/analyze_results.py \
--current ./benchmark_results \
--baseline ./baseline_metrics \
--output ./report.md

- name: Upload Benchmark Results
uses: actions/upload-artifact@v4
with:
name: benchmark-results
path: ./benchmark_results
retention-days: 90

- name: Upload Benchmark Visualizations
uses: actions/upload-artifact@v4
with:
name: benchmark-visualizations
path: ./benchmark_viz
retention-days: 90
Loading
Loading