Skip to content

EHT-1222: Revamp bencher output #114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
spoutn1k wants to merge 13 commits into
base: main
Choose a base branch
from
Open

EHT-1222: Revamp bencher output #114

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
695e858
Omit non-threshold metrics in bencher output
spoutn1k Sep 12, 2025
2803d5d
Add memory-benchmarking
spoutn1k Sep 22, 2025
a4dfacf
Simplify usage
spoutn1k Sep 22, 2025
c249c2b
Generic trace_memory command
spoutn1k Sep 22, 2025
065b5f3
Move runtime creation to trace_memory command
spoutn1k Sep 22, 2025
6bee26f
Ditch criterion in memory benchmarks
spoutn1k Sep 23, 2025
b1ed519
Update scrape memory
spoutn1k Sep 23, 2025
cc7c6c9
Update workflows
spoutn1k Sep 23, 2025
a0d3083
Keep stdout clear
spoutn1k Sep 23, 2025
71b6991
Output MiBs again
spoutn1k Sep 23, 2025
5088f4b
empty
spoutn1k Sep 23, 2025
7848e6d
Rename benchmarks
spoutn1k Sep 23, 2025
77a7335
Use divan
spoutn1k Sep 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/lustre-metrics-bench.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ jobs:
--err \
--adapter rust_iai_callgrind \
--github-actions '${{ secrets.GITHUB_TOKEN }}' \
--ci-only-thresholds \
"cargo bench --bench lustre_metrics"

- name: Compare Lustre Metrics Benchmarks with main branch
Expand All @@ -74,4 +75,5 @@ jobs:
--err \
--adapter rust_iai_callgrind \
--github-actions '${{ secrets.GITHUB_TOKEN }}' \
--ci-only-thresholds \
"cargo bench --bench lustre_metrics"
21 changes: 12 additions & 9 deletions .github/workflows/quota-parsing.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,16 @@ jobs:
--token '${{ secrets.BENCHER_API_TOKEN }}' \
--branch main \
--testbed ci-runner \
--threshold-measure Latency \
--threshold-measure latency \
--threshold-test t_test \
--threshold-max-sample-size 64 \
--threshold-lower-boundary 0.95 \
--threshold-upper-boundary 0.95 \
--err \
--adapter rust_criterion \
--github-actions '${{ secrets.GITHUB_TOKEN }}' \
--ci-only-thresholds \
--ci-id '${{ github.workflow_ref }}-performance' \
"cargo bench --bench combine_performance"

- name: Compare quota parsing metrics benchmarks with main branch
Expand All @@ -52,13 +54,10 @@ jobs:
--err \
--adapter rust_criterion \
--github-actions '${{ secrets.GITHUB_TOKEN }}' \
--ci-only-thresholds \
--ci-id '${{ github.workflow_ref }}-performance' \
"cargo bench --bench combine_performance"

- name: Run quota parsing memory usage benchmark
run: |
# Run the benchmark first (suppress Criterion output)
cargo bench --bench combine_memory 2>&1 > /dev/null

- name: Track quota parsing memory usage benchmark on main branch
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
run: |
Expand All @@ -67,15 +66,17 @@ jobs:
--token '${{ secrets.BENCHER_API_TOKEN }}' \
--branch main \
--testbed ci-runner \
--threshold-measure peak_rss_mib \
--threshold-measure start_rss_mib \
--threshold-test t_test \
--threshold-max-sample-size 64 \
--threshold-lower-boundary 0.95 \
--threshold-upper-boundary 0.95 \
--err \
--adapter json \
--github-actions '${{ secrets.GITHUB_TOKEN }}' \
--file lustre-collector/combine_mem_usage.json
--ci-only-thresholds \
--ci-id '${{ github.workflow_ref }}-memory' \
"cargo bench --bench combine_memory"

- name: Compare quota parsing memory metrics with main branch
if: github.event_name == 'pull_request'
Expand All @@ -90,4 +91,6 @@ jobs:
--err \
--adapter json \
--github-actions '${{ secrets.GITHUB_TOKEN }}' \
--file lustre-collector/combine_mem_usage.json
--ci-only-thresholds \
--ci-id '${{ github.workflow_ref }}-memory' \
"cargo bench --bench combine_memory"
13 changes: 6 additions & 7 deletions .github/workflows/scrape-memory-metrics-bench.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,6 @@ jobs:
target/
key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}

- name: Run Scrape Memory Metrics Benchmark
run: |
# Run the benchmark first (suppress Criterion output)
cargo bench --bench scrape_memory_metrics 2>&1 > /dev/null

- name: Track scrape memory metrics benchmark on main branch
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
run: |
Expand All @@ -52,7 +47,9 @@ jobs:
--err \
--adapter json \
--github-actions '${{ secrets.GITHUB_TOKEN }}' \
--file lustrefs-exporter/scrape_allocations_results.json
--ci-only-thresholds \
--ci-id '${{ github.workflow_ref }}' \
"cargo bench --bench scrape_memory_metrics"

- name: Compare scrape memory metrics with main branch
if: github.event_name == 'pull_request'
Expand All @@ -67,4 +64,6 @@ jobs:
--err \
--adapter json \
--github-actions '${{ secrets.GITHUB_TOKEN }}' \
--file lustrefs-exporter/scrape_allocations_results.json
--ci-only-thresholds \
--ci-id '${{ github.workflow_ref }}' \
"cargo bench --bench scrape_memory_metrics"
50 changes: 50 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[workspace]
members = ['lustre-collector', 'lustrefs-exporter']
members = ['lustre-collector', 'lustrefs-exporter', "memory-benchmarking"]
resolver = "3"

[workspace.package]
Expand Down
1 change: 1 addition & 0 deletions lustre-collector/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ version.workspace = true
[dependencies]
clap = { workspace = true, features = ["derive"] }
combine.workspace = true
memory-benchmarking = { version = "0.11.0", path = "../memory-benchmarking" }
serde = { workspace = true, features = ["derive"] }
serde_json.workspace = true
serde_yaml.workspace = true
Expand Down
Loading
Loading