Skip to content

feat(benchmarks): add reproducible slim benchmark suite #155

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
muscariello merged 25 commits into from
Mar 25, 2026
Merged

feat(benchmarks): add reproducible slim benchmark suite #155

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
a0d1d38
feat: add performance and security tools
muscariello Mar 20, 2026
b87c3a2
feat(bench): enhanced benchmark tool and suite
muscariello Mar 20, 2026
dfdb156
feat(benchmarks): add reproducible slim benchmark suite
muscariello Mar 21, 2026
0949c9b
fix(benchmarks): provision slim bindings in ci
muscariello Mar 21, 2026
605864f
chore(samples): drop unrelated poetry locks
muscariello Mar 21, 2026
e867283
chore(pr): drop planning docs
muscariello Mar 21, 2026
096295a
fix(benchmarks): install compatible slimctl in ci
muscariello Mar 21, 2026
76f14c7
feat(benchmarks): stream partial results to stdout
muscariello Mar 21, 2026
cd9f6f5
feat(benchmarks): add stdout summary markers
muscariello Mar 21, 2026
e6724d5
fix(benchmarks): stream ci logs and sample cpu on linux
muscariello Mar 21, 2026
6bfabeb
chore(benchmarks): increase smoke repeat count
muscariello Mar 21, 2026
19de996
chore(benchmarks): print markdown reports in ci logs
muscariello Mar 21, 2026
269173d
chore(benchmarks): run capacity checks on prs
muscariello Mar 21, 2026
fe7ae95
docs(benchmarks): clarify capacity throughput labels
muscariello Mar 21, 2026
aff43cb
chore(benchmarks): lengthen ci capacity duration
muscariello Mar 21, 2026
8a44f68
chore(benchmarks): default capacity sweeps to one client
muscariello Mar 21, 2026
8bff5bc
chore(benchmarks): raise capacity payload defaults
muscariello Mar 21, 2026
db03f24
chore(benchmarks): default capacity payloads to 16kb
muscariello Mar 21, 2026
97858e8
chore(benchmarks): rename capacity task variants
muscariello Mar 21, 2026
8d7ab52
docs(benchmarks): align capacity descriptions
muscariello Mar 21, 2026
42b73d2
feat(benchmarks): add modular slim benchmark suite
muscariello Mar 22, 2026
3f90f60
chore(integrations): remove security probe changes from benchmark pr
muscariello Mar 22, 2026
3faacb4
refactor(benchmarks): remove slim benchmark tools directory
muscariello Mar 22, 2026
b1d69fe
refactor(benchmarks): use gonum t confidence intervals
muscariello Mar 22, 2026
eb30448
feat(benchmarks): improve slim benchmark reporting and ci stats
muscariello Mar 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
299 changes: 299 additions & 0 deletions .github/workflows/test-benchmarks-slim.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,299 @@
# Copyright AGNTCY Contributors (https://github.com/agntcy)
# SPDX-License-Identifier: Apache-2.0

name: test-benchmarks-slim

on:
pull_request:
paths:
- '.github/workflows/test-benchmarks-slim.yaml'
- 'Taskfile.yml'
- 'benchmarks/Taskfile.yml'
- 'benchmarks/agntcy-slim/**'
workflow_dispatch:
inputs:
run_capacity:
description: 'Run bounded capacity sweeps in addition to the smoke suite'
required: false
default: false
type: boolean

jobs:
slim-benchmark-smoke:
runs-on: ubuntu-latest

permissions:
contents: read

steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Setup Environment
uses: ./.github/actions/setup-env
with:
go: true

- name: Install slimctl
shell: bash
run: |
task benchmarks:slim:deps:slimctl-download SLIMCTL_PATH="$HOME/.local/bin/slimctl"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"

- name: Run SLIM benchmark smoke suite
shell: bash
run: |
set -o pipefail
REPORT_DIR="benchmarks/agntcy-slim/reports"
mkdir -p "$REPORT_DIR"
stdbuf -oL -eL task benchmarks:slim:benchmark:ci:suite-smoke 2>&1 | tee "$REPORT_DIR/ci-smoke.log"

- name: Build smoke markdown artifact
if: always()
shell: bash
run: |
REPORT_DIR="benchmarks/agntcy-slim/reports"
OUTPUT_FILE="$REPORT_DIR/ci-smoke-report.md"
mkdir -p "$REPORT_DIR"

{
echo "# SLIM Benchmark CI Smoke Report"
echo
echo "- Workflow: ${GITHUB_WORKFLOW}"
echo "- Job: slim-benchmark-smoke"
echo "- Ref: ${GITHUB_REF}"
echo "- SHA: ${GITHUB_SHA}"
echo

if [ -f "$REPORT_DIR/ci-smoke.log" ]; then
echo "## Live Progress Markers"
echo
grep -E 'BENCHMARK_RESULT|MODE_SUMMARY|CAPACITY_SWEEP_STEP|CAPACITY_CASE_SUMMARY' "$REPORT_DIR/ci-smoke.log" || true
echo
fi

if [ -f "$REPORT_DIR/suite_summary.md" ]; then
echo "## Suite Summary"
echo
cat "$REPORT_DIR/suite_summary.md"
echo
fi

if [ -f "$REPORT_DIR/technical_report.md" ]; then
echo "## Technical Report"
echo
cat "$REPORT_DIR/technical_report.md"
echo
fi
} > "$OUTPUT_FILE"

- name: Print smoke markdown report
if: always()
shell: bash
run: |
REPORT_DIR="benchmarks/agntcy-slim/reports"
for report in \
"$REPORT_DIR/ci-smoke-report.md" \
"$REPORT_DIR/suite_summary.md" \
"$REPORT_DIR/technical_report.md"
do
if [ -f "$report" ]; then
echo "::group::$(basename "$report")"
cat "$report"
echo "::endgroup::"
fi
done

- name: Publish smoke step summary
if: always()
shell: bash
run: |
REPORT_DIR="benchmarks/agntcy-slim/reports"

{
echo "# SLIM Benchmark Smoke Summary"
echo
echo "- Artifact: ci-smoke-report.md"
echo "- Workflow: ${GITHUB_WORKFLOW}"
echo "- Ref: ${GITHUB_REF}"
echo "- SHA: ${GITHUB_SHA}"
echo

echo "## Markdown Reports"
echo
for report in "$REPORT_DIR"/*.md; do
if [ -f "$report" ]; then
echo "- $(basename "$report")"
fi
done
echo

if [ -f "$REPORT_DIR/ci-smoke.log" ]; then
grep -E 'BENCHMARK_RESULT|MODE_SUMMARY|CAPACITY_SWEEP_STEP|CAPACITY_CASE_SUMMARY' "$REPORT_DIR/ci-smoke.log" | tail -n 20 || true
echo
fi

if [ -f "$REPORT_DIR/suite_summary.md" ]; then
sed -n '1,80p' "$REPORT_DIR/suite_summary.md"
echo
fi
} >> "$GITHUB_STEP_SUMMARY"

- name: Upload smoke markdown report
if: always()
uses: actions/upload-artifact@v4
with:
name: slim-benchmark-smoke-report
path: |
benchmarks/agntcy-slim/reports/*.md
benchmarks/agntcy-slim/reports/ci-smoke.log
retention-days: 30

slim-benchmark-capacity:
if: ${{ github.event_name == 'pull_request' || (github.event_name == 'workflow_dispatch' && inputs.run_capacity) }}
runs-on: ubuntu-latest

permissions:
contents: read

steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Setup Environment
uses: ./.github/actions/setup-env
with:
go: true

- name: Install slimctl
shell: bash
run: |
task benchmarks:slim:deps:slimctl-download SLIMCTL_PATH="$HOME/.local/bin/slimctl"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"

- name: Run SLIM benchmark capacity suite
shell: bash
run: |
set -o pipefail
REPORT_DIR="benchmarks/agntcy-slim/reports"
mkdir -p "$REPORT_DIR"
stdbuf -oL -eL task benchmarks:slim:benchmark:ci:capacity 2>&1 | tee "$REPORT_DIR/ci-capacity.log"

- name: Build capacity markdown artifact
if: always()
shell: bash
run: |
REPORT_DIR="benchmarks/agntcy-slim/reports"
OUTPUT_FILE="$REPORT_DIR/ci-capacity-report.md"
mkdir -p "$REPORT_DIR"

{
echo "# SLIM Benchmark CI Capacity Report"
echo
echo "- Workflow: ${GITHUB_WORKFLOW}"
echo "- Job: slim-benchmark-capacity"
echo "- Ref: ${GITHUB_REF}"
echo "- SHA: ${GITHUB_SHA}"
echo

if [ -f "$REPORT_DIR/ci-capacity.log" ]; then
echo "## Live Progress Markers"
echo
grep -E 'BENCHMARK_RESULT|MODE_SUMMARY|CAPACITY_SWEEP_STEP|CAPACITY_CASE_SUMMARY' "$REPORT_DIR/ci-capacity.log" || true
echo
fi

if [ -f "$REPORT_DIR/suite_summary.md" ]; then
echo "## Suite Summary"
echo
cat "$REPORT_DIR/suite_summary.md"
echo
fi

if [ -f "$REPORT_DIR/technical_report.md" ]; then
echo "## Technical Report"
echo
cat "$REPORT_DIR/technical_report.md"
echo
fi

if [ -f "$REPORT_DIR/capacity_sweep.md" ]; then
echo "## Capacity Sweep"
echo
cat "$REPORT_DIR/capacity_sweep.md"
echo
fi
} > "$OUTPUT_FILE"

- name: Print capacity markdown report
if: always()
shell: bash
run: |
REPORT_DIR="benchmarks/agntcy-slim/reports"
for report in \
"$REPORT_DIR/ci-capacity-report.md" \
"$REPORT_DIR/suite_summary.md" \
"$REPORT_DIR/technical_report.md" \
"$REPORT_DIR/capacity_sweep.md" \
"$REPORT_DIR/capacity-fire-and-forget.md" \
"$REPORT_DIR/capacity-request-reply.md" \
"$REPORT_DIR/capacity-write.md"
do
if [ -f "$report" ]; then
echo "::group::$(basename "$report")"
cat "$report"
echo "::endgroup::"
fi
done

- name: Publish capacity step summary
if: always()
shell: bash
run: |
REPORT_DIR="benchmarks/agntcy-slim/reports"

{
echo "# SLIM Benchmark Capacity Summary"
echo
echo "- Artifact: ci-capacity-report.md"
echo "- Workflow: ${GITHUB_WORKFLOW}"
echo "- Ref: ${GITHUB_REF}"
echo "- SHA: ${GITHUB_SHA}"
echo

echo "## Markdown Reports"
echo
for report in "$REPORT_DIR"/*.md; do
if [ -f "$report" ]; then
echo "- $(basename "$report")"
fi
done
echo

if [ -f "$REPORT_DIR/ci-capacity.log" ]; then
grep -E 'BENCHMARK_RESULT|MODE_SUMMARY|CAPACITY_SWEEP_STEP|CAPACITY_CASE_SUMMARY' "$REPORT_DIR/ci-capacity.log" | tail -n 40 || true
echo
fi

if [ -f "$REPORT_DIR/capacity_sweep.md" ]; then
sed -n '1,220p' "$REPORT_DIR/capacity_sweep.md"
echo
elif [ -f "$REPORT_DIR/suite_summary.md" ]; then
sed -n '1,80p' "$REPORT_DIR/suite_summary.md"
echo
fi
} >> "$GITHUB_STEP_SUMMARY"

- name: Upload capacity markdown report
if: always()
uses: actions/upload-artifact@v4
with:
name: slim-benchmark-capacity-report
path: |
benchmarks/agntcy-slim/reports/*.md
benchmarks/agntcy-slim/reports/ci-capacity.log
retention-days: 30
Loading
Loading