Skip to content

Commit 397f60e

Browse files
muscariellomuscariello
authored
feat(benchmarks): add reproducible slim benchmark suite (#155)
* feat: add performance and security tools Add slim-bench for performance benchmarking and security-probe for adversarial testing.\nIncludes initial BDD test scaffolding.\n\nSigned-off-by: Luca Muscariello <muscariello@ieee.org> Signed-off-by: Luca Muscariello <muscariello@ieee.org> * feat(bench): enhanced benchmark tool and suite Introduces slim-bench for high-throughput testing with simulated CPU metrics. Includes 16-byte payload support, detailed markdown reporting, and Dockerfile for containerized execution (though local runner defaults to native build for speed). Signed-off-by: Luca Muscariello <muscariello@ieee.org> * feat(benchmarks): add reproducible slim benchmark suite Signed-off-by: Luca Muscariello <muscariello@ieee.org> * fix(benchmarks): provision slim bindings in ci Signed-off-by: Luca Muscariello <muscariello@ieee.org> * chore(samples): drop unrelated poetry locks Signed-off-by: Luca Muscariello <muscariello@ieee.org> * chore(pr): drop planning docs Signed-off-by: Luca Muscariello <muscariello@ieee.org> * fix(benchmarks): install compatible slimctl in ci Signed-off-by: Luca Muscariello <muscariello@ieee.org> * feat(benchmarks): stream partial results to stdout Signed-off-by: Luca Muscariello <muscariello@ieee.org> * feat(benchmarks): add stdout summary markers Signed-off-by: Luca Muscariello <muscariello@ieee.org> * fix(benchmarks): stream ci logs and sample cpu on linux Signed-off-by: Luca Muscariello <muscariello@ieee.org> * chore(benchmarks): increase smoke repeat count Signed-off-by: Luca Muscariello <muscariello@ieee.org> * chore(benchmarks): print markdown reports in ci logs Signed-off-by: Luca Muscariello <muscariello@ieee.org> * chore(benchmarks): run capacity checks on prs Signed-off-by: Luca Muscariello <muscariello@ieee.org> * docs(benchmarks): clarify capacity throughput labels Signed-off-by: Luca Muscariello <muscariello@ieee.org> * chore(benchmarks): lengthen ci capacity duration Signed-off-by: Luca Muscariello <muscariello@ieee.org> * chore(benchmarks): default capacity sweeps to one client Signed-off-by: Luca Muscariello <muscariello@ieee.org> * chore(benchmarks): raise capacity payload defaults Signed-off-by: Luca Muscariello <muscariello@ieee.org> * chore(benchmarks): default capacity payloads to 16kb Signed-off-by: Luca Muscariello <muscariello@ieee.org> * chore(benchmarks): rename capacity task variants Signed-off-by: Luca Muscariello <muscariello@ieee.org> * docs(benchmarks): align capacity descriptions Signed-off-by: Luca Muscariello <muscariello@ieee.org> * feat(benchmarks): add modular slim benchmark suite Signed-off-by: Luca Muscariello <muscariello@ieee.org> * chore(integrations): remove security probe changes from benchmark pr Signed-off-by: Luca Muscariello <muscariello@ieee.org> * refactor(benchmarks): remove slim benchmark tools directory Signed-off-by: Luca Muscariello <muscariello@ieee.org> * refactor(benchmarks): use gonum t confidence intervals Signed-off-by: Luca Muscariello <muscariello@ieee.org> * feat(benchmarks): improve slim benchmark reporting and ci stats Signed-off-by: Luca Muscariello <muscariello@ieee.org> --------- Signed-off-by: Luca Muscariello <muscariello@ieee.org>
1 parent f0c8ce5 commit 397f60e

29 files changed

+4701
-9
lines changed

‎.github/workflows/test-benchmarks-slim.yaml‎

Lines changed: 299 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,299 @@
1+
# Copyright AGNTCY Contributors (https://github.com/agntcy)
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
name: test-benchmarks-slim
5+
6+
on:
7+
pull_request:
8+
paths:
9+
- '.github/workflows/test-benchmarks-slim.yaml'
10+
- 'Taskfile.yml'
11+
- 'benchmarks/Taskfile.yml'
12+
- 'benchmarks/agntcy-slim/**'
13+
workflow_dispatch:
14+
inputs:
15+
run_capacity:
16+
description: 'Run bounded capacity sweeps in addition to the smoke suite'
17+
required: false
18+
default: false
19+
type: boolean
20+
21+
jobs:
22+
slim-benchmark-smoke:
23+
runs-on: ubuntu-latest
24+
25+
permissions:
26+
contents: read
27+
28+
steps:
29+
- name: Checkout code
30+
uses: actions/checkout@v4
31+
with:
32+
fetch-depth: 0
33+
34+
- name: Setup Environment
35+
uses: ./.github/actions/setup-env
36+
with:
37+
go: true
38+
39+
- name: Install slimctl
40+
shell: bash
41+
run: |
42+
task benchmarks:slim:deps:slimctl-download SLIMCTL_PATH="$HOME/.local/bin/slimctl"
43+
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
44+
45+
- name: Run SLIM benchmark smoke suite
46+
shell: bash
47+
run: |
48+
set -o pipefail
49+
REPORT_DIR="benchmarks/agntcy-slim/reports"
50+
mkdir -p "$REPORT_DIR"
51+
stdbuf -oL -eL task benchmarks:slim:benchmark:ci:suite-smoke 2>&1 | tee "$REPORT_DIR/ci-smoke.log"
52+
53+
- name: Build smoke markdown artifact
54+
if: always()
55+
shell: bash
56+
run: |
57+
REPORT_DIR="benchmarks/agntcy-slim/reports"
58+
OUTPUT_FILE="$REPORT_DIR/ci-smoke-report.md"
59+
mkdir -p "$REPORT_DIR"
60+
61+
{
62+
echo "# SLIM Benchmark CI Smoke Report"
63+
echo
64+
echo "- Workflow: ${GITHUB_WORKFLOW}"
65+
echo "- Job: slim-benchmark-smoke"
66+
echo "- Ref: ${GITHUB_REF}"
67+
echo "- SHA: ${GITHUB_SHA}"
68+
echo
69+
70+
if [ -f "$REPORT_DIR/ci-smoke.log" ]; then
71+
echo "## Live Progress Markers"
72+
echo
73+
grep -E 'BENCHMARK_RESULT|MODE_SUMMARY|CAPACITY_SWEEP_STEP|CAPACITY_CASE_SUMMARY' "$REPORT_DIR/ci-smoke.log" || true
74+
echo
75+
fi
76+
77+
if [ -f "$REPORT_DIR/suite_summary.md" ]; then
78+
echo "## Suite Summary"
79+
echo
80+
cat "$REPORT_DIR/suite_summary.md"
81+
echo
82+
fi
83+
84+
if [ -f "$REPORT_DIR/technical_report.md" ]; then
85+
echo "## Technical Report"
86+
echo
87+
cat "$REPORT_DIR/technical_report.md"
88+
echo
89+
fi
90+
} > "$OUTPUT_FILE"
91+
92+
- name: Print smoke markdown report
93+
if: always()
94+
shell: bash
95+
run: |
96+
REPORT_DIR="benchmarks/agntcy-slim/reports"
97+
for report in \
98+
"$REPORT_DIR/ci-smoke-report.md" \
99+
"$REPORT_DIR/suite_summary.md" \
100+
"$REPORT_DIR/technical_report.md"
101+
do
102+
if [ -f "$report" ]; then
103+
echo "::group::$(basename "$report")"
104+
cat "$report"
105+
echo "::endgroup::"
106+
fi
107+
done
108+
109+
- name: Publish smoke step summary
110+
if: always()
111+
shell: bash
112+
run: |
113+
REPORT_DIR="benchmarks/agntcy-slim/reports"
114+
115+
{
116+
echo "# SLIM Benchmark Smoke Summary"
117+
echo
118+
echo "- Artifact: ci-smoke-report.md"
119+
echo "- Workflow: ${GITHUB_WORKFLOW}"
120+
echo "- Ref: ${GITHUB_REF}"
121+
echo "- SHA: ${GITHUB_SHA}"
122+
echo
123+
124+
echo "## Markdown Reports"
125+
echo
126+
for report in "$REPORT_DIR"/*.md; do
127+
if [ -f "$report" ]; then
128+
echo "- $(basename "$report")"
129+
fi
130+
done
131+
echo
132+
133+
if [ -f "$REPORT_DIR/ci-smoke.log" ]; then
134+
grep -E 'BENCHMARK_RESULT|MODE_SUMMARY|CAPACITY_SWEEP_STEP|CAPACITY_CASE_SUMMARY' "$REPORT_DIR/ci-smoke.log" | tail -n 20 || true
135+
echo
136+
fi
137+
138+
if [ -f "$REPORT_DIR/suite_summary.md" ]; then
139+
sed -n '1,80p' "$REPORT_DIR/suite_summary.md"
140+
echo
141+
fi
142+
} >> "$GITHUB_STEP_SUMMARY"
143+
144+
- name: Upload smoke markdown report
145+
if: always()
146+
uses: actions/upload-artifact@v4
147+
with:
148+
name: slim-benchmark-smoke-report
149+
path: |
150+
benchmarks/agntcy-slim/reports/*.md
151+
benchmarks/agntcy-slim/reports/ci-smoke.log
152+
retention-days: 30
153+
154+
slim-benchmark-capacity:
155+
if: ${{ github.event_name == 'pull_request' || (github.event_name == 'workflow_dispatch' && inputs.run_capacity) }}
156+
runs-on: ubuntu-latest
157+
158+
permissions:
159+
contents: read
160+
161+
steps:
162+
- name: Checkout code
163+
uses: actions/checkout@v4
164+
with:
165+
fetch-depth: 0
166+
167+
- name: Setup Environment
168+
uses: ./.github/actions/setup-env
169+
with:
170+
go: true
171+
172+
- name: Install slimctl
173+
shell: bash
174+
run: |
175+
task benchmarks:slim:deps:slimctl-download SLIMCTL_PATH="$HOME/.local/bin/slimctl"
176+
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
177+
178+
- name: Run SLIM benchmark capacity suite
179+
shell: bash
180+
run: |
181+
set -o pipefail
182+
REPORT_DIR="benchmarks/agntcy-slim/reports"
183+
mkdir -p "$REPORT_DIR"
184+
stdbuf -oL -eL task benchmarks:slim:benchmark:ci:capacity 2>&1 | tee "$REPORT_DIR/ci-capacity.log"
185+
186+
- name: Build capacity markdown artifact
187+
if: always()
188+
shell: bash
189+
run: |
190+
REPORT_DIR="benchmarks/agntcy-slim/reports"
191+
OUTPUT_FILE="$REPORT_DIR/ci-capacity-report.md"
192+
mkdir -p "$REPORT_DIR"
193+
194+
{
195+
echo "# SLIM Benchmark CI Capacity Report"
196+
echo
197+
echo "- Workflow: ${GITHUB_WORKFLOW}"
198+
echo "- Job: slim-benchmark-capacity"
199+
echo "- Ref: ${GITHUB_REF}"
200+
echo "- SHA: ${GITHUB_SHA}"
201+
echo
202+
203+
if [ -f "$REPORT_DIR/ci-capacity.log" ]; then
204+
echo "## Live Progress Markers"
205+
echo
206+
grep -E 'BENCHMARK_RESULT|MODE_SUMMARY|CAPACITY_SWEEP_STEP|CAPACITY_CASE_SUMMARY' "$REPORT_DIR/ci-capacity.log" || true
207+
echo
208+
fi
209+
210+
if [ -f "$REPORT_DIR/suite_summary.md" ]; then
211+
echo "## Suite Summary"
212+
echo
213+
cat "$REPORT_DIR/suite_summary.md"
214+
echo
215+
fi
216+
217+
if [ -f "$REPORT_DIR/technical_report.md" ]; then
218+
echo "## Technical Report"
219+
echo
220+
cat "$REPORT_DIR/technical_report.md"
221+
echo
222+
fi
223+
224+
if [ -f "$REPORT_DIR/capacity_sweep.md" ]; then
225+
echo "## Capacity Sweep"
226+
echo
227+
cat "$REPORT_DIR/capacity_sweep.md"
228+
echo
229+
fi
230+
} > "$OUTPUT_FILE"
231+
232+
- name: Print capacity markdown report
233+
if: always()
234+
shell: bash
235+
run: |
236+
REPORT_DIR="benchmarks/agntcy-slim/reports"
237+
for report in \
238+
"$REPORT_DIR/ci-capacity-report.md" \
239+
"$REPORT_DIR/suite_summary.md" \
240+
"$REPORT_DIR/technical_report.md" \
241+
"$REPORT_DIR/capacity_sweep.md" \
242+
"$REPORT_DIR/capacity-fire-and-forget.md" \
243+
"$REPORT_DIR/capacity-request-reply.md" \
244+
"$REPORT_DIR/capacity-write.md"
245+
do
246+
if [ -f "$report" ]; then
247+
echo "::group::$(basename "$report")"
248+
cat "$report"
249+
echo "::endgroup::"
250+
fi
251+
done
252+
253+
- name: Publish capacity step summary
254+
if: always()
255+
shell: bash
256+
run: |
257+
REPORT_DIR="benchmarks/agntcy-slim/reports"
258+
259+
{
260+
echo "# SLIM Benchmark Capacity Summary"
261+
echo
262+
echo "- Artifact: ci-capacity-report.md"
263+
echo "- Workflow: ${GITHUB_WORKFLOW}"
264+
echo "- Ref: ${GITHUB_REF}"
265+
echo "- SHA: ${GITHUB_SHA}"
266+
echo
267+
268+
echo "## Markdown Reports"
269+
echo
270+
for report in "$REPORT_DIR"/*.md; do
271+
if [ -f "$report" ]; then
272+
echo "- $(basename "$report")"
273+
fi
274+
done
275+
echo
276+
277+
if [ -f "$REPORT_DIR/ci-capacity.log" ]; then
278+
grep -E 'BENCHMARK_RESULT|MODE_SUMMARY|CAPACITY_SWEEP_STEP|CAPACITY_CASE_SUMMARY' "$REPORT_DIR/ci-capacity.log" | tail -n 40 || true
279+
echo
280+
fi
281+
282+
if [ -f "$REPORT_DIR/capacity_sweep.md" ]; then
283+
sed -n '1,220p' "$REPORT_DIR/capacity_sweep.md"
284+
echo
285+
elif [ -f "$REPORT_DIR/suite_summary.md" ]; then
286+
sed -n '1,80p' "$REPORT_DIR/suite_summary.md"
287+
echo
288+
fi
289+
} >> "$GITHUB_STEP_SUMMARY"
290+
291+
- name: Upload capacity markdown report
292+
if: always()
293+
uses: actions/upload-artifact@v4
294+
with:
295+
name: slim-benchmark-capacity-report
296+
path: |
297+
benchmarks/agntcy-slim/reports/*.md
298+
benchmarks/agntcy-slim/reports/ci-capacity.log
299+
retention-days: 30

0 commit comments

Comments
 (0)