Skip to content

dhuseby/feat/perf baseline iperf #766

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dhuseby merged 9 commits into from
Dec 16, 2025
Merged

dhuseby/feat/perf baseline iperf #766

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
c63f477
adding iperf baseline test
Dec 15, 2025
cbbba60
merging master
Dec 15, 2025
ddce66b
fixed the impls measurements
Dec 15, 2025
bd9ec10
fixing action and workflows to support perf properly
Dec 15, 2025
30b52b8
clean up DEBUG and Rust perf warnings
Dec 15, 2025
35c46c7
remove redundant detect-changes
Dec 15, 2025
0bbb65a
add box plotting
Dec 15, 2025
143455f
remove box-plot.json generation
Dec 15, 2025
ade26cc
fixed baselines and graphing
Dec 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 56 additions & 2 deletions .github/actions/detect-changed-impls/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ outputs:
changed-routers:
description: "Pipe-separated list of changed router IDs (hole-punch only)"
value: ${{ steps.detect.outputs.changed-routers }}
changed-baselines:
description: "Pipe-separated list of changed baseline IDs (perf only)"
value: ${{ steps.detect.outputs.changed-baselines }}
has-changes:
description: "Boolean indicating if any changes were detected"
value: ${{ steps.detect.outputs.has-changes }}
Expand Down Expand Up @@ -66,6 +69,7 @@ runs:
CHANGED_IMPLS=""
CHANGED_RELAYS=""
CHANGED_ROUTERS=""
CHANGED_BASELINES=""
IMPLS_YAML_CHANGED="false"
IMPLS_FOLDER_CHANGED="false"

Expand Down Expand Up @@ -140,6 +144,29 @@ runs:
done
fi

# For perf: also check baselines
if [ "$TEST_TYPE" = "perf" ]; then
# Check baselines
OLD_BASELINES=$(yq eval '.baselines[].id' /tmp/impls-old.yaml 2>/dev/null | sort || echo "")
NEW_BASELINES=$(yq eval '.baselines[].id' /tmp/impls-new.yaml 2>/dev/null | sort || echo "")

for baseline_id in $NEW_BASELINES; do
if echo "$OLD_BASELINES" | grep -q "^${baseline_id}$"; then
# Exists in old, check if content changed
OLD_BASELINE=$(yq eval ".baselines[] | select(.id == \"$baseline_id\")" /tmp/impls-old.yaml 2>/dev/null || echo "")
NEW_BASELINE=$(yq eval ".baselines[] | select(.id == \"$baseline_id\")" /tmp/impls-new.yaml 2>/dev/null || echo "")
if [ "$OLD_BASELINE" != "$NEW_BASELINE" ]; then
CHANGED_BASELINES="${CHANGED_BASELINES:+$CHANGED_BASELINES|}$baseline_id"
echo " → Changed baseline: $baseline_id"
fi
else
# New baseline
CHANGED_BASELINES="${CHANGED_BASELINES:+$CHANGED_BASELINES|}$baseline_id"
echo " → New baseline: $baseline_id"
fi
done
fi

rm -f /tmp/impls-old.yaml /tmp/impls-new.yaml
fi

Expand Down Expand Up @@ -226,20 +253,47 @@ runs:
fi
done
fi

# For perf: also check baselines
if [ "$TEST_TYPE" = "perf" ]; then
# Check baselines
BASELINE_COUNT=$(yq eval '.baselines | length' "$IMPLS_FILE" 2>/dev/null || echo "0")
for ((i=0; i<BASELINE_COUNT; i++)); do
SOURCE_TYPE=$(yq eval ".baselines[$i].source.type" "$IMPLS_FILE" 2>/dev/null)
if [ "$SOURCE_TYPE" = "local" ]; then
SOURCE_PATH=$(yq eval ".baselines[$i].source.path" "$IMPLS_FILE")
FULL_SOURCE_PATH="perf/$SOURCE_PATH"

if [[ "$changed_file" == "$FULL_SOURCE_PATH"* ]]; then
BASELINE_ID=$(yq eval ".baselines[$i].id" "$IMPLS_FILE")
if [[ "|$CHANGED_BASELINES|" != *"|$BASELINE_ID|"* ]]; then
CHANGED_BASELINES="${CHANGED_BASELINES:+$CHANGED_BASELINES|}$BASELINE_ID"
echo " ✓ Matched to baseline: $BASELINE_ID"
fi
fi
fi
done
fi
done <<< "$IMPLS_FOLDER_FILES"
fi

# Output all results
echo "changed-impls=$CHANGED_IMPLS" >> $GITHUB_OUTPUT
echo "changed-relays=$CHANGED_RELAYS" >> $GITHUB_OUTPUT
echo "changed-routers=$CHANGED_ROUTERS" >> $GITHUB_OUTPUT
echo "changed-baselines=$CHANGED_BASELINES" >> $GITHUB_OUTPUT
echo "impls-yaml-changed=$IMPLS_YAML_CHANGED" >> $GITHUB_OUTPUT
echo "impls-folder-changed=$IMPLS_FOLDER_CHANGED" >> $GITHUB_OUTPUT

# Set has-changes
if [ -n "$CHANGED_IMPLS" ] || [ -n "$CHANGED_RELAYS" ] || [ -n "$CHANGED_ROUTERS" ]; then
if [ -n "$CHANGED_IMPLS" ] || [ -n "$CHANGED_RELAYS" ] || [ -n "$CHANGED_ROUTERS" ] || [ -n "$CHANGED_BASELINES" ]; then
echo "has-changes=true" >> $GITHUB_OUTPUT
echo "→ Summary: impls=$CHANGED_IMPLS, relays=$CHANGED_RELAYS, routers=$CHANGED_ROUTERS"
# Update summary to include baselines
if [ "$TEST_TYPE" = "perf" ]; then
echo "→ Summary: impls=$CHANGED_IMPLS, relays=$CHANGED_RELAYS, routers=$CHANGED_ROUTERS, baselines=$CHANGED_BASELINES"
else
echo "→ Summary: impls=$CHANGED_IMPLS, relays=$CHANGED_RELAYS, routers=$CHANGED_ROUTERS"
fi
else
echo "has-changes=false" >> $GITHUB_OUTPUT
echo "→ No relevant changes detected"
Expand Down
11 changes: 10 additions & 1 deletion .github/actions/run-bash-perf-test/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,9 +171,18 @@ runs:
${{ steps.find-test-pass.outputs.test-pass-dir }}/results.yaml
${{ steps.find-test-pass.outputs.test-pass-dir }}/results.md
${{ steps.find-test-pass.outputs.test-pass-dir }}/results.html
${{ steps.find-test-pass.outputs.test-pass-dir }}/boxplot-data.json
if-no-files-found: warn

- name: Upload box plot images
if: always()
uses: actions/upload-artifact@v4
with:
name: perf-test-box-plot-${{ steps.find-test-pass.outputs.test-pass-name }}
path: |
${{ steps.find-test-pass.outputs.test-pass-dir }}/boxplot-*.png
if-no-files-found: ignore
compression-level: 0

- name: Upload snapshot archive
if: always() && inputs.snapshot == 'true'
uses: actions/upload-artifact@v4
Expand Down
117 changes: 0 additions & 117 deletions .github/workflows/daily-full-interop.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,45 +120,6 @@ on:
type: boolean

jobs:
check-transport-changes:
if: github.event_name != 'workflow_dispatch'
runs-on: [self-hosted, linux, x64, ephemeral]
outputs:
has-changes: ${{ steps.check.outputs.has-changes }}
impls-yaml-changed: ${{ steps.check.outputs.impls-yaml-changed }}
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Check transport changes in last 24 hours
id: check
shell: bash
run: |
# Check if anything in transport/ changed in last 24 hours
CHANGES=$(git log --oneline --since="24 hours ago" -- "transport/**" | wc -l)

if [ "$CHANGES" -eq 0 ]; then
echo "has-changes=false" >> $GITHUB_OUTPUT
echo "impls-yaml-changed=false" >> $GITHUB_OUTPUT
echo "→ No changes in transport/ in last 24 hours, skipping tests"
exit 0
fi

echo "has-changes=true" >> $GITHUB_OUTPUT

# Check specifically if impls.yaml changed in last 24 hours
IMPLS_CHANGES=$(git log --oneline --since="24 hours ago" -- "transport/impls.yaml" | wc -l)

if [ "$IMPLS_CHANGES" -gt 0 ]; then
echo "impls-yaml-changed=true" >> $GITHUB_OUTPUT
echo "→ transport/impls.yaml changed in last 24h, will force matrix rebuild"
else
echo "impls-yaml-changed=false" >> $GITHUB_OUTPUT
echo "→ Other transport files changed, using cached matrix if available"
fi

resolve-transport-parameters:
runs-on: [self-hosted, linux, x64, ephemeral]
outputs:
Expand Down Expand Up @@ -219,45 +180,6 @@ jobs:
echo "DEBUG: should-run-tests output = '${{ steps.resolve.outputs.should-run-tests }}'"
echo "DEBUG: github.event_name = '${{ github.event_name }}'"

check-hole-punch-changes:
if: github.event_name != 'workflow_dispatch'
runs-on: [self-hosted, linux, x64, ephemeral]
outputs:
has-changes: ${{ steps.check.outputs.has-changes }}
impls-yaml-changed: ${{ steps.check.outputs.impls-yaml-changed }}
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Check hole-punch changes in last 24 hours
id: check
shell: bash
run: |
# Check if anything in hole-punch/ changed in last 24 hours
CHANGES=$(git log --oneline --since="24 hours ago" -- "hole-punch/**" | wc -l)

if [ "$CHANGES" -eq 0 ]; then
echo "has-changes=false" >> $GITHUB_OUTPUT
echo "impls-yaml-changed=false" >> $GITHUB_OUTPUT
echo "→ No changes in hole-punch/ in last 24 hours, skipping tests"
exit 0
fi

echo "has-changes=true" >> $GITHUB_OUTPUT

# Check specifically if impls.yaml changed in last 24 hours
IMPLS_CHANGES=$(git log --oneline --since="24 hours ago" -- "hole-punch/impls.yaml" | wc -l)

if [ "$IMPLS_CHANGES" -gt 0 ]; then
echo "impls-yaml-changed=true" >> $GITHUB_OUTPUT
echo "→ hole-punch/impls.yaml changed in last 24h, will force matrix rebuild"
else
echo "impls-yaml-changed=false" >> $GITHUB_OUTPUT
echo "→ Other hole-punch files changed, using cached matrix if available"
fi

resolve-hole-punch-parameters:
runs-on: [self-hosted, linux, x64, ephemeral]
outputs:
Expand Down Expand Up @@ -324,45 +246,6 @@ jobs:
fi
fi

check-perf-changes:
if: github.event_name != 'workflow_dispatch'
runs-on: [self-hosted, linux, x64, ephemeral]
outputs:
has-changes: ${{ steps.check.outputs.has-changes }}
impls-yaml-changed: ${{ steps.check.outputs.impls-yaml-changed }}
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Check perf changes in last 24 hours
id: check
shell: bash
run: |
# Check if anything in perf/ changed in last 24 hours
CHANGES=$(git log --oneline --since="24 hours ago" -- "perf/**" | wc -l)

if [ "$CHANGES" -eq 0 ]; then
echo "has-changes=false" >> $GITHUB_OUTPUT
echo "impls-yaml-changed=false" >> $GITHUB_OUTPUT
echo "→ No changes in perf/ in last 24 hours, skipping tests"
exit 0
fi

echo "has-changes=true" >> $GITHUB_OUTPUT

# Check specifically if impls.yaml changed in last 24 hours
IMPLS_CHANGES=$(git log --oneline --since="24 hours ago" -- "perf/impls.yaml" | wc -l)

if [ "$IMPLS_CHANGES" -gt 0 ]; then
echo "impls-yaml-changed=true" >> $GITHUB_OUTPUT
echo "→ perf/impls.yaml changed in last 24h, will force matrix rebuild"
else
echo "impls-yaml-changed=false" >> $GITHUB_OUTPUT
echo "→ Other perf files changed, using cached matrix if available"
fi

resolve-perf-parameters:
runs-on: [self-hosted, linux, x64, ephemeral]
outputs:
Expand Down
22 changes: 0 additions & 22 deletions .github/workflows/hole-punch-interop-pr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,28 +65,6 @@ concurrency:
cancel-in-progress: true

jobs:
detect-changes:
if: github.event_name != 'workflow_dispatch'
runs-on: [self-hosted, linux, x64, ephemeral]
outputs:
changed-impls: ${{ steps.detect.outputs.changed-impls }}
changed-relays: ${{ steps.detect.outputs.changed-relays }}
changed-routers: ${{ steps.detect.outputs.changed-routers }}
has-changes: ${{ steps.detect.outputs.has-changes }}
impls-yaml-changed: ${{ steps.detect.outputs.impls-yaml-changed }}
impls-folder-changed: ${{ steps.detect.outputs.impls-folder-changed }}
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Detect changed implementations/relays/routers
id: detect
uses: ./.github/actions/detect-changed-impls
with:
test-type: hole-punch

resolve-parameters:
runs-on: [self-hosted, linux, x64, ephemeral]
outputs:
Expand Down
24 changes: 2 additions & 22 deletions .github/workflows/perf-pr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,26 +60,6 @@ concurrency:
cancel-in-progress: true

jobs:
detect-changes:
if: github.event_name != 'workflow_dispatch'
runs-on: [self-hosted, linux, x64, ephemeral]
outputs:
changed-impls: ${{ steps.detect.outputs.changed-impls }}
has-changes: ${{ steps.detect.outputs.has-changes }}
impls-yaml-changed: ${{ steps.detect.outputs.impls-yaml-changed }}
impls-folder-changed: ${{ steps.detect.outputs.impls-folder-changed }}
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Detect changed implementations
id: detect
uses: ./.github/actions/detect-changed-impls
with:
test-type: perf

resolve-parameters:
runs-on: [self-hosted, linux, x64, ephemeral]
outputs:
Expand Down Expand Up @@ -127,8 +107,8 @@ jobs:
# Automatic trigger - use change detection outputs
echo "test-select=${{ steps.detect.outputs.changed-impls }}" >> $GITHUB_OUTPUT
echo "test-ignore=!${{ steps.detect.outputs.changed-impls }}" >> $GITHUB_OUTPUT
echo "baseline-select=" >> $GITHUB_OUTPUT
echo "baseline-ignore=" >> $GITHUB_OUTPUT
echo "baseline-select=${{ steps.detect.outputs.changed-baselines }}" >> $GITHUB_OUTPUT
echo "baseline-ignore=!${{ steps.detect.outputs.changed-baselines }}" >> $GITHUB_OUTPUT
echo "iterations=10" >> $GITHUB_OUTPUT
echo "force-matrix-rebuild=${{ steps.detect.outputs.impls-yaml-changed }}" >> $GITHUB_OUTPUT
# Force image rebuild if impls.yaml changed (commit SHA changes require new images)
Expand Down
20 changes: 0 additions & 20 deletions .github/workflows/transport-interop-pr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,26 +45,6 @@ concurrency:
cancel-in-progress: true

jobs:
detect-changes:
if: github.event_name != 'workflow_dispatch'
runs-on: [self-hosted, linux, x64, ephemeral]
outputs:
changed-impls: ${{ steps.detect.outputs.changed-impls }}
has-changes: ${{ steps.detect.outputs.has-changes }}
impls-yaml-changed: ${{ steps.detect.outputs.impls-yaml-changed }}
impls-folder-changed: ${{ steps.detect.outputs.impls-folder-changed }}
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Detect changed implementations
id: detect
uses: ./.github/actions/detect-changed-impls
with:
test-type: transport

resolve-parameters:
runs-on: [self-hosted, linux, x64, ephemeral]
outputs:
Expand Down
Loading
Loading