Skip to content

Commit 333b05a

Browse files
oberstetoberstet
committed
Upload verification files to GitHub Releases + fix PEP 668 issue
This commit completes the end-to-end chain-of-custody security architecture and fixes a PEP 668 error that was blocking macOS builds. ## Changes: 1. **Upload verification files to GitHub Releases** (Issue #1716) - Modified release.yml to copy verification files with prefixes: * wheels-CHECKSUMS.sha256, wheels-VALIDATION.txt * docker-CHECKSUMS.sha256, docker-VALIDATION.txt, docker-build-info.txt * arm64-CHECKSUMS.sha256, arm64-VALIDATION.txt, arm64-build-info.txt * autobahn-*.verify.txt (source distribution verification) - Verification files provide transparency for supply chain integrity - Files uploaded to GitHub Releases alongside wheels - Files excluded from PyPI uploads (PyPI only accepts .whl/.tar.gz) 2. **Fix PEP 668 externally-managed-environment error** - Added --break-system-packages flag to all pip install commands - Applies to: wheels.yml, wheels-docker.yml, wheels-arm64.yml, release.yml - Safe for ephemeral CI runners that are destroyed after each run - Fixes error: "externally-managed-environment" on macOS runners ## Security Architecture (Complete): Build → Validate → SHA256 → Upload Artifacts ↓ ↓ ↓ ↓ .whl VALIDATION CHECKSUMS (together) ↓ Download ↓ Re-verify SHA256 ↓ GitHub Release + PyPI (with verification (wheels only) files for audit) All three wheel-building workflows now have complete chain-of-custody: - wheels (Linux/macOS/Windows) - wheels-docker (manylinux x86_64) - wheels-arm64 (manylinux ARM64)
1 parent 3a0ac63 commit 333b05a

File tree

4 files changed

+101
-24
lines changed

4 files changed

+101
-24
lines changed

‎.github/workflows/release.yml‎

Lines changed: 86 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -520,22 +520,51 @@ jobs:
520520
echo "==> Consolidating all artifacts into unified release directory..."
521521
mkdir -p release-artifacts
522522
523-
# Copy wheels from wheels workflow
523+
# Copy wheels and verification files from wheels workflow
524524
if [ -d "dist" ]; then
525525
echo "Copying wheels workflow artifacts..."
526526
find dist -type f \( -name "*.whl" -o -name "*.tar.gz" \) -exec cp {} release-artifacts/ \;
527+
# Copy verification escort files with "wheels-" prefix to avoid naming collisions
528+
if [ -f "dist/CHECKSUMS.sha256" ]; then
529+
cp dist/CHECKSUMS.sha256 release-artifacts/wheels-CHECKSUMS.sha256
530+
fi
531+
if [ -f "dist/VALIDATION.txt" ]; then
532+
cp dist/VALIDATION.txt release-artifacts/wheels-VALIDATION.txt
533+
fi
534+
# Copy source distribution verification reports (already have unique names)
535+
find dist -type f -name "*.verify.txt" -exec cp {} release-artifacts/ \; 2>/dev/null || true
527536
fi
528537
529-
# Copy wheels from wheels-docker workflow
538+
# Copy wheels and verification files from wheels-docker workflow
530539
if [ -d "wheelhouse" ]; then
531540
echo "Copying wheels-docker workflow artifacts..."
532541
find wheelhouse -type f \( -name "*.whl" -o -name "*.tar.gz" \) -exec cp {} release-artifacts/ \;
542+
# Copy verification escort files with "docker-" prefix to avoid naming collisions
543+
if [ -f "wheelhouse/CHECKSUMS.sha256" ]; then
544+
cp wheelhouse/CHECKSUMS.sha256 release-artifacts/docker-CHECKSUMS.sha256
545+
fi
546+
if [ -f "wheelhouse/VALIDATION.txt" ]; then
547+
cp wheelhouse/VALIDATION.txt release-artifacts/docker-VALIDATION.txt
548+
fi
549+
if [ -f "wheelhouse/build-info.txt" ]; then
550+
cp wheelhouse/build-info.txt release-artifacts/docker-build-info.txt
551+
fi
533552
fi
534553
535-
# Copy ARM64 wheels from wheels-arm64 workflow
554+
# Copy ARM64 wheels and verification files from wheels-arm64 workflow
536555
if [ -d "wheelhouse-arm64" ]; then
537556
echo "Copying wheels-arm64 workflow artifacts..."
538557
find wheelhouse-arm64 -type f \( -name "*.whl" -o -name "*.tar.gz" \) -exec cp {} release-artifacts/ \;
558+
# Copy verification escort files with "arm64-" prefix to avoid naming collisions
559+
if [ -f "wheelhouse-arm64/CHECKSUMS.sha256" ]; then
560+
cp wheelhouse-arm64/CHECKSUMS.sha256 release-artifacts/arm64-CHECKSUMS.sha256
561+
fi
562+
if [ -f "wheelhouse-arm64/VALIDATION.txt" ]; then
563+
cp wheelhouse-arm64/VALIDATION.txt release-artifacts/arm64-VALIDATION.txt
564+
fi
565+
if [ -f "wheelhouse-arm64/build-info.txt" ]; then
566+
cp wheelhouse-arm64/build-info.txt release-artifacts/arm64-build-info.txt
567+
fi
539568
fi
540569
541570
# Copy wstest conformance results
@@ -564,6 +593,11 @@ jobs:
564593
echo ""
565594
echo "Wheels: $(find release-artifacts -name "*.whl" | wc -l)"
566595
echo "Source dists: $(find release-artifacts -name "*.tar.gz" ! -name "flatbuffers-schema.tar.gz" ! -name "autobahn-python-websocket-conformance-*.tar.gz" | wc -l)"
596+
echo "Verification files (chain-of-custody):"
597+
echo " - SHA256 checksums: $(find release-artifacts -name "*CHECKSUMS.sha256" | wc -l)"
598+
echo " - Build validation: $(find release-artifacts -name "*VALIDATION.txt" | wc -l)"
599+
echo " - Source verification: $(find release-artifacts -name "*.verify.txt" | wc -l)"
600+
echo " - Build metadata: $(find release-artifacts -name "*build-info.txt" | wc -l)"
567601
echo "Wstest reports: $(find release-artifacts -name "*wstest*" | wc -l)"
568602
echo "FlatBuffers schema: $(ls release-artifacts/flatbuffers-schema.tar.gz 2>/dev/null && echo 'packaged' || echo 'not found')"
569603
echo "Conformance reports: $(ls release-artifacts/autobahn-python-websocket-conformance-*.tar.gz 2>/dev/null && echo 'packaged' || echo 'not found')"
@@ -580,8 +614,9 @@ jobs:
580614
echo ""
581615
echo "Installing twine for validation..."
582616
# Install both packaging and twine from master for PEP 639 (Core Metadata 2.4) support
583-
python3 -m pip install git+https://github.com/pypa/packaging.git
584-
python3 -m pip install git+https://github.com/pypa/twine.git
617+
# Use --break-system-packages for consistency (safe in CI)
618+
python3 -m pip install --break-system-packages git+https://github.com/pypa/packaging.git
619+
python3 -m pip install --break-system-packages git+https://github.com/pypa/twine.git
585620
echo ""
586621
587622
echo "==> Validation environment:"
@@ -1146,22 +1181,51 @@ jobs:
11461181
echo "==> Consolidating all artifacts into unified release directory..."
11471182
mkdir -p release-artifacts
11481183
1149-
# Copy wheels from wheels workflow
1184+
# Copy wheels and verification files from wheels workflow
11501185
if [ -d "dist" ]; then
11511186
echo "Copying wheels workflow artifacts..."
11521187
find dist -type f \( -name "*.whl" -o -name "*.tar.gz" \) -exec cp {} release-artifacts/ \;
1188+
# Copy verification escort files with "wheels-" prefix to avoid naming collisions
1189+
if [ -f "dist/CHECKSUMS.sha256" ]; then
1190+
cp dist/CHECKSUMS.sha256 release-artifacts/wheels-CHECKSUMS.sha256
1191+
fi
1192+
if [ -f "dist/VALIDATION.txt" ]; then
1193+
cp dist/VALIDATION.txt release-artifacts/wheels-VALIDATION.txt
1194+
fi
1195+
# Copy source distribution verification reports (already have unique names)
1196+
find dist -type f -name "*.verify.txt" -exec cp {} release-artifacts/ \; 2>/dev/null || true
11531197
fi
11541198
1155-
# Copy wheels from wheels-docker workflow
1199+
# Copy wheels and verification files from wheels-docker workflow
11561200
if [ -d "wheelhouse" ]; then
11571201
echo "Copying wheels-docker workflow artifacts..."
11581202
find wheelhouse -type f \( -name "*.whl" -o -name "*.tar.gz" \) -exec cp {} release-artifacts/ \;
1203+
# Copy verification escort files with "docker-" prefix to avoid naming collisions
1204+
if [ -f "wheelhouse/CHECKSUMS.sha256" ]; then
1205+
cp wheelhouse/CHECKSUMS.sha256 release-artifacts/docker-CHECKSUMS.sha256
1206+
fi
1207+
if [ -f "wheelhouse/VALIDATION.txt" ]; then
1208+
cp wheelhouse/VALIDATION.txt release-artifacts/docker-VALIDATION.txt
1209+
fi
1210+
if [ -f "wheelhouse/build-info.txt" ]; then
1211+
cp wheelhouse/build-info.txt release-artifacts/docker-build-info.txt
1212+
fi
11591213
fi
11601214
1161-
# Copy ARM64 wheels from wheels-arm64 workflow
1215+
# Copy ARM64 wheels and verification files from wheels-arm64 workflow
11621216
if [ -d "wheelhouse-arm64" ]; then
11631217
echo "Copying wheels-arm64 workflow artifacts..."
11641218
find wheelhouse-arm64 -type f \( -name "*.whl" -o -name "*.tar.gz" \) -exec cp {} release-artifacts/ \;
1219+
# Copy verification escort files with "arm64-" prefix to avoid naming collisions
1220+
if [ -f "wheelhouse-arm64/CHECKSUMS.sha256" ]; then
1221+
cp wheelhouse-arm64/CHECKSUMS.sha256 release-artifacts/arm64-CHECKSUMS.sha256
1222+
fi
1223+
if [ -f "wheelhouse-arm64/VALIDATION.txt" ]; then
1224+
cp wheelhouse-arm64/VALIDATION.txt release-artifacts/arm64-VALIDATION.txt
1225+
fi
1226+
if [ -f "wheelhouse-arm64/build-info.txt" ]; then
1227+
cp wheelhouse-arm64/build-info.txt release-artifacts/arm64-build-info.txt
1228+
fi
11651229
fi
11661230
11671231
# Copy wstest conformance results
@@ -1190,6 +1254,11 @@ jobs:
11901254
echo ""
11911255
echo "Wheels: $(find release-artifacts -name "*.whl" | wc -l)"
11921256
echo "Source dists: $(find release-artifacts -name "*.tar.gz" ! -name "flatbuffers-schema.tar.gz" ! -name "autobahn-python-websocket-conformance-*.tar.gz" | wc -l)"
1257+
echo "Verification files (chain-of-custody):"
1258+
echo " - SHA256 checksums: $(find release-artifacts -name "*CHECKSUMS.sha256" | wc -l)"
1259+
echo " - Build validation: $(find release-artifacts -name "*VALIDATION.txt" | wc -l)"
1260+
echo " - Source verification: $(find release-artifacts -name "*.verify.txt" | wc -l)"
1261+
echo " - Build metadata: $(find release-artifacts -name "*build-info.txt" | wc -l)"
11931262
echo "Wstest reports: $(find release-artifacts -name "*wstest*" | wc -l)"
11941263
echo "FlatBuffers schema: $(ls release-artifacts/flatbuffers-schema.tar.gz 2>/dev/null && echo 'packaged' || echo 'not found')"
11951264
echo "Conformance reports: $(ls release-artifacts/autobahn-python-websocket-conformance-*.tar.gz 2>/dev/null && echo 'packaged' || echo 'not found')"
@@ -1206,8 +1275,9 @@ jobs:
12061275
echo ""
12071276
echo "Installing twine for validation..."
12081277
# Install both packaging and twine from master for PEP 639 (Core Metadata 2.4) support
1209-
python3 -m pip install git+https://github.com/pypa/packaging.git
1210-
python3 -m pip install git+https://github.com/pypa/twine.git
1278+
# Use --break-system-packages for consistency (safe in CI)
1279+
python3 -m pip install --break-system-packages git+https://github.com/pypa/packaging.git
1280+
python3 -m pip install --break-system-packages git+https://github.com/pypa/twine.git
12111281
echo ""
12121282
12131283
echo "==> Validation environment:"
@@ -1792,8 +1862,9 @@ jobs:
17921862
echo "Last chance to catch corrupted packages before PyPI upload."
17931863
echo ""
17941864
# Install both packaging and twine from master for PEP 639 (Core Metadata 2.4) support
1795-
python3 -m pip install git+https://github.com/pypa/packaging.git
1796-
python3 -m pip install git+https://github.com/pypa/twine.git
1865+
# Use --break-system-packages for consistency (safe in CI)
1866+
python3 -m pip install --break-system-packages git+https://github.com/pypa/packaging.git
1867+
python3 -m pip install --break-system-packages git+https://github.com/pypa/twine.git
17971868
echo ""
17981869
17991870
echo "==> Validation environment:"
@@ -1885,8 +1956,9 @@ jobs:
18851956
run: |
18861957
echo "==> Publishing to PyPI using twine from master..."
18871958
# Install bleeding-edge packaging and twine for PEP 639 support
1888-
python3 -m pip install git+https://github.com/pypa/packaging.git
1889-
python3 -m pip install git+https://github.com/pypa/twine.git
1959+
# Use --break-system-packages for consistency (safe in CI)
1960+
python3 -m pip install --break-system-packages git+https://github.com/pypa/packaging.git
1961+
python3 -m pip install --break-system-packages git+https://github.com/pypa/twine.git
18901962
18911963
echo "Upload environment:"
18921964
echo "twine: $(twine --version)"

‎.github/workflows/wheels-arm64.yml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -201,8 +201,9 @@ jobs:
201201
echo ""
202202
echo "Installing twine for validation..."
203203
# Install both packaging and twine from master for PEP 639 (Core Metadata 2.4) support
204-
python3 -m pip install git+https://github.com/pypa/packaging.git
205-
python3 -m pip install git+https://github.com/pypa/twine.git
204+
# Use --break-system-packages for consistency (safe in CI)
205+
python3 -m pip install --break-system-packages git+https://github.com/pypa/packaging.git
206+
python3 -m pip install --break-system-packages git+https://github.com/pypa/twine.git
206207
echo ""
207208
208209
echo "==> Validation environment:"

‎.github/workflows/wheels-docker.yml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -257,8 +257,9 @@ jobs:
257257
# Ensure pip is available for the Python being used
258258
python3 -m ensurepip --upgrade 2>/dev/null || true
259259
# Install both packaging and twine from master for PEP 639 (Core Metadata 2.4) support
260-
python3 -m pip install git+https://github.com/pypa/packaging.git
261-
python3 -m pip install git+https://github.com/pypa/twine.git
260+
# Use --break-system-packages for consistency (safe in CI containers)
261+
python3 -m pip install --break-system-packages git+https://github.com/pypa/packaging.git
262+
python3 -m pip install --break-system-packages git+https://github.com/pypa/twine.git
262263
echo ""
263264
264265
echo "==> Validation environment:"

‎.github/workflows/wheels.yml‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -216,8 +216,9 @@ jobs:
216216
echo "======================================================================"
217217
echo ""
218218
echo "Installing twine for validation..."
219-
python3 -m pip install git+https://github.com/pypa/packaging.git
220-
python3 -m pip install git+https://github.com/pypa/twine.git
219+
# Use --break-system-packages for consistency (safe in CI)
220+
python3 -m pip install --break-system-packages git+https://github.com/pypa/packaging.git
221+
python3 -m pip install --break-system-packages git+https://github.com/pypa/twine.git
221222
echo ""
222223
223224
echo "==> Validation environment:"
@@ -391,8 +392,9 @@ jobs:
391392
echo "==> Validating Wheel Integrity (macOS)"
392393
echo "======================================================================"
393394
echo ""
394-
python3 -m pip install git+https://github.com/pypa/packaging.git
395-
python3 -m pip install git+https://github.com/pypa/twine.git
395+
# Use --break-system-packages since this is an ephemeral CI runner
396+
python3 -m pip install --break-system-packages git+https://github.com/pypa/packaging.git
397+
python3 -m pip install --break-system-packages git+https://github.com/pypa/twine.git
396398
echo ""
397399
398400
VALIDATION_FILE="dist/VALIDATION.txt"
@@ -488,8 +490,9 @@ jobs:
488490
Write-Host "==> Validating Wheel Integrity (Windows)"
489491
Write-Host "======================================================================"
490492
Write-Host ""
491-
python -m pip install git+https://github.com/pypa/packaging.git
492-
python -m pip install git+https://github.com/pypa/twine.git
493+
# Use --break-system-packages for consistency (safe in CI)
494+
python -m pip install --break-system-packages git+https://github.com/pypa/packaging.git
495+
python -m pip install --break-system-packages git+https://github.com/pypa/twine.git
493496
Write-Host ""
494497
495498
$validationFile = "dist\VALIDATION.txt"

0 commit comments

Comments
 (0)