Conversation
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
There was a problem hiding this comment.
Pull request overview
This PR adds a new GitHub Actions workflow for building and publishing nightly wheels of fVDB to a custom S3-based pip index. The workflow runs daily at 8am UTC (or on manual dispatch), checks for new commits in the last 24 hours, builds wheels for multiple Python versions (3.10-3.13) with PyTorch 2.8 and CUDA 12.8, and publishes them to an S3 bucket following the PEP 503 Simple Repository API specification.
Changes:
- Adds automated nightly build and publish workflow with commit-gating, EC2 runner provisioning, wheel building with manylinux repair, S3 publishing with 30-day pruning, and proper PEP 503 index generation
- Documents nightly builds in README with installation instructions for the custom pip index
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
.github/workflows/nightly-publish.yml |
Complete nightly workflow with commit check gate, EC2 runner lifecycle management, wheel building across Python matrix, manylinux compliance via auditwheel, and S3 publishing with index generation |
README.md |
Documents nightly build installation instructions with pip command and version format explanation |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| DELETED=0 | ||
| aws s3 ls "s3://${S3_BUCKET}/${SIMPLE_PREFIX}/${{ steps.proj.outputs.normalized }}/" 2>/dev/null | while read -r line; do | ||
| DATE_PART=$(echo "$line" | awk '{print $1}') | ||
| FILE_PART=$(echo "$line" | awk '{print $4}') | ||
| if [[ "$FILE_PART" == *.whl ]] && [[ "$DATE_PART" < "$CUTOFF" ]]; then | ||
| echo "Removing: $FILE_PART (uploaded $DATE_PART)" | ||
| aws s3 rm "s3://${S3_BUCKET}/${SIMPLE_PREFIX}/${{ steps.proj.outputs.normalized }}/$FILE_PART" --only-show-errors | ||
| DELETED=$((DELETED + 1)) | ||
| fi | ||
| done | ||
| echo "Pruned $DELETED old nightly wheel(s)" |
There was a problem hiding this comment.
The pruning logic has a potential issue: the DELETED counter increment happens inside a subshell created by the pipe, so it will not affect the final echo statement. The counter will always show 0 regardless of how many files are deleted. Consider restructuring to avoid the pipe subshell or use process substitution instead.
| run: | | ||
| TORCH_TAG="$(echo "${{ matrix.torch-version }}" | tr -d '.')" | ||
| CUDA_TAG="$(echo "${{ matrix.cuda-version }}" | tr -d '.')" | ||
| NIGHTLY_VERSION="0.0.0.dev$(date -u '+%Y%m%d')+pt${TORCH_TAG}.cu${CUDA_TAG}" |
There was a problem hiding this comment.
The version format uses a local build identifier after the plus sign (e.g., +pt28.cu128). According to PEP 440, local version identifiers are not allowed in versions uploaded to PyPI and should not be used in pre-release versions on public indexes. While this might work for a custom S3 index, it may cause issues with pip's version comparison and compatibility checks. Consider using a format like "0.0.0.dev20250224.pt28cu128" without the plus sign to ensure broader compatibility.
| NIGHTLY_VERSION="0.0.0.dev$(date -u '+%Y%m%d')+pt${TORCH_TAG}.cu${CUDA_TAG}" | |
| NIGHTLY_VERSION="0.0.0.dev$(date -u '+%Y%m%d').pt${TORCH_TAG}cu${CUDA_TAG}" |
| pip install --pre fvdb-core --extra-index-url https://fvdb-packages.s3.us-east-2.amazonaws.com/simple-nightly/ | ||
| ``` | ||
|
|
||
| Nightly builds use a synthetic version (`0.0.0.devYYYYMMDD`) and require `--pre`. |
There was a problem hiding this comment.
The documentation describes the nightly version format as "0.0.0.devYYYYMMDD", but the actual implementation in the workflow generates versions like "0.0.0.devYYYYMMDD+pt28.cu128" (with PyTorch and CUDA tags). The documentation should be updated to reflect the actual format, or provide a note that different builds may include additional build identifiers for PyTorch and CUDA versions.
| Nightly builds use a synthetic version (`0.0.0.devYYYYMMDD`) and require `--pre`. | |
| Nightly builds use a synthetic base version (`0.0.0.devYYYYMMDD`), and some builds include additional build identifiers (for example, `+pt28.cu128` for specific PyTorch/CUDA combinations). These builds require `--pre`. |
No description provided.