Skip to content

Add comprehensive GitHub Copilot instructions for PyPI publish action development #375

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 6 commits into
base: unstable/v1
Choose a base branch
from
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 143 additions & 0 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
# PyPI Publish GitHub Action

PyPI Publish is a Docker-based GitHub Action for uploading Python distribution packages to PyPI and TestPyPI. The action supports both traditional API tokens and modern Trusted Publishing (OIDC) authentication, with optional PEP 740 attestation generation.

**Always reference these instructions first and fallback to search or bash commands only when you encounter unexpected information that does not match the info here.**

## Working Effectively

### Bootstrap, Build, and Test the Repository
- Check Python version: `python3 --version` (should be 3.12+)
- Check Docker availability: `which docker`
- Test Python syntax validation: `python3 -m py_compile *.py`
- Test shell script syntax: `bash -n twine-upload.sh`
- Install basic validation tools: `pip3 install --user pre-commit` (may fail due to network limitations)

### Docker Build Process
- **CRITICAL**: Docker build takes 10-20 minutes to complete. NEVER CANCEL. Set timeout to 30+ minutes.
- Build Docker image: `time docker build . -t pypi-publish-test`
- **NOTE**: In sandboxed environments, Docker build may fail due to SSL certificate or network timeouts. This is a known limitation, not a code issue.

### Testing and Validation
- **NEVER CANCEL**: Smoke tests take 2 minutes. Wait for completion.
- Run Python script validation:
- `mkdir -p test-dist && echo "test" > test-dist/test.txt`
- `python3 print-hash.py test-dist` (should show SHA256/MD5/BLAKE2-256 hashes)
- `python3 print-pkg-names.py test-dist` (should complete without error)
- Check workflow syntax: `find .github/workflows/ -name "*.yml" -exec echo "Checking {}" \; -exec yamllint {} \;`

### Linting and Code Quality
- Pre-commit hooks setup: `pre-commit run --all-files` (may fail due to network limitations)
- **Manual validation when pre-commit fails**:
- Python syntax: `python3 -m py_compile *.py`
- Shell script syntax: `bash -n twine-upload.sh`
- YAML syntax: `python3 -c "import yaml; yaml.safe_load(open('.github/workflows/build-and-push-docker-image.yml'))"`

## Manual Validation Scenarios

### Always Test After Making Changes
1. **Python Script Changes**: Run `python3 -m py_compile <changed_file>.py` to verify syntax
2. **Shell Script Changes**: Run `bash -n twine-upload.sh` to verify syntax
3. **Docker Changes**: Test build with `docker build . -t test-build` (expect 10-20 minute build time)
4. **Action YAML Changes**: Validate syntax with `yamllint action.yml`

### Functional Testing Scenarios
- **Hash Validation**: Always test `python3 print-hash.py <test-directory>` after changes to hash calculation
- **OIDC Flow**: Cannot be fully tested locally - relies on GitHub's OIDC token exchange
- **Attestation Generation**: Cannot be fully tested without proper sigstore setup

## Timeout Values and Build Expectations

- **Docker Build**: 10-20 minutes (set timeout to 30+ minutes, NEVER CANCEL)
- **Smoke Tests**: 2 minutes (set timeout to 5+ minutes)
- **Pre-commit All Files**: 2-5 minutes (may fail due to network issues)
- **Python Syntax Check**: < 30 seconds
- **Shell Script Validation**: < 10 seconds

## Common Build and Network Issues

### Known Limitations
- **Docker build failures**: SSL certificate verification errors or network timeouts are common in sandboxed environments
- **pip install failures**: PyPI connectivity issues may prevent package installation
- **pre-commit failures**: Network timeouts during hook installation are expected in restricted environments

### Workarounds
- Use manual validation commands when automated tools fail
- Test individual Python files instead of full dependency installation
- Rely on GitHub Actions CI for full integration testing

## Key Repository Structure

### Root Directory Contents
```
.
├── .github/
│ └── workflows/ # CI/CD workflows
├── requirements/ # Python dependencies
├── action.yml # Main GitHub Action definition
├── Dockerfile # Container build instructions
├── twine-upload.sh # Main upload orchestration script
├── oidc-exchange.py # OIDC token exchange for Trusted Publishing
├── attestations.py # PEP 740 attestation generation
├── print-hash.py # File hash calculation and display
├── print-pkg-names.py # Package name extraction from distributions
├── create-docker-action.py # Dynamic Docker action creation
└── .pre-commit-config.yaml # Linting and code quality configuration
```

### Critical Files to Know
- **action.yml**: Defines all input parameters and action metadata - edit when adding new features
- **twine-upload.sh**: Core logic for upload process - main place for upload behavior changes
- **Dockerfile**: Container build process - edit when changing Python version or dependencies
- **requirements/runtime.txt**: Pinned dependency versions - autogenerated, do not edit directly
- **requirements/runtime.in**: High-level dependency specifications - edit to add new dependencies

### Dependencies and Requirements
- **Python Version**: 3.13 (in Docker), 3.12+ for local development
- **Key Dependencies**: twine >=6.1, id ~=1.0, pypi-attestations ~=0.0.26, sigstore ~=3.5.1
- **Dependency Management**: Uses pip-tools - run `pip-compile` to regenerate requirement files

### Workflow Patterns
- **build-and-push-docker-image.yml**: Builds and publishes the Docker container (10 min timeout)
- **reusable-smoke-test.yml**: Comprehensive functionality testing (2 min timeout)
- Smoke tests create stub packages and test upload to a devpi test server
- Tests run on ubuntu-22.04 and ubuntu-24.04

## Development Best Practices

### Always Validate Before Committing
- Run `python3 -m py_compile *.py` to check Python syntax
- Run `bash -n twine-upload.sh` to check shell script syntax
- Test any changes with local test scenarios shown above
- If pre-commit works: `pre-commit run --all-files`

### When Modifying Core Logic
- **Shell Script Changes**: Always test syntax and review bash error handling (set -Eeuo pipefail)
- **Python Changes**: Test with both valid and invalid inputs
- **Docker Changes**: Expect long build times and potential network failures in CI
- **Action Inputs**: Update both action.yml and twine-upload.sh if adding new parameters

### Security Considerations
- This action handles sensitive credentials (API tokens, OIDC tokens)
- Always validate input sanitization in twine-upload.sh
- OIDC exchange in oidc-exchange.py is security-critical code
- Attestation generation involves cryptographic signing

## Troubleshooting

### "Docker build fails with SSL errors"
- This is expected in sandboxed environments
- The build works in proper CI environments
- Use manual validation commands instead

### "pre-commit installation fails"
- Network connectivity issue in sandboxed environment
- Use individual linting tools: `python3 -m py_compile`, `bash -n`, `yamllint`

### "Cannot test OIDC flow locally"
- OIDC requires GitHub's runtime environment
- Test syntax and structure, but rely on CI for integration testing

### "pip install timeouts"
- Common in restricted network environments
- Focus on syntax validation and structure testing instead