Skip to content

Commit 33c99f6

Browse files
RonnyPfannschmidtRonnyPfannschmidt
committed
document recommend ci additions for pypi/test-pypi uploads
closes #311
1 parent 59b5b92 commit 33c99f6

File tree

2 files changed

+296
-1
lines changed

2 files changed

+296
-1
lines changed

‎CHANGELOG.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@
1010
- fix #1059: add `SETUPTOOLS_SCM_PRETEND_METADATA` environment variable to override individual ScmVersion fields
1111
- add `scm` parameter support to `get_version()` function for nested SCM configuration
1212
- fix #987: expand documentation on git archival files and add cli tools for good defaults
13+
- fix #311: document github/gitlab ci pipelines that enable auto-upload to test-pypi/pypi
14+
1315

1416
### Changed
1517

‎docs/integrations.md

Lines changed: 294 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,4 +47,297 @@ build:
4747
- export SETUPTOOLS_SCM_OVERRIDES_FOR_${READTHEDOCS_PROJECT//-/_}='{scm.git.pre_parse="fail_on_shallow"}'
4848
```
4949

50-
This configuration uses the `SETUPTOOLS_SCM_OVERRIDES_FOR_${NORMALIZED_DIST_NAME}` environment variable to override the `scm.git.pre_parse` setting specifically for your project when building on ReadTheDocs, forcing setuptools-scm to fail with a clear error if the repository is shallow.
50+
This configuration uses the `SETUPTOOLS_SCM_OVERRIDES_FOR_${NORMALIZED_DIST_NAME}` environment variable to override the `scm.git.pre_parse` setting specifically for your project when building on ReadTheDocs, forcing setuptools-scm to fail with a clear error if the repository is shallow.
51+
52+
## CI/CD and Package Publishing
53+
54+
### Publishing to PyPI from CI/CD
55+
56+
When publishing packages to PyPI or test-PyPI from CI/CD pipelines, you often need to remove local version components that are not allowed on public package indexes according to [PEP 440](https://peps.python.org/pep-0440/#local-version-identifiers).
57+
58+
setuptools-scm provides the `no-local-version` local scheme and environment variable overrides to handle this scenario cleanly.
59+
60+
#### The Problem
61+
62+
By default, setuptools-scm generates version numbers like:
63+
- `1.2.3.dev4+g1a2b3c4d5` (development version with git hash)
64+
- `1.2.3+dirty` (dirty working directory)
65+
66+
These local version components (`+g1a2b3c4d5`, `+dirty`) prevent uploading to PyPI.
67+
68+
#### The Solution
69+
70+
Use the `SETUPTOOLS_SCM_OVERRIDES_FOR_${NORMALIZED_DIST_NAME}` environment variable to override the `local_scheme` to `no-local-version` when building for upload to PyPI.
71+
72+
### GitHub Actions Example
73+
74+
Here's a complete GitHub Actions workflow that:
75+
- Runs tests on all branches
76+
- Uploads development versions to test-PyPI from feature branches
77+
- Uploads development versions to PyPI from the main branch (with no-local-version)
78+
- Uploads tagged releases to PyPI (using exact tag versions)
79+
80+
```yaml title=".github/workflows/ci.yml"
81+
name: CI/CD
82+
83+
on:
84+
push:
85+
branches: ["main", "develop"]
86+
pull_request:
87+
branches: ["main", "develop"]
88+
release:
89+
types: [published]
90+
91+
jobs:
92+
test:
93+
runs-on: ubuntu-latest
94+
strategy:
95+
matrix:
96+
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
97+
98+
steps:
99+
- uses: actions/checkout@v4
100+
with:
101+
# Fetch full history for setuptools-scm
102+
fetch-depth: 0
103+
104+
- name: Set up Python ${{ matrix.python-version }}
105+
uses: actions/setup-python@v4
106+
with:
107+
python-version: ${{ matrix.python-version }}
108+
109+
- name: Install dependencies
110+
run: |
111+
python -m pip install --upgrade pip
112+
pip install build pytest
113+
pip install -e .
114+
115+
- name: Run tests
116+
run: pytest
117+
118+
publish-test-pypi:
119+
needs: test
120+
runs-on: ubuntu-latest
121+
if: github.event_name == 'push' && github.ref != 'refs/heads/main'
122+
env:
123+
# Replace MYPACKAGE with your actual package name (normalized)
124+
# For package "my-awesome.package", use "MY_AWESOME_PACKAGE"
125+
SETUPTOOLS_SCM_OVERRIDES_FOR_MYPACKAGE: '{"local_scheme": "no-local-version"}'
126+
127+
steps:
128+
- uses: actions/checkout@v4
129+
with:
130+
fetch-depth: 0
131+
132+
- name: Set up Python
133+
uses: actions/setup-python@v4
134+
with:
135+
python-version: "3.11"
136+
137+
- name: Install build dependencies
138+
run: |
139+
python -m pip install --upgrade pip
140+
pip install build twine
141+
142+
- name: Build package
143+
run: python -m build
144+
145+
- name: Upload to test-PyPI
146+
uses: pypa/gh-action-pypi-publish@release/v1
147+
with:
148+
repository-url: https://test.pypi.org/legacy/
149+
password: ${{ secrets.TEST_PYPI_API_TOKEN }}
150+
151+
publish-pypi:
152+
needs: test
153+
runs-on: ubuntu-latest
154+
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
155+
env:
156+
# Replace MYPACKAGE with your actual package name (normalized)
157+
SETUPTOOLS_SCM_OVERRIDES_FOR_MYPACKAGE: '{"local_scheme": "no-local-version"}'
158+
159+
steps:
160+
- uses: actions/checkout@v4
161+
with:
162+
fetch-depth: 0
163+
164+
- name: Set up Python
165+
uses: actions/setup-python@v4
166+
with:
167+
python-version: "3.11"
168+
169+
- name: Install build dependencies
170+
run: |
171+
python -m pip install --upgrade pip
172+
pip install build twine
173+
174+
- name: Build package
175+
run: python -m build
176+
177+
- name: Upload to PyPI
178+
uses: pypa/gh-action-pypi-publish@release/v1
179+
with:
180+
password: ${{ secrets.PYPI_API_TOKEN }}
181+
182+
publish-release:
183+
needs: test
184+
runs-on: ubuntu-latest
185+
if: github.event_name == 'release'
186+
187+
steps:
188+
- uses: actions/checkout@v4
189+
with:
190+
fetch-depth: 0
191+
192+
- name: Set up Python
193+
uses: actions/setup-python@v4
194+
with:
195+
python-version: "3.11"
196+
197+
- name: Install build dependencies
198+
run: |
199+
python -m pip install --upgrade pip
200+
pip install build twine
201+
202+
- name: Build package
203+
run: python -m build
204+
205+
- name: Upload to PyPI
206+
uses: pypa/gh-action-pypi-publish@release/v1
207+
with:
208+
password: ${{ secrets.PYPI_API_TOKEN }}
209+
```
210+
211+
### GitLab CI Example
212+
213+
Here's an equivalent GitLab CI configuration:
214+
215+
```yaml title=".gitlab-ci.yml"
216+
stages:
217+
- test
218+
- publish
219+
220+
variables:
221+
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
222+
223+
cache:
224+
paths:
225+
- .cache/pip/
226+
227+
before_script:
228+
- python -m pip install --upgrade pip
229+
230+
test:
231+
stage: test
232+
image: python:3.11
233+
script:
234+
- pip install build pytest
235+
- pip install -e .
236+
- pytest
237+
parallel:
238+
matrix:
239+
- PYTHON_VERSION: ["3.8", "3.9", "3.10", "3.11", "3.12"]
240+
image: python:${PYTHON_VERSION}
241+
242+
publish-test-pypi:
243+
stage: publish
244+
image: python:3.11
245+
variables:
246+
TWINE_USERNAME: __token__
247+
TWINE_PASSWORD: $TEST_PYPI_API_TOKEN
248+
# Replace MYPACKAGE with your actual package name (normalized)
249+
SETUPTOOLS_SCM_OVERRIDES_FOR_MYPACKAGE: '{"local_scheme": "no-local-version"}'
250+
script:
251+
- pip install build twine
252+
- python -m build
253+
- twine upload --repository testpypi dist/*
254+
rules:
255+
- if: $CI_COMMIT_BRANCH != "main" && $CI_PIPELINE_SOURCE == "push"
256+
257+
publish-pypi:
258+
stage: publish
259+
image: python:3.11
260+
variables:
261+
TWINE_USERNAME: __token__
262+
TWINE_PASSWORD: $PYPI_API_TOKEN
263+
# Replace MYPACKAGE with your actual package name (normalized)
264+
SETUPTOOLS_SCM_OVERRIDES_FOR_MYPACKAGE: '{"local_scheme": "no-local-version"}'
265+
script:
266+
- pip install build twine
267+
- python -m build
268+
- twine upload dist/*
269+
rules:
270+
- if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
271+
272+
publish-release:
273+
stage: publish
274+
image: python:3.11
275+
variables:
276+
TWINE_USERNAME: __token__
277+
TWINE_PASSWORD: $PYPI_API_TOKEN
278+
script:
279+
- pip install build twine
280+
- python -m build
281+
- twine upload dist/*
282+
rules:
283+
- if: $CI_COMMIT_TAG
284+
```
285+
286+
### Configuration Details
287+
288+
#### Environment Variable Format
289+
290+
The environment variable `SETUPTOOLS_SCM_OVERRIDES_FOR_${NORMALIZED_DIST_NAME}` must be set where:
291+
292+
1. **`${NORMALIZED_DIST_NAME}`** is your package name normalized according to PEP 503:
293+
- Convert to uppercase
294+
- Replace hyphens and dots with underscores
295+
- Examples: `my-package` → `MY_PACKAGE`, `my.package` → `MY_PACKAGE`
296+
297+
2. **Value** must be a valid TOML inline table format:
298+
```bash
299+
SETUPTOOLS_SCM_OVERRIDES_FOR_MYPACKAGE='{"local_scheme": "no-local-version"}'
300+
```
301+
302+
#### Alternative: pyproject.toml Configuration
303+
304+
Instead of environment variables, you can configure this in your `pyproject.toml`:
305+
306+
```toml title="pyproject.toml"
307+
[tool.setuptools_scm]
308+
# Use no-local-version by default for CI builds
309+
local_scheme = "no-local-version"
310+
```
311+
312+
However, the environment variable approach is preferred for CI/CD as it allows different schemes for local development vs. CI builds.
313+
314+
#### Version Examples
315+
316+
**Development versions from main branch** (with `local_scheme = "no-local-version"`):
317+
- Development commit: `1.2.3.dev4+g1a2b3c4d5` → `1.2.3.dev4` ✅ (uploadable to PyPI)
318+
- Dirty working directory: `1.2.3+dirty` → `1.2.3` ✅ (uploadable to PyPI)
319+
320+
**Tagged releases** (without overrides, using default local scheme):
321+
- Tagged commit: `1.2.3` → `1.2.3` ✅ (uploadable to PyPI)
322+
- Tagged release on dirty workdir: `1.2.3+dirty` → `1.2.3+dirty` ❌ (should not happen in CI)
323+
324+
### Security Notes
325+
326+
- Store PyPI API tokens as repository secrets
327+
- Use separate tokens for test-PyPI and production PyPI
328+
- Consider using [Trusted Publishers](https://docs.pypi.org/trusted-publishers/) for enhanced security
329+
330+
### Troubleshooting
331+
332+
**Package name normalization**: If your override isn't working, verify the package name normalization:
333+
334+
```python
335+
import re
336+
dist_name = "my-awesome.package"
337+
normalized = re.sub(r"[-_.]+", "-", dist_name)
338+
env_var_name = normalized.replace("-", "_").upper()
339+
print(f"SETUPTOOLS_SCM_OVERRIDES_FOR_{env_var_name}")
340+
# Output: SETUPTOOLS_SCM_OVERRIDES_FOR_MY_AWESOME_PACKAGE
341+
```
342+
343+
**Fetch depth**: Always use `fetch-depth: 0` in GitHub Actions to ensure setuptools-scm has access to the full git history for proper version calculation.

0 commit comments

Comments
 (0)