-
Notifications
You must be signed in to change notification settings - Fork 278
Publish to PyPI via GitHub CI #893
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
EpicWink
wants to merge
30
commits into
pypa:main
Choose a base branch
from
EpicWink:publish-ci-workflow
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+159
−21
Open
Changes from 21 commits
Commits
Show all changes
30 commits
Select commit
Hold shift + click to select a range
a3400f1
Publish to PyPI via GitHub CI
EpicWink b7f3f7e
Don't cache pip in publish CI job
EpicWink 759c36f
Improve language in noxfile comment
EpicWink 70457e5
Use latest Python version available
EpicWink 671d6ab
Switch PyPA action version comment to tag
EpicWink 944e4ee
Split build out from release CI job
EpicWink cc9692b
Document approvals required for release
EpicWink d855d62
Link to version page in release deployment
EpicWink 30cb9f2
Merge remote-tracking branch 'upstream/main' into publish-ci-workflow
EpicWink 6d6ba8e
Separate install from build in release CI job
EpicWink ed64a23
Link to CI job approval documentation
EpicWink b2a9652
Add punctuation to release process documentation
EpicWink 94b6576
Check built distribution during release
EpicWink a0fc13e
Run twine check in strict mode during release
EpicWink f250545
Run twine-check in linting CI workflow
EpicWink 9edb7a5
Remove redundant Python-setup step in release-build CI job
EpicWink ef36fbd
Switch to 'git switch' for switching branches
EpicWink c43d566
Merge branch 'main' into publish-ci-workflow
EpicWink 5f36418
Switch to release 'published' event for CI trigger
EpicWink 75cee2c
Revert "Run twine-check in linting CI workflow"
EpicWink 2bc8119
Merge remote-tracking branch 'upstream/main' into publish-ci-workflow
EpicWink d81fd82
Name checkout step
EpicWink 100ed2f
Document install-only detection
EpicWink 2eae318
Merge remote-tracking branch 'upstream/main' into publish-ci-workflow
EpicWink f5494b8
Use 'pypa/gh-action-pypi-publish' GitHub action v1.13
EpicWink 7b7127e
Make publish job require build job
EpicWink b6b7da6
Merge remote-tracking branch 'upstream/main' into publish-ci-workflow
EpicWink 2c405b9
Format and add type annotations
EpicWink 1ebc7e5
Merge remote-tracking branch 'upstream/main' into publish-ci-workflow
EpicWink 861784f
Switch to manual workflow run
EpicWink File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| name: Publish | ||
|
|
||
| on: | ||
| release: | ||
| types: [published] | ||
|
|
||
| env: | ||
| FORCE_COLOR: 1 | ||
|
|
||
| jobs: | ||
| build: | ||
| runs-on: ubuntu-latest | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | ||
EpicWink marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| # pipx is pre-installed in 'ubuntu' VMs | ||
|
|
||
| - name: Provision nox environment | ||
| run: pipx run nox --install-only | ||
|
|
||
| - name: Build distribution via nox | ||
| run: pipx run nox --no-install --error-on-missing-interpreters -s release_build | ||
|
|
||
| - name: Upload distribution | ||
| uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # 4.6.2 | ||
| with: | ||
| name: packages | ||
| path: dist/ | ||
| if-no-files-found: error | ||
| compression-level: 0 | ||
EpicWink marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| publish: | ||
EpicWink marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| environment: | ||
| name: pypi | ||
| url: https://pypi.org/project/packaging/${{ github.ref_name }} | ||
| permissions: | ||
| id-token: write | ||
EpicWink marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| runs-on: ubuntu-latest | ||
|
|
||
| steps: | ||
| - name: Download distribution | ||
| uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # 4.3.0 | ||
| with: | ||
| name: packages | ||
| path: dist/ | ||
|
|
||
| - name: Publish distribution to PyPI | ||
| uses: pypa/gh-action-pypi-publish@76f52bc884231f62b9a034ebfe128415bbaabdfc # v1.12.4 | ||
EpicWink marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| with: | ||
| print-hash: true | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the last of my concerns. Having this inverse trigger can lead to weird sync issues.
For example, last week we were releasing
pip-toolswhere this trigger is still used. And we ended up with a Git tag in repo and a GitHub Release existing for like 15-20 hours with no PyPI release, due to some approval challenges. Having aworkflow_dispatchcan prevent it when the sequence of actions performed is different.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I assume you want the workflow to instead be:
mainHow do you prevent accidental triggering of publish GitHub workflow? A solution here is to prevent upload except when the wheel's version is of the form "X.Y.Z".
What if there's a problem during tag or release creation, where you have a version on PyPI with no associated tag? A solution here is to yank, but in my opinion this is a much worse outcome than the problems you mentioned, as these can be pinned by downstream users.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Having
environment: pypion the publish job allows configuring the GitHub Environment (calledpypi) protection rules. I normally add "required reviewers" in there (just adding oneself works). This makes it so when GH reaches the job, it pauses and does not proceed until whoever's listed clickedApproveon GH UI.I like having an input with the version field in
workflow_dispatch: https://github.com/cherrypy/cheroot/blob/ac9b6e5/.github/workflows/ci-cd.yml#L19-L28. And this event is only used for publishing so I don't tend to add checks for it. Though, I do have sanity-checks for the dist names. But yes, a check like you mention could be helpful too, I suppose.To solve this, I separate the GH Release creation into a separate job, so that:
And so with this in mind, in my workflows this doesn't constitute a problem. I'm very intentional about that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As I said above:
I've never used
workflow_dispatchfor package release, so I'm not confident enough to write workflow definitions using it.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@EpicWink I can help. Should I send you a patch?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@webknjaz yes please
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've encountered a significant issue with using
workflow_dispatchto publish the workflow. In cryptography's publish workflow, theworkflow_dispatchmust be run frommain, but the wheel can be built from any random branch/tag, as there's no hard linking from the ref the publish workflow is run on. Incryptography's case, the publish workflow uses a custom download-artifact action, which makes it harder to follow.This break in the provenance chain, and I cannot rely on an artefact being generated by a GitHub workflow on a specific Git commit.
@webknjaz is there a way to better link the publish workflow to the build workflow?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think it matters really. Especially since in case of
packaging, the artifact is coming from the same workflow run (just like in the Cheroot example). I don't think thatworkflow_dispatchis significant in this case as we wouldn't be referencing the run-id from another workflow run for retrieving the artifacts. If you want this to be buildable from an arbitrary commit, not justmain'sHEAD, there's no issue with defining an input underworkflow_dispatchfor commit SHA or similar and passing it toactions/checkout(ex: https://github.com/tox-dev/workflow/blob/208490c75f7f6b81e2698cc959f24d264c462d57/.github/workflows/reusable-tox.yml#L25-L32). That'd be the commit you'd create a GH release for in a post-PyPI job.If you're concerned about getting the version for the tag, it's possible to get it via
python -c "import build.util as bu; print(bu.project_wheel_metadata('.')['Version'])")(https://github.com/nedbat/scriv/pull/144/files#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52R16)Though, it's probably a better idea to retrieve it from the built metadata once there's a wheel in the build job, then expose it as a job output and reuse in other jobs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Build will be able to provide it directly in the next release as JSON. Not sure if that helps, as you'd still need to parse the JSON, but if that's easy, it might be simpler.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any progress here? We are probably going to release someone soon (days to weeks, perhaps?), possibly with an RC first, so that would be an opportunity to test this out, if that's helpful.