Add support for PyPI trusted publisher and NPM provenance (#511)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Steven Silvester <[email protected]>

Author: 3 people

docs/source/get_started/making_release_from_releaser.md

@@ -16,6 +16,10 @@ already uses Jupyter Releaser.
 
 - Add the token as `ADMIN_GITHUB_TOKEN` in the [repository secrets](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository) of your fork. The token must have `repo` and `workflow` scopes.
 
+- Set up PyPI:
+
+<details><summary>Using PyPI token (legacy way)</summary>
+
 - If the repo generates PyPI release(s), create a scoped PyPI [token](https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#saving-credentials-on-github). We recommend using a scoped token for security reasons.
 
 - You can store the token as `PYPI_TOKEN` in your fork's `Secrets`.
@@ -34,8 +38,21 @@ already uses Jupyter Releaser.
     owner1/repo1/path/to/package2,token2
     ```
 
+</details>
+
+<details><summary>Using PyPI trusted publisher (modern way)</summary>
+
+- Set up your PyPI project by [adding a trusted publisher](https://docs.pypi.org/trusted-publishers/adding-a-publisher/)
+  - if you use the example workflows, the _workflow name_ is `publish-release.yml` (or `full-release.yml`) and the
+    _environment_ should be left blank.
+- Ensure the publish release job as `permissions`: `id-token : write` (see the [documentation](https://docs.pypi.org/trusted-publishers/using-a-publisher/))
+
+</details>
+
 - If the repo generates npm release(s), add access token for [npm](https://docs.npmjs.com/creating-and-viewing-access-tokens), saved as `NPM_TOKEN` in "Secrets".
 
+> If you want to set _provenance_ on your package, you need to ensure the publish release job as `permissions`: `id-token : write` (see the [documentation](https://docs.npmjs.com/generating-provenance-statements#publishing-packages-with-provenance-via-github-actions)).
+
 ## Prep Release
 
 - Go to the "Actions" tab in your fork of `jupyter_releaser`

docs/source/how_to_guides/convert_repo_from_releaser.md

@@ -23,7 +23,11 @@ A. Prep the `jupyter_releaser` fork:
   [repository secrets](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository).
   The token will need "public_repo", and "repo:status" permissions.
 
-- [ ] Add access token for the [PyPI registry](https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#saving-credentials-on-github) stored as `PYPI_TOKEN`.
+- [ ] Set up PyPI:
+
+<details><summary>Using PyPI token (legacy way)</summary>
+
+- Add access token for the [PyPI registry](https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#saving-credentials-on-github) stored as `PYPI_TOKEN`.
   _Note_ For security reasons, it is recommended that you scope the access
   to a single repository, and use a variable called `PYPI_TOKEN_MAP` that is formatted as follows:
 
@@ -39,8 +43,21 @@ A. Prep the `jupyter_releaser` fork:
   owner1/repo1/path/to/package2,token2
   ```
 
+</details>
+
+<details><summary>Using PyPI trusted publisher (modern way)</summary>
+
+- Set up your PyPI project by [adding a trusted publisher](https://docs.pypi.org/trusted-publishers/adding-a-publisher/)
+  - if you use the example workflows, the _workflow name_ is `publish-release.yml` (or `full-release.yml`) and the
+    _environment_ should be left blank.
+- Ensure the publish release job as `permissions`: `id-token : write` (see the [documentation](https://docs.pypi.org/trusted-publishers/using-a-publisher/))
+
+</details>
+
 - [ ] If needed, add access token for [npm](https://docs.npmjs.com/creating-and-viewing-access-tokens), saved as `NPM_TOKEN`.
 
+> If you want to set _provenance_ on your package, you need to ensure the publish release job as `permissions`: `id-token : write` (see the [documentation](https://docs.npmjs.com/generating-provenance-statements#publishing-packages-with-provenance-via-github-actions)).
+
 B. Prep target repository:
 
 - [ ] Switch to Markdown Changelog

docs/source/how_to_guides/convert_repo_from_repo.md

@@ -9,7 +9,9 @@ See checklist below for details:
 - Markdown changelog
 - Bump version configuration (if using Python), for example [hatch](https://hatch.pypa.io/latest/)
 - [Access token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) with access to target GitHub repo to run GitHub Actions.
-- Access token for the [PyPI registry](https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#saving-credentials-on-github)
+- Set up:
+  - \[_modern way_\] [Add a trusted publisher](https://docs.pypi.org/trusted-publishers/adding-a-publisher/) to your PyPI project
+  - \[_legacy way_\] Access token for the [PyPI registry](https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#saving-credentials-on-github)
 - If needed, access token for [npm](https://docs.npmjs.com/creating-and-viewing-access-tokens).
 
 ## Checklist for Adoption
@@ -22,10 +24,27 @@ See checklist below for details:
   access token to allow for branch protection rules, which block the pushing
   of commits when using the `GITHUB_TOKEN`, even when run from an admin user
   account.
-- [ ] Add access token for the [PyPI registry](https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#saving-credentials-on-github) stored as `PYPI_TOKEN`.
+
+- [ ] Set up PyPI:
+
+<details><summary>Using PyPI token (legacy way)</summary>
+
+- Add access token for the [PyPI registry](https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#saving-credentials-on-github) stored as `PYPI_TOKEN`.
   _Note_ For security reasons, it is recommended that you scope the access
   to a single repository. Additionally, this token should belong to a
   machine account and not a user account.
+
+</details>
+
+<details><summary>Using PyPI trusted publisher (modern way)</summary>
+
+- Set up your PyPI project by [adding a trusted publisher](https://docs.pypi.org/trusted-publishers/adding-a-publisher/)
+  - if you use the example workflows, the _workflow name_ is `publish-release.yml` (or `full-release.yml`) and the
+    _environment_ should be left blank.
+- Ensure the publish release job as `permissions`: `id-token : write` (see the [documentation](https://docs.pypi.org/trusted-publishers/using-a-publisher/))
+
+</details>
+
 - [ ] If needed, add access token for [npm](https://docs.npmjs.com/creating-and-viewing-access-tokens), saved as `NPM_TOKEN`. Again this should
   be created using a machine account that only has publish access.
 - [ ] Ensure that only trusted users with 2FA have admin access to the

example-workflows/full-release.yml

@@ -25,6 +25,10 @@ on:
 jobs:
   full_release:
     runs-on: ubuntu-latest
+    permissions:
+      # This is useful if you want to use PyPI trusted publisher
+      # and NPM provenance
+      id-token: write
     steps:
       - uses: jupyterlab/maintainer-tools/.github/actions/base-setup@v1
 
@@ -51,9 +55,10 @@ jobs:
       - name: Finalize Release
         id: finalize-release
         env:
-          PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
-          PYPI_TOKEN_MAP: ${{ secrets.PYPI_TOKEN_MAP }}
-          TWINE_USERNAME: __token__
+          # The following are needed if you use legacy PyPI set up
+          # PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
+          # PYPI_TOKEN_MAP: ${{ secrets.PYPI_TOKEN_MAP }}
+          # TWINE_USERNAME: __token__
           NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
         uses: jupyter-server/jupyter-releaser/.github/actions/finalize-release@v2
         with:

example-workflows/publish-release.yml

@@ -15,6 +15,10 @@ on:
 jobs:
   publish_release:
     runs-on: ubuntu-latest
+    permissions:
+      # This is useful if you want to use PyPI trusted publisher
+      # and NPM provenance
+      id-token: write
     steps:
       - uses: jupyterlab/maintainer-tools/.github/actions/base-setup@v1
 
@@ -30,9 +34,10 @@ jobs:
       - name: Finalize Release
         id: finalize-release
         env:
-          PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
-          PYPI_TOKEN_MAP: ${{ secrets.PYPI_TOKEN_MAP }}
-          TWINE_USERNAME: __token__
+          # The following are needed if you use legacy PyPI set up
+          # PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
+          # PYPI_TOKEN_MAP: ${{ secrets.PYPI_TOKEN_MAP }}
+          # TWINE_USERNAME: __token__
           NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
         uses: jupyter-server/jupyter-releaser/.github/actions/finalize-release@v2
         with:

jupyter_releaser/lib.py

@@ -342,6 +342,11 @@ def publish_assets(  # noqa
     if release_url and len(glob(f"{dist_dir}/*.whl")):
         twine_token = python.get_pypi_token(release_url, python_package_path)
 
+        if twine_token:
+            # tell GitHub Actions to mask the token in any console logs,
+            # to avoid leaking it
+            util.run(f'echo "::add-mask::{twine_token}"')
+
     if dry_run:
         # Start local pypi server with no auth, allowing overwrites,
         # in a temporary directory

jupyter_releaser/npm.py

@@ -135,6 +135,13 @@ def handle_npm_config(npm_token):
             auth_entry = ""
 
     text += f"\n{reg_entry}\n{auth_entry}"
+
+    if os.environ.get(util.GH_ID_TOKEN_TOKEN_VAR, ""):
+        util.log("Turning on NPM provenance as id-token permission is set.")
+        # See documentation https://docs.npmjs.com/generating-provenance-statements
+        # Also https://github.blog/2023-04-19-introducing-npm-package-provenance/
+        text += "\nprovenance=true"
+
     text = text.strip() + "\n"
     util.log(f"writing npm config to {npmrc}")
     npmrc.write_text(text, encoding="utf-8")

jupyter_releaser/python.py

@@ -2,20 +2,26 @@
 # Copyright (c) Jupyter Development Team.
 # Distributed under the terms of the Modified BSD License.
 import atexit
+import json
 import os
 import os.path as osp
 import re
 import shlex
 from glob import glob
+from io import BytesIO
 from pathlib import Path
 from subprocess import PIPE, CalledProcessError, Popen
 from tempfile import TemporaryDirectory
 
+import requests
+
 from jupyter_releaser import util
 
 PYPROJECT = util.PYPROJECT
 SETUP_PY = util.SETUP_PY
 
+PYPI_GH_API_TOKEN_URL = "https://pypi.org/_/oidc/github/mint-token"  # noqa
+
 
 def build_dist(dist_dir, clean=True):
     """Build the python dist files into a dist folder"""
@@ -95,11 +101,60 @@ def check_dist(
             util.run(cmd)
 
 
+def fetch_pypi_api_token() -> "str":
+    """Fetch the PyPI API token for trusted publishers
+
+    This implements the manual steps described in https://docs.pypi.org/trusted-publishers/using-a-publisher/
+    as of June 19th, 2023.
+
+    It returns an empty string if it fails.
+    """
+    util.log("Fetching PyPI OIDC token...")
+
+    url = os.environ.get(util.GH_ID_TOKEN_URL_VAR, "")
+    auth = os.environ.get(util.GH_ID_TOKEN_TOKEN_VAR, "")
+    if not url or not auth:
+        util.log(
+            "Please verify that you have granted `id-token: write` permission to the publish workflow."
+        )
+        return ""
+
+    headers = {"Authorization": f"bearer {auth}", "Accept": "application/octet-stream"}
+
+    sink = BytesIO()
+    with requests.get(f"{url}&audience=pypi", headers=headers, stream=True, timeout=60) as r:
+        r.raise_for_status()
+        for chunk in r.iter_content(chunk_size=8192):
+            sink.write(chunk)
+    sink.seek(0)
+    oidc_token = json.loads(sink.read().decode("utf-8")).get("value", "")
+
+    if not oidc_token:
+        util.log("Failed to fetch the OIDC token from PyPI.")
+        return ""
+
+    util.log("Fetching PyPI API token...")
+    sink = BytesIO()
+    with requests.post(PYPI_GH_API_TOKEN_URL, json={"token": oidc_token}, timeout=10) as r:
+        r.raise_for_status()
+        for chunk in r.iter_content(chunk_size=8192):
+            sink.write(chunk)
+    sink.seek(0)
+    api_token = json.loads(sink.read().decode("utf-8")).get("token", "")
+
+    return api_token
+
+
 def get_pypi_token(release_url, python_package):
     """Get the PyPI token
 
     Note: Do not print the token in CI since it will not be sanitized
     if it comes from the PYPI_TOKEN_MAP"""
+    trusted_token = os.environ.get(util.GH_ID_TOKEN_TOKEN_VAR, "")
+
+    if trusted_token:
+        return fetch_pypi_api_token()
+
     twine_pwd = os.environ.get("PYPI_TOKEN", "")
     pypi_token_map = os.environ.get("PYPI_TOKEN_MAP", "").replace(r"\n", "\n")
     if pypi_token_map and release_url:

jupyter_releaser/util.py

@@ -57,6 +57,9 @@
 
 GIT_FETCH_CMD = "git fetch origin --filter=blob:none --quiet"
 
+GH_ID_TOKEN_URL_VAR = "ACTIONS_ID_TOKEN_REQUEST_URL"  # noqa
+GH_ID_TOKEN_TOKEN_VAR = "ACTIONS_ID_TOKEN_REQUEST_TOKEN"  # noqa
+
 
 def run(cmd, **kwargs):
     """Run a command as a subprocess and get the output as a string"""