Pypi release pipeline: design documents

design-docs/adr/0006-codetracer-python-recorder-pypi-release.md: 
design-docs/codetracer-python-recorder-pypi-release-implementation-plan.md: 

Signed-off-by: Tzanko Matev <[email protected]>

Author: tzanko-matev

design-docs/adr/0006-codetracer-python-recorder-pypi-release.md

@@ -0,0 +1,98 @@
+# ADR 0006: PyPI Release Strategy for `codetracer-python-recorder`
+
+- **Status:** Proposed
+- **Date:** 2025-10-13
+- **Deciders:** Codetracer Runtime & Tooling Leads
+- **Consulted:** Release Engineering, Developer Experience, Platform Reliability
+- **Informed:** Product Management, Support, Security
+
+## Context
+
+We are ready to publish the first public release of the Rust-backed `codetracer-python-recorder`
+module to PyPI. The business goal is to distribute an officially supported recording module for
+Python 3.12 and 3.13 across Linux, macOS, and Windows, with both binary wheels and a source
+distribution that downstream teams can audit and rebuild.
+
+A repository review shows that:
+
+- Packaging metadata in `codetracer-python-recorder/pyproject.toml` still targets Python 3.8+,
+  omits Trove classifiers for supported versions and platforms, and does not expose project URLs
+  or the README as the long description.
+- The `Justfile` and CI only support developer builds (`maturin develop`) and Linux test jobs;
+  there is no automated release workflow and no multi-platform wheel builds.
+- `tool.maturin` configuration does not yet declare source-distribution rules, exclude build
+  artefacts, or ensure that `Cargo.toml` / `pyproject.toml` versions stay synchronized.
+- There is no documented path for staging releases on TestPyPI or for securely authenticating a CI
+  workflow to PyPI.
+
+We consulted the Python Packaging User Guide packaging flow
+([packaging.python.org](https://packaging.python.org/en/latest/flow/)) and the maturin distribution
+guide ([maturin.rs](https://www.maturin.rs/distribution.html)). Key takeaways include:
+
+- Every release must produce both wheels and an sdist, validate metadata with tooling, and verify
+  installability before uploading.
+- PyPI now recommends OpenID Connect “Trusted Publishing” for CI-driven uploads instead of storing
+  long-lived API tokens.
+- maturin provides first-class support for building manylinux, macOS universal2, and Windows wheels,
+  plus TestPyPI/PyPI uploads from CI runners.
+
+## Decision
+
+1. **Metadata hardening:** Update `pyproject.toml` to `requires-python = ">=3.12,<3.14"`, include
+   the project README as the long description, add platform-specific Trove classifiers (Linux,
+   macOS, Windows; CPython 3.12/3.13; Rust), and publish canonical URLs (homepage, repository,
+   issues). Mirror the version in `Cargo.toml` and add a guard that compares Rust and Python
+   versions during CI.
+2. **Distribution artefacts:** Continue to use maturin as the build backend. For the initial public
+   release, build per-version wheels (`cp312` and `cp313`) for
+   - manylinux2014 `x86_64` and `aarch64`,
+   - macOS universal2 (`x86_64` + `arm64`),
+   - Windows `amd64`.
+   Produce a source distribution via `maturin sdist`, ensuring `Cargo.lock`, Rust sources, and Python
+   shim modules are included while excluding wheel artefacts (`target/`, compiled `.so` files).
+   Evaluate enabling the `pyo3/abi3-py312` feature after the first release to collapse per-version
+   wheels.
+3. **Pre-release verification:** Extend the release pipeline to run unit tests against the built
+   artefacts, execute smoke installs (`pip install` from the local wheel and sdist), and run the CLI
+   (`python -m codetracer_python_recorder --help`) before any upload step.
+4. **Trusted publishing workflow:** Create a dedicated GitHub Actions workflow triggered by
+   annotated tags (e.g., `recorder-v*`). The workflow will:
+   - Build and test wheels on each platform matrix job.
+   - Upload artefacts to a staging job that performs TestPyPI publishing via maturin.
+   - Require a manual approval (environment protection) before promoting the same artefacts to the
+     production PyPI repository.
+   Configure the PyPI project as a Trusted Publisher for the repository so uploads rely on OIDC
+   tokens instead of stored secrets.
+5. **Versioning & change management:** Adopt a documented version-bump process that updates both
+   `pyproject.toml` and `Cargo.toml`, updates the changelog/release notes, and tags releases using
+   `recorder-vMAJOR.MINOR.PATCH`. Enforce semantic-versioning semantics via review, and block
+   accidental reuse of versions by asserting that the requested tag matches the metadata.
+6. **Documentation & support:** Add a release checklist to the repository (under `design-docs`) that
+   references PyPI’s packaging flow steps (metadata validation, TestPyPI verification, final
+   release), and document how downstream consumers install the wheel (platform notes, supported
+   Python versions).
+
+## Alternatives Considered
+
+- **Manual, developer-driven uploads:** Rejected because manual wheel builds are error-prone,
+  difficult to reproduce across platforms, and conflict with PyPI’s recommendation to use trusted
+  CI publishing.
+- **cibuildwheel-backed workflow:** Considered but rejected for this release; maturin already owns
+  the build backend, integrates with PyO3, and provides the required cross-platform coverage with
+  less indirection.
+- **Limiting support to Linux and source distributions:** Rejected because product requirements call
+  for macOS and Windows parity from day one, and maturin’s cross-platform build support removes the
+  main barrier to doing so.
+
+## Consequences
+
+- **Positive:** Reproducible, tested release artefacts; reduced risk of shipping mismatched metadata;
+  streamlined release process that matches the official packaging flow; improved supply-chain
+  posture through OIDC trusted publishing.
+- **Negative:** Longer CI pipelines (multi-platform builds) and the operational overhead of
+  configuring PyPI Trusted Publishing. macOS universal2 builds increase macOS runner usage, and
+  cross-compiling for `aarch64` may extend Linux build times.
+- **Risks & Mitigations:** maturin cross-build failures are possible—mitigate by caching Rust
+  dependencies and adding smoke tests that catch platform-specific regressions early. If Trusted
+  Publishing is unavailable during initial setup, fall back temporarily to a scoped PyPI API token
+  stored in GitHub environments while tracking completion of the OIDC configuration.

design-docs/codetracer-python-recorder-pypi-release-implementation-plan.md

@@ -0,0 +1,84 @@
+# `codetracer-python-recorder` PyPI Release – Implementation Plan
+
+This plan captures the work required to implement ADR 0006 (“PyPI Release Strategy for
+`codetracer-python-recorder`”) and prepare the project for automated publishing.
+
+---
+
+## Workstream 1 – Package metadata & repository hygiene
+
+1. **Tighten project metadata**
+   - Update `codetracer-python-recorder/pyproject.toml` with `requires-python = ">=3.12,<3.14"`,
+     `readme = "README.md"`, Trove classifiers for CPython 3.12/3.13 and the three target
+     platforms, and `project.urls` entries (`Homepage`, `Repository`, `Issues`, `Changelog`).
+   - Ensure licensing information references the repository MIT license file.
+2. **Synchronize Rust/Python versions**
+   - Add a repository script (e.g., `scripts/check_recorder_version.py`) that verifies the version in
+     `pyproject.toml` matches `Cargo.toml`.
+   - Wire the check into CI (pull requests and release workflow) so mismatches fail fast.
+3. **Curate source-distribution contents**
+   - Expand `[tool.maturin.sdist]` to include Rust sources, `Cargo.lock`, and Python shim modules
+     while excluding build artefacts (`target/`, cached wheels, compiled `.so` files).
+   - Add a `.gitignore` entry for `codetracer_python_recorder/*.so` if not already covered.
+4. **Document supported environments**
+   - Update `codetracer-python-recorder/README.md` with explicit notes on supported Python versions,
+     OS targets, and a short install section showing `pip install codetracer-python-recorder`.
+   - Add a release checklist under `design-docs/` describing version bumps, tagging, and the TestPyPI
+     verification gate.
+
+## Workstream 2 – Build & test enhancements
+
+1. **Augment local build tooling**
+   - Extend `Justfile` targets (`build`, `build-all`) to invoke `maturin build --release --sdist`
+     for the configured Python interpreters (3.12, 3.13) and to drop artefacts under
+     `codetracer-python-recorder/target`.
+   - Provide a `just smoke-wheel` recipe that creates a virtual environment, installs the freshly
+     built wheel/sdist, and runs `python -m codetracer_python_recorder --help`.
+2. **Strengthen automated tests**
+   - Ensure existing Rust `cargo nextest` and Python `pytest` suites run as part of every release
+     build.
+   - Add an integration smoke test that exercises `start_tracing` and `stop_tracing` through the
+     Python facade with temporary directories to guard against obvious regressions.
+3. **Cache dependencies**
+   - Configure `maturin` to use `--locked` builds and enable Rust/Python dependency caching (UV,
+     Cargo) in CI to keep build times predictable across platforms.
+
+## Workstream 3 – Cross-platform build & publish automation
+
+1. **Create release workflow**
+   - Add `.github/workflows/recorder-release.yml` triggered by annotated tags (`recorder-v*`) and a
+     manual `workflow_dispatch`.
+   - Define a matrix for:
+     - Linux: `ubuntu-latest` using `messense/maturin-action` to build manylinux2014 `x86_64` and
+       `aarch64` wheels.
+     - macOS: `macos-13` to build universal2 wheels via `maturin build --universal2`.
+     - Windows: `windows-latest` to build `win_amd64` wheels.
+   - Run `just test` (or explicit Rust/Python test commands) before building artefacts on every job.
+2. **Stage artefacts**
+   - Upload all wheels and the sdist as GitHub Actions artefacts.
+   - Add a downstream job that collects the artefacts, performs `pip install` smoke tests on Linux,
+     and publishes to TestPyPI using `maturin upload --repository testpypi`.
+3. **Trusted publishing setup**
+   - Register the GitHub repository as a PyPI Trusted Publisher, mapping the release workflow to the
+     PyPI project.
+   - Configure GitHub environments to require manual approval (“Promote to PyPI”) and run
+     `maturin upload --repository pypi` only after TestPyPI verification succeeds.
+   - Document fallback instructions for using scoped API tokens if Trusted Publishing cannot be
+     completed before the first release.
+
+## Workstream 4 – Operational readiness & documentation
+
+1. **Release management**
+   - Prepare a CHANGELOG entry for v0.1.0 (or the first publicly published version) following
+     Conventional Commits.
+   - Document how to create and push annotated tags (`git tag -a recorder-vX.Y.Z`).
+2. **Post-publish verification**
+   - Describe the post-release validation steps (download from PyPI on each platform, run CLI smoke
+     tests, monitor PyPI stats).
+   - Create an issue template for future release tracking that references the checklist.
+
+---
+
+**Exit criteria:** The new release workflow successfully publishes a TestPyPI build from CI, a human
+approves promotion to PyPI via the protected environment, and the published package installs and
+passes smoke tests on Linux, macOS, and Windows for Python 3.12 and 3.13.