Coverage reports

Author: tzanko-matev

design-docs/test-suite-coverage-plan.md

@@ -9,19 +9,27 @@
 - **Rust:** Use `cargo llvm-cov` to aggregate unit and integration test coverage. This tool integrates with `nextest` and produces both lcov and HTML outputs. It works with the existing `nix develop` environment once `llvm-tools-preview` is available (already pulled by rustup in Nix environment).
 - **Python:** Use `pytest --cov` with the `coverage` plugin. Restrict collection to the `codetracer_python_recorder` package to avoid noise from site-packages. Generate both terminal summaries and Cobertura XML for upload.
 
+## Prerequisites & Dependencies
+- Add `cargo-llvm-cov` to the dev environment so the Just targets and CI runners share the same binary. In the Nix shell, include the package and ensure the Rust toolchain exposes `llvm-tools-preview`.
+- Extend the UV `dev` dependency group with `pytest-cov` and `coverage[toml]` so Python coverage instrumentation is reproducible locally and in CI.
+- Standardise coverage outputs under `codetracer-python-recorder/target/coverage` to keep artefacts inside the Rust crate. Use `target/coverage/{rust,python}` for per-language assets and a top-level `index.txt` to note the run metadata if needed later.
+
 ## Execution Strategy
 1. **Local Workflow**
-   - Add convenience Just targets:
-     - `just coverage-rust` → `uv run cargo llvm-cov --no-default-features --lcov --output-path target/coverage/rust.lcov` (also emit HTML under `target/coverage/html`).
-     - `just coverage-python` → `uv run --group dev --group test pytest --cov=codetracer_python_recorder --cov-report=term --cov-report=xml:target/coverage/python.xml tests/python`.
-   - Document usage in `tests/README.md` so developers can opt-in.
+   - Add convenience Just targets that mirror the default test steps:
+     - `just coverage-rust` → `uv run cargo llvm-cov --manifest-path codetracer-python-recorder/Cargo.toml --no-default-features --nextest --lcov --output-path codetracer-python-recorder/target/coverage/rust/lcov.info --html codetracer-python-recorder/target/coverage/rust/html`.
+     - `just coverage-python` → `uv run --group dev --group test pytest --cov=codetracer_python_recorder --cov-report=term --cov-report=xml:codetracer-python-recorder/target/coverage/python/coverage.xml codetracer-python-recorder/tests/python`.
+     - `just coverage` wrapper → runs the Rust step followed by the Python step so developers get both artefacts with one command, matching the eventual CI flow.
+   - Ensure the commands create their output directories (`target/coverage/rust` and `target/coverage/python`) before writing results to avoid failures on first use.
+   - Document the workflow in `codetracer-python-recorder/tests/README.md` (and reference the top-level `README` if needed) so contributors know when to run the coverage helpers versus the regular test splits.
 
 2. **CI Integration (non-blocking first pass)**
-   - Extend `.github/workflows/ci.yml` with optional coverage steps that run after the primary test steps on one Python version (e.g., 3.12) to limit runtime.
-   - Upload artefacts:
-     - Rust: `rust-coverage.lcov` + zipped HTML report.
-     - Python: `python-coverage.xml` + terminal summary (already in logs).
-   - Mark coverage steps as `continue-on-error: true` initially so transient issues don’t fail the pipeline.
+   - Extend `.github/workflows/ci.yml` with optional `coverage-rust` and `coverage-python` jobs that depend on the primary test jobs and only run when `matrix.python-version == '3.12'` and `matrix.os == 'ubuntu-latest'` to avoid duplicate collection.
+   - Reuse the Just targets so CI mirrors local behaviour. Inject `RUSTFLAGS`/`RUSTDOCFLAGS` from the test jobs’ cache to avoid rebuilding dependencies.
+   - Publish artefacts via `actions/upload-artifact`:
+     - Rust: `codetracer-python-recorder/target/coverage/rust/lcov.info` plus a gzipped archive of the HTML folder.
+     - Python: `codetracer-python-recorder/target/coverage/python/coverage.xml` for future parsing.
+   - Mark coverage steps with `continue-on-error: true` during the stabilisation phase and note the run IDs in the job summary for quick retrieval.
 
 3. **Reporting & Visualisation**
    - Use GitHub Actions artefacts for report retrieval.
@@ -35,6 +43,13 @@
    - Remove `continue-on-error` and make coverage generation mandatory.
    - Introduce thresholds (e.g., fail if Rust line coverage < 70% or Python < 60%)—subject to discussion with the Runtime Tracing Team.
 
+## Implementation Checklist
+- [ ] Update development environment dependencies (`flake.nix`, `pyproject.toml`) to support coverage tooling out of the box.
+- [ ] Add `just coverage-rust`, `just coverage-python`, and `just coverage` helpers with directory bootstrapping.
+- [ ] Refresh documentation (`codetracer-python-recorder/tests/README.md` and top-level testing guide) with coverage instructions.
+- [ ] Extend CI workflow with non-blocking coverage jobs and artefact upload.
+- [ ] Review initial coverage artefacts to set baseline thresholds before enforcement.
+
 ## Risks & Mitigations
 - **Runtime overhead:** Coverage runs are slower. Mitigate by limiting to a single matrix entry and caching `target/coverage` directories if needed.
 - **Report size:** HTML artefacts can be large. Compress before upload and prune historical runs as necessary.
@@ -45,3 +60,5 @@
 - Implement the local Just targets and update documentation.
 - Extend CI workflow with optional coverage steps (post-tests) and artefact upload.
 - Align with the developer experience team before enforcing thresholds.
+
+_Status tracking lives in `design-docs/test-suite-coverage-plan.status.md`._

design-docs/test-suite-coverage-plan.status.md

@@ -0,0 +1,13 @@
+# Test Suite Coverage Plan Status
+
+## Current Status
+- ✅ Plan doc expanded with prerequisites, detailed Just targets, CI strategy, and an implementation checklist (see `design-docs/test-suite-coverage-plan.md`).
+- 🚧 Implementation: add coverage tooling dependencies to the dev shell and UV groups so local + CI runners share the same setup.
+- ⏳ Implementation: land `just coverage-*` helpers and update developer documentation.
+- ⏳ Implementation: wire optional coverage jobs into `.github/workflows/ci.yml` with artefact uploads.
+- ⏳ Assessment: capture baseline coverage numbers before proposing enforcement thresholds.
+
+## Next Steps
+1. Update environment dependencies for coverage (`cargo-llvm-cov`, `pytest-cov`, `coverage[toml]`).
+2. Introduce the Just coverage commands and document how to use them.
+3. Extend CI with non-blocking coverage collection and review the initial artefacts.