Refactor repo structure to prepare addition of a Rust recorder (#10)

We've prepared the repo to start working on a second Rust-based
recorder. We've added a POC Rust module and CI scripts to verify that it
builds correctly

Author: tzanko-matev

.github/workflows/ci.yml

@@ -7,32 +7,39 @@ on:
     branches: [main]
 
 jobs:
-  tests:
-    name: Tests on ${{ matrix.os }}
-    runs-on: ${{ matrix.os }}
-    strategy:
-      matrix:
-        os: [ubuntu-latest, macos-latest, windows-latest]
-    steps:
-      - uses: actions/checkout@v4
-      - name: Install make on Windows
-        if: runner.os == 'Windows'
-        run: choco install make -y
-      - uses: actions/setup-python@v5
-        with:
-          python-version: '3.x'
-      - name: Run tests
-        shell: bash
-        run: make test
-
   nix-tests:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
       - uses: cachix/install-nix-action@v27
         with:
-          nix_path: nixpkgs=channel:nixos-24.05
+          nix_path: nixpkgs=channel:nixos-25.05
           extra_nix_config: |
             experimental-features = nix-command flakes
       - name: Run tests via Nix
-        run: nix develop --command make test
+        run: nix develop --command just test
+
+  rust-tests:
+    name: Rust module test on ${{ matrix.os }} (Python ${{ matrix.python-version }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+        python-version: ["10", "11", "12", "13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: 3.${{ matrix.python-version }}
+      - uses: astral-sh/setup-uv@v4
+      - uses: messense/maturin-action@v1
+        with:
+          command: build
+          args: --interpreter python3.${{ matrix.python-version }} -m crates/codetracer-python-recorder/Cargo.toml --release
+      - name: Install and test built wheel with uv (pytest)
+        shell: bash
+        run: |
+          v=${{matrix.python-version}}
+          file=(crates/codetracer-python-recorder/target/wheels/*.whl)
+          file="${file[0]}"
+          uv run -p python3.$v --with "${file}" --with pytest -- python -m pytest crates/codetracer-python-recorder/test -q

.gitignore

@@ -1,2 +1,6 @@
 .direnv/
 **/__pycache__/
+.aider*
+.venv/
+**/target/
+build

AGENTS.md

@@ -1,13 +1,26 @@
 # Instructions for Codex
 
-To run the test suite, execute:
+This repository contains two related projects:
+
+- codetracer-pure-python-recorder — the original pure-Python tracer.
+- codetracer-python-recorder — a Rust-backed Python module built with PyO3 and maturin.
+
+To run the Python test suite for the pure-Python tracer, execute:
 
 ```
-make test
+just test
 ```
 
 The tester executes a number of sample programs in `tests/programs` and compares their outputs to the fixtures in `tests/fixtures`.
 
+To build and locally develop-install the Rust-backed module:
+
+```
+just build-rust
+# or:
+maturin develop -m crates/codetracer-python-recorder/Cargo.toml
+```
+
 # Code quality guidelines
 
 - Strive to achieve high code quality.
@@ -33,4 +46,4 @@ https://www.conventionalcommits.org/en/v1.0.0/
 
 In the remaining lines, provide a short description of the implemented functionality.
 Provide sufficient details for the justification of each design decision if multiple
-approaches were considered.
+approaches were considered.

Justfile

@@ -0,0 +1,48 @@
+# Development helpers for the monorepo
+
+# Python versions used for multi-version testing/building with uv
+PY_VERSIONS := "3.10 3.11 3.12 3.13"
+PY_SHORT_VERSIONS := "10 11 12 13"
+# Print toolchain versions to verify the dev environment
+env:
+    python3 --version
+    cargo --version
+    rustc --version
+    maturin --version
+
+# Create a local virtualenv for Python tooling
+venv:
+    test -d .venv || python3 -m venv .venv
+
+# Build and develop-install the Rust-backed Python module
+build-rust:
+    test -d .venv || python3 -m venv .venv
+    VIRTUAL_ENV=.venv maturin develop -m crates/codetracer-python-recorder/Cargo.toml
+
+# Smoke test the Rust module after build
+smoke-rust:
+    .venv/bin/python -m pip install -U pip pytest
+    .venv/bin/python -m pytest crates/codetracer-python-recorder/test -q
+
+# Run the Python test suite for the pure-Python recorder
+test:
+    python3 -m unittest discover -v
+
+# Run the test suite across multiple Python versions using uv
+test-uv-all:
+    uv python install {{PY_VERSIONS}}
+    for v in {{PY_VERSIONS}}; do uv run -p "$v" -m unittest discover -v; done
+
+# Build wheels for all target Python versions with maturin
+build-rust-uv-all:
+    for v in {{PY_VERSIONS}}; do \
+        maturin build --interpreter "python$v" -m crates/codetracer-python-recorder/Cargo.toml --release; \
+    done
+
+# Smoke the built Rust wheels across versions using uv
+smoke-rust-uv-all:
+    for v in {{PY_SHORT_VERSIONS}}; do \
+        file=(crates/codetracer-python-recorder/target/wheels/codetracer_python_recorder-*-cp3$v-cp3$v-*.whl); \
+        file="${file[0]}"; \
+        uv run -p "python3.$v" --with "${file}" --with pytest -- python -m pytest crates/codetracer-python-recorder/test -q; \
+    done

Makefile

@@ -1,4 +1,4 @@
+# This Makefile is deprecated. Use 'just test' instead.
 .PHONY: test
-
 test:
-	python3 -m unittest discover -v
+	@echo "Deprecated: Use 'just test' instead." && false

README.md

@@ -1,44 +1,57 @@
-## codetracer-python-recorder
+## CodeTracer Recorders (Monorepo)
 
-An unfinished prototype of a recorder of Python programs that produces [CodeTracer](https://github.com/metacraft-labs/CodeTracer) traces.
+This repository now hosts two related projects:
 
-> [!WARNING]
-> Currently it is in a very early phase: we're welcoming contribution and discussion!
+- codetracer-pure-python-recorder — the existing pure-Python prototype that records [CodeTracer](https://github.com/metacraft-labs/CodeTracer) traces using sys.settrace.
+- codetracer-python-recorder — a new, Rust-backed Python extension module (PyO3) intended to provide a faster and more featureful recorder.
 
+> [!WARNING]
+> Both projects are early-stage prototypes. Contributions and discussion are welcome!
 
-### Usage
+### codetracer-pure-python-recorder
 
-Install the package with `pip` or `uv`:
+Install from PyPI:
 
 ```bash
-pip install codetracer-python-recorder
+pip install codetracer-pure-python-recorder
 ```
 
-Then invoke the recorder as a command line tool:
+CLI usage:
 
 ```bash
 codetracer-record <path to python file>
 # produces several trace json files in the current directory
 # or in the folder of `$CODETRACER_DB_TRACE_PATH` if such an env var is defined
 ```
 
-During development you can also run it directly with
+During development you can also run it directly:
 
 ```bash
-python trace.py <path to python file>
+python src/trace.py <path to python file>
 # produces several trace json files in the current directory
 # or in the folder of `$CODETRACER_DB_TRACE_PATH` if such an env var is defined
 ```
 
-however you probably want to use it in combination with CodeTracer, which would be released soon.
+### codetracer-python-recorder (Rust-backed)
+
+A separate Python module implemented in Rust with PyO3 and built via maturin lives under:
+crates/codetracer-python-recorder/
+
+Basic workflow:
+
+- Build/dev install the Rust module:
+  - maturin develop -m crates/codetracer-python-recorder/Cargo.toml
+- Use in Python:
+  - from codetracer_python_recorder import hello
+  - hello()
 
-## Future directions
+### Future directions
 
 The current Python support is an unfinished prototype. We can finish it. In the future, it may be expanded to function in a way to similar to the more complete implementations, e.g. [Noir](https://github.com/blocksense-network/noir/tree/blocksense/tooling/tracer).
 
 Currently it's very similar to our [Ruby tracer](https://github.com/metacraft-labs/ct-ruby-tracer)
 
-### Current approach: sys.settrace API
+#### Current approach: sys.settrace API
 
 Currently we're using the sys.settrace API: https://docs.python.org/3/library/sys.html#sys.settrace .
 This is very flexible and can function with probably multiple Python versions out of the box. 
@@ -49,15 +62,15 @@ However, this is limited:
 
 For other languages, we've used a more deeply integrated approach: patching the interpreter or VM itself (e.g. Noir).
 
-### Patching the VM
+#### Patching the VM
 
 This can be a good approach for Python as well: it can let us record more precisely subvalues, assignments and subexpressions and to let
 some CodeTracer features work in a deeper/better way.
 
 One usually needs to add additional logic to places where new opcodes/lines are being ran, and to call entries/exits. Additionally
 tracking assignments can be a great addition, but it really depends on the interpreter internals.
 
-### Filtering
+#### Filtering
 
 It would be useful to have a way to record in detail only certain periods of the program, or certain functions or modules: 
 we plan on expanding the [trace format](https://github.com/metacraft-labs/runtime_tracing/) and CodeTracer' support, so that this is possible. It would let one be able to record interesting