dev-env: Refactor the repository structure to add the Rust-based recorder

AGENTS.md: 
README.md: 
crates/codetracer-python-recorder/Cargo.toml: 
crates/codetracer-python-recorder/pyproject.toml: 
crates/codetracer-python-recorder/src/lib.rs: 
flake.nix: 

Signed-off-by: Tzanko Matev <[email protected]>

Author: tzanko-matev

AGENTS.md

@@ -1,13 +1,24 @@
 # Instructions for Codex
 
-To run the test suite, execute:
+This repository contains two related projects:
+
+- codetracer-pure-python-recorder — the original pure-Python tracer.
+- codetracer-python-recorder — a Rust-backed Python module built with PyO3 and maturin.
+
+To run the Python test suite for the pure-Python tracer, execute:
 
 ```
 make test
 ```
 
 The tester executes a number of sample programs in `tests/programs` and compares their outputs to the fixtures in `tests/fixtures`.
 
+To build and locally develop-install the Rust-backed module:
+
+```
+maturin develop -m crates/codetracer-python-recorder/Cargo.toml
+```
+
 # Code quality guidelines
 
 - Strive to achieve high code quality.
@@ -33,4 +44,4 @@ https://www.conventionalcommits.org/en/v1.0.0/
 
 In the remaining lines, provide a short description of the implemented functionality.
 Provide sufficient details for the justification of each design decision if multiple
-approaches were considered.
+approaches were considered.

README.md

@@ -1,44 +1,57 @@
-## codetracer-python-recorder
+## CodeTracer Recorders (Monorepo)
 
-An unfinished prototype of a recorder of Python programs that produces [CodeTracer](https://github.com/metacraft-labs/CodeTracer) traces.
+This repository now hosts two related projects:
 
-> [!WARNING]
-> Currently it is in a very early phase: we're welcoming contribution and discussion!
+- codetracer-pure-python-recorder — the existing pure-Python prototype that records [CodeTracer](https://github.com/metacraft-labs/CodeTracer) traces using sys.settrace.
+- codetracer-python-recorder — a new, Rust-backed Python extension module (PyO3) intended to provide a faster and more featureful recorder.
 
+> [!WARNING]
+> Both projects are early-stage prototypes. Contributions and discussion are welcome!
 
-### Usage
+### codetracer-pure-python-recorder
 
-Install the package with `pip` or `uv`:
+Install from PyPI:
 
 ```bash
-pip install codetracer-python-recorder
+pip install codetracer-pure-python-recorder
 ```
 
-Then invoke the recorder as a command line tool:
+CLI usage:
 
 ```bash
 codetracer-record <path to python file>
 # produces several trace json files in the current directory
 # or in the folder of `$CODETRACER_DB_TRACE_PATH` if such an env var is defined
 ```
 
-During development you can also run it directly with
+During development you can also run it directly:
 
 ```bash
-python trace.py <path to python file>
+python src/trace.py <path to python file>
 # produces several trace json files in the current directory
 # or in the folder of `$CODETRACER_DB_TRACE_PATH` if such an env var is defined
 ```
 
-however you probably want to use it in combination with CodeTracer, which would be released soon.
+### codetracer-python-recorder (Rust-backed)
+
+A separate Python module implemented in Rust with PyO3 and built via maturin lives under:
+crates/codetracer-python-recorder/
+
+Basic workflow:
+
+- Build/dev install the Rust module:
+  - maturin develop -m crates/codetracer-python-recorder/Cargo.toml
+- Use in Python:
+  - from codetracer_python_recorder import hello
+  - hello()
 
-## Future directions
+### Future directions
 
 The current Python support is an unfinished prototype. We can finish it. In the future, it may be expanded to function in a way to similar to the more complete implementations, e.g. [Noir](https://github.com/blocksense-network/noir/tree/blocksense/tooling/tracer).
 
 Currently it's very similar to our [Ruby tracer](https://github.com/metacraft-labs/ct-ruby-tracer)
 
-### Current approach: sys.settrace API
+#### Current approach: sys.settrace API
 
 Currently we're using the sys.settrace API: https://docs.python.org/3/library/sys.html#sys.settrace .
 This is very flexible and can function with probably multiple Python versions out of the box. 
@@ -49,15 +62,15 @@ However, this is limited:
 
 For other languages, we've used a more deeply integrated approach: patching the interpreter or VM itself (e.g. Noir).
 
-### Patching the VM
+#### Patching the VM
 
 This can be a good approach for Python as well: it can let us record more precisely subvalues, assignments and subexpressions and to let
 some CodeTracer features work in a deeper/better way.
 
 One usually needs to add additional logic to places where new opcodes/lines are being ran, and to call entries/exits. Additionally
 tracking assignments can be a great addition, but it really depends on the interpreter internals.
 
-### Filtering
+#### Filtering
 
 It would be useful to have a way to record in detail only certain periods of the program, or certain functions or modules: 
 we plan on expanding the [trace format](https://github.com/metacraft-labs/runtime_tracing/) and CodeTracer' support, so that this is possible. It would let one be able to record interesting

crates/codetracer-python-recorder/Cargo.toml

@@ -0,0 +1,14 @@
+[package]
+name = "codetracer-python-recorder"
+version = "0.1.0"
+edition = "2021"
+license = "MIT"
+description = "Rust-backed Python module for CodeTracer recording (PyO3)"
+repository = "https://github.com/metacraft-labs/codetracer-python-recorder"
+
+[lib]
+name = "codetracer_python_recorder"
+crate-type = ["cdylib"]
+
+[dependencies]
+pyo3 = { version = "0.21", features = ["extension-module"] }

crates/codetracer-python-recorder/pyproject.toml

@@ -0,0 +1,22 @@
+[build-system]
+requires = ["maturin>=1.5,<2"]
+build-backend = "maturin"
+
+[project]
+name = "codetracer-python-recorder"
+version = "0.1.0"
+description = "Rust-backed Python module for CodeTracer recording (PyO3)"
+authors = [{name = "Metacraft Labs Ltd"}]
+license = {text = "MIT"}
+requires-python = ">=3.8"
+classifiers = [
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Rust",
+]
+
+[tool.maturin]
+# Build the PyO3 extension module
+bindings = "pyo3"
+# Use the library name as the Python import name: codetracer_python_recorder

crates/codetracer-python-recorder/src/lib.rs

@@ -0,0 +1,16 @@
+use pyo3::prelude::*;
+
+/// codetracer_python_recorder
+///
+/// Minimal placeholder for the Rust-backed recorder. This exposes a trivial
+/// function to verify the module builds and imports successfully.
+#[pyfunction]
+fn hello() -> PyResult<String> {
+    Ok("Hello from codetracer-python-recorder (Rust)".to_string())
+}
+
+#[pymodule]
+fn codetracer_python_recorder(_py: Python<'_>, m: &PyModule) -> PyResult<()> {
+    m.add_function(wrap_pyfunction!(hello, m)?)?;
+    Ok(())
+}

flake.nix

@@ -1,5 +1,5 @@
 {
-  description = "Development environment for codetracer-python-recorder";
+  description = "Development environment for CodeTracer recorders (pure-python and rust-backed)";
 
   inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";
 
@@ -12,7 +12,22 @@
         let pkgs = import nixpkgs { inherit system; };
         in {
           default = pkgs.mkShell {
-            packages = with pkgs; [ bashInteractive python3 just git-lfs ];
+            packages = with pkgs; [
+              bashInteractive
+              python3
+              just
+              git-lfs
+
+              # Rust toolchain for the Rust-backed Python module
+              cargo
+              rustc
+              rustfmt
+              clippy
+
+              # Build tooling for Python extensions
+              maturin
+              pkg-config
+            ];
           };
         });
     };