Initial Python tracer based on runtime_tracing

Summary of changes

- Implemented a Rust-backed tracer using runtime_tracing.
- Added a concrete implementation of the codetracer-python-recorder/src/tracer.rs trait.
- Wired the runtime tracer into the public Python API (start/stop/flush).

Key details

- New tracer implementation: src/runtime_tracer.rs
    - Struct RuntimeTracer backed by runtime_tracing::NonStreamingTraceWriter.
    - Maps selected sys.monitoring events (CALL, LINE, PY_RETURN) to runtime_tracing:
    - CALL: registers function and call (without capturing full arg lists yet).
    - LINE: registers a Step.
    - PY_RETURN: registers return value (optionally captured as ValueRecord).
- 
Minimal value encoder for None, bool, int, str; falls back to Raw string for others.
- 
Begins writing metadata, paths, and events on start and flushes them on finish.
- 
Exposes helper derive_sidecar_paths(events_path) -> (metadata.json, paths.json).
- 
Extended Tracer trait: src/tracer.rs
    - Now Tracer: Send + Any (to safely store in static Mutex).
    - Added downcasting support: fn as_any(&mut self) for optional future use.
    - Added default lifecycle hooks:
    - fn flush(&mut self, _py) -> PyResult<()> { Ok(()) }
    - fn finish(&mut self, _py) -> PyResult<()> { Ok(()) }
- 
Added flush_installed_tracer(py) to flush current tracer without uninstalling.
- 
uninstall_tracer(py) now calls tracer.finish(py) before unhooking callbacks.
- 
Python API integration: src/lib.rs
    - start_tracing(path, format, capture_values, source_roots)
    - Prevents double-start via ACTIVE flag.
    - Creates RuntimeTracer, derives output file names:
      - events: path (user-provided)
      - metadata: path with extension metadata.json
      - paths: path with extension paths.json
    - For simplicity and to remain object-safe in this environment, “binary” maps to the non-streaming BinaryV0 writer. JSON is supported as json.
    - Installs tracer via sys.monitoring and flips ACTIVE to true.
- stop_tracing()
    - Uninstalls the tracer (which calls finish on the tracer) and sets ACTIVE false.
- flush_tracing()
    - Calls flush_installed_tracer(py). For non-streaming formats this writes events to the file; for streaming formats (not used here) this would be a no-op by design.
- is_tracing(): returns ACTIVE.

Behavioral notes

- Formats:
    - json: uses JSON non-streaming writer.
    - binary: mapped to BinaryV0 non-streaming writer in this implementation to avoid relying on the private streaming writer module and to keep the tracer object Send-safe.
- Sidecar files:
    - metadata: .metadata.json
    - paths: .paths.json
    - events: 
- Values capturing:
    - Optional via capture_values flag. Basic types (None, bool, int, str) are handled; all others fall back to Raw.

What I didn’t change

- Existing tests and their tracer implementations (PrintTracer, CountingTracer) remain compatible because all new Tracer methods are default no-ops.

Next steps (optional)

- Expand event interest set (e.g., exceptions, C_RETURN/C_RAISE) to record richer traces.
- Enhance value capture and variable bindings for arguments and locals (requires more Python-level context).
- Consider supporting streaming binary output once a public API for the streaming writer is exposed (or if constraints allow depending on runtime_tracing crate updates).

Commands to run

- Build and tests rely on your environment’s Python/PyO3 toolchain. The repo’s recommended way:
    - just venv 3.13 dev
    - just test

This implementation hooks into sys.monitoring, records with runtime_tracing, and exposes start/stop/flush in the Python module, keeping the code defensive, testable, and focused on the requested trait
implementation.
Signed-off-by: Tzanko Matev <[email protected]>

Activate tracing on script entry
codetracer-python-recorder/codetracer_python_recorder/__main__.py: 
codetracer-python-recorder/codetracer_python_recorder/api.py: 
codetracer-python-recorder/src/lib.rs: 
codetracer-python-recorder/src/runtime_tracer.rs: 
trace.json: 
trace.paths.json: 

Signed-off-by: Tzanko Matev <[email protected]>

Only trace files in a whitelist (experiment)
codetracer-python-recorder/codetracer_python_recorder/__main__.py: 
codetracer-python-recorder/src/lib.rs: 
codetracer-python-recorder/src/runtime_tracer.rs: 

Signed-off-by: Tzanko Matev <[email protected]>

Author: tzanko-matev

codetracer-python-recorder/Cargo.lock

@@ -20,6 +20,8 @@ runtime_tracing = "0.14.0"
 bitflags = "2.4"
 once_cell = "1.19"
 dashmap = "5.5"
+log = "0.4"
+env_logger = "0.11"
 
 [dev-dependencies]
 pyo3 = { version = "0.25.1", features = ["auto-initialize"] }

codetracer-python-recorder/Cargo.toml

@@ -2,16 +2,15 @@
 
 This module exposes a minimal interface for starting and stopping
 runtime traces. The heavy lifting is delegated to the
-`codetracer_python_recorder` Rust extension which will eventually hook
-into `runtime_tracing` and `sys.monitoring`.  For now the Rust side only
-maintains placeholder state and performs no actual tracing.
+`codetracer_python_recorder` Rust extension which hooks
+into `runtime_tracing` and `sys.monitoring`.
 """
 from __future__ import annotations
 
 import contextlib
 import os
 from pathlib import Path
-from typing import Iterable, Iterator, Optional
+from typing import Iterator, Optional
 
 from .codetracer_python_recorder import (
     flush_tracing as _flush_backend,
@@ -27,31 +26,34 @@
 _active_session: Optional["TraceSession"] = None
 
 
-def _normalize_source_roots(source_roots: Iterable[os.PathLike | str] | None) -> Optional[list[str]]:
-    if source_roots is None:
-        return None
-    return [str(Path(p)) for p in source_roots]
-
-
 def start(
     path: os.PathLike | str,
     *,
     format: str = DEFAULT_FORMAT,
-    capture_values: bool = True,
-    source_roots: Iterable[os.PathLike | str] | None = None,
+    start_on_enter: os.PathLike | str | None = None,
 ) -> "TraceSession":
     """Start a global trace session.
 
-    Parameters mirror the design document.  The current implementation
-    merely records the active state on the Rust side and performs no
-    tracing.
+    - ``path``: Target directory where trace files will be written.
+      Files created: ``trace.json``/``trace.bin``, ``trace_metadata.json``, ``trace_paths.json``.
+    - ``format``: Either ``binary`` or ``json`` (controls events file name/format).
+    - ``start_on_enter``: Optional file path; when provided, tracing remains
+      paused until the tracer observes execution entering this file. Useful to
+      avoid recording interpreter and import startup noise when launching a
+      script via the CLI.
+
+    The current implementation records trace data through a Rust backend.
     """
     global _active_session
     if _is_tracing_backend():
         raise RuntimeError("tracing already active")
 
     trace_path = Path(path)
-    _start_backend(str(trace_path), format, capture_values, _normalize_source_roots(source_roots))
+    _start_backend(
+        str(trace_path),
+        format,
+        str(Path(start_on_enter)) if start_on_enter is not None else None,
+    )
     session = TraceSession(path=trace_path, format=format)
     _active_session = session
     return session
@@ -86,15 +88,11 @@ def trace(
     path: os.PathLike | str,
     *,
     format: str = DEFAULT_FORMAT,
-    capture_values: bool = True,
-    source_roots: Iterable[os.PathLike | str] | None = None,
 ) -> Iterator["TraceSession"]:
     """Context manager helper for scoped tracing."""
     session = start(
         path,
         format=format,
-        capture_values=capture_values,
-        source_roots=source_roots,
     )
     try:
         yield session
@@ -133,11 +131,7 @@ def _auto_start_from_env() -> None:
     if not path:
         return
     fmt = os.getenv("CODETRACER_FORMAT", DEFAULT_FORMAT)
-    capture_env = os.getenv("CODETRACER_CAPTURE_VALUES")
-    capture = True
-    if capture_env is not None:
-        capture = capture_env.lower() not in {"0", "false", "no"}
-    start(path, format=fmt, capture_values=capture)
+    start(path, format=fmt)
 
 
 _auto_start_from_env()

codetracer-python-recorder/codetracer_python_recorder/api.py

@@ -1,36 +1,127 @@
+use std::fs;
+use std::path::{PathBuf, Path};
 use std::sync::atomic::{AtomicBool, Ordering};
+use std::sync::Once;
 
 use pyo3::exceptions::PyRuntimeError;
 use pyo3::prelude::*;
+use std::fmt;
 
 pub mod code_object;
 pub mod tracer;
+mod runtime_tracer;
 pub use crate::code_object::{CodeObjectRegistry, CodeObjectWrapper};
 pub use crate::tracer::{install_tracer, uninstall_tracer, EventSet, Tracer};
 
 /// Global flag tracking whether tracing is active.
 static ACTIVE: AtomicBool = AtomicBool::new(false);
 
-/// Start tracing. Placeholder implementation that simply flips the
-/// global active flag and ignores all parameters.
+// Initialize Rust logging once per process. Defaults to debug for this crate
+// unless overridden by RUST_LOG. This helps surface debug! output during dev.
+static INIT_LOGGER: Once = Once::new();
+
+fn init_rust_logging_with_default(default_filter: &str) {
+    INIT_LOGGER.call_once(|| {
+        let env = env_logger::Env::default().default_filter_or(default_filter);
+        // Use a compact format with timestamps and targets to aid debugging.
+        let mut builder = env_logger::Builder::from_env(env);
+        builder
+            .format_timestamp_micros()
+            .format_target(true);
+        let _ = builder.try_init();
+    });
+}
+
+/// Start tracing using sys.monitoring and runtime_tracing writer.
 #[pyfunction]
 fn start_tracing(
-    _path: &str,
-    _format: &str,
-    _capture_values: bool,
-    _source_roots: Option<Vec<String>>,
+    path: &str,
+    format: &str,
+    activation_path: Option<&str>,
 ) -> PyResult<()> {
-    if ACTIVE.swap(true, Ordering::SeqCst) {
+    // Ensure logging is ready before any tracer logs might be emitted.
+    // Default only our crate to debug to avoid excessive verbosity from deps.
+    init_rust_logging_with_default("codetracer_python_recorder=debug");
+    if ACTIVE.load(Ordering::SeqCst) {
         return Err(PyRuntimeError::new_err("tracing already active"));
     }
-    Ok(())
+
+    // Interpret `path` as a directory where trace files will be written.
+    let out_dir = Path::new(path);
+    if out_dir.exists() && !out_dir.is_dir() {
+        return Err(PyRuntimeError::new_err("trace path exists and is not a directory"));
+    }
+    if !out_dir.exists() {
+        // Best-effort create the directory tree
+        fs::create_dir_all(&out_dir)
+            .map_err(|e| PyRuntimeError::new_err(format!("failed to create trace directory: {}", e)))?;
+    }
+
+    // Map format string to enum
+    let fmt = match format.to_lowercase().as_str() {
+        "json" => runtime_tracing::TraceEventsFileFormat::Json,
+        // Use BinaryV0 for "binary" to avoid streaming writer here.
+        "binary" | "binaryv0" | "binary_v0" | "b0" => runtime_tracing::TraceEventsFileFormat::BinaryV0,
+        //TODO AI! We need to assert! that the format is among the known values.
+        other => {
+            eprintln!("Unknown format '{}', defaulting to binary (v0)", other);
+            runtime_tracing::TraceEventsFileFormat::BinaryV0
+        }
+    };
+    
+    // Build output file paths inside the directory.
+    let (events_path, meta_path, paths_path) = match fmt {
+        runtime_tracing::TraceEventsFileFormat::Json => (
+            out_dir.join("trace.json"),
+            out_dir.join("trace_metadata.json"),
+            out_dir.join("trace_paths.json"),
+        ),
+        _ => (
+            out_dir.join("trace.bin"),
+            out_dir.join("trace_metadata.json"),
+            out_dir.join("trace_paths.json"),
+        ),
+    };
+
+    // Activation path: when set, tracing starts only after entering it.
+    let activation_path = activation_path.map(|s| Path::new(s));
+
+    Python::with_gil(|py| {
+        // Program and args: keep minimal; Python-side API stores full session info if needed
+        let sys = py.import("sys")?;
+        let argv = sys.getattr("argv")?;
+        let program: String = argv
+            .get_item(0)?
+            .extract::<String>()?;
+        //TODO: Error-handling. What to do if argv is empty? Does this ever happen?
+
+        let mut tracer = runtime_tracer::RuntimeTracer::new(
+            &program,
+            &[],
+            fmt,
+            activation_path,
+        );
+
+        // Start location: prefer activation path, otherwise best-effort argv[0]
+        let start_path: &Path = activation_path.unwrap_or(Path::new(&program));
+        tracer.begin(&meta_path, &paths_path, &events_path, start_path, 1)?;
+
+        // Install callbacks
+        install_tracer(py, Box::new(tracer))?;
+        ACTIVE.store(true, Ordering::SeqCst);
+        Ok(())
+    })
 }
 
 /// Stop tracing by resetting the global flag.
 #[pyfunction]
 fn stop_tracing() -> PyResult<()> {
-    ACTIVE.store(false, Ordering::SeqCst);
-    Ok(())
+    Python::with_gil(|py| {
+        // Uninstall triggers finish() on tracer implementation.
+        uninstall_tracer(py)?;
+        ACTIVE.store(false, Ordering::SeqCst);
+        Ok(())
+    })
 }
 
 /// Query whether tracing is currently active.
@@ -39,15 +130,18 @@ fn is_tracing() -> PyResult<bool> {
     Ok(ACTIVE.load(Ordering::SeqCst))
 }
 
-/// Flush buffered trace data. No-op placeholder for now.
+/// Flush buffered trace data (best-effort, non-streaming formats only).
 #[pyfunction]
 fn flush_tracing() -> PyResult<()> {
-    Ok(())
+    Python::with_gil(|py| crate::tracer::flush_installed_tracer(py))
 }
 
 /// Python module definition.
 #[pymodule]
 fn codetracer_python_recorder(_py: Python<'_>, m: &Bound<'_, PyModule>) -> PyResult<()> {
+    // Initialize logging on import so users see logs without extra setup.
+    // Respect RUST_LOG if present; otherwise default to debug for this crate.
+    init_rust_logging_with_default("codetracer_python_recorder=debug");
     m.add_function(wrap_pyfunction!(start_tracing, m)?)?;
     m.add_function(wrap_pyfunction!(stop_tracing, m)?)?;
     m.add_function(wrap_pyfunction!(is_tracing, m)?)?;