WS6-4

codetracer-python-recorder/README.md: 
codetracer-python-recorder/codetracer_python_recorder/cli.py: 
codetracer-python-recorder/resources/trace_filters/builtin_default.toml: 
codetracer-python-recorder/src/session/bootstrap.rs: 
codetracer-python-recorder/src/trace_filter/config.rs: 
codetracer-python-recorder/tests/python/test_cli_integration.py: 
design-docs/US0028 - Configurable Python trace filters.md: 
design-docs/adr/0009-configurable-trace-filters.md: 
design-docs/configurable-trace-filters-implementation-plan.md: 
design-docs/configurable-trace-filters-implementation-plan.status.md: 
design-docs/py-api-001.md: 
docs/onboarding/trace-filters.md: 
tf.toml: 
trace-out/trace_metadata.json: 
trace-out/trace_paths.json: 

Signed-off-by: Tzanko Matev <[email protected]>

Author: tzanko-matev

codetracer-python-recorder/README.md

@@ -55,6 +55,7 @@ or activated virtual environments behave identically to `python script.py`.
 - Filter files are TOML with `[meta]`, `[scope]`, and `[[scope.rules]]` tables. Rules evaluate in declaration order and can tweak both execution (`exec`) and value decisions (`value_default`).
 - Supported selector domains: `pkg`, `file`, `obj` for scopes; `local`, `global`, `arg`, `ret`, `attr` for value policies. Match types default to `glob` and also accept `regex` or `literal` (e.g. `local:regex:^(metric|masked)_\w+$`).
 - Default discovery: `.codetracer/trace-filter.toml` next to the traced script. Chain additional files via CLI (`--trace-filter path_a --trace-filter path_b`), environment variable (`CODETRACER_TRACE_FILTER=path_a::path_b`), or Python helpers (`trace(..., trace_filter=[path_a, path_b])`). Later entries override earlier ones when selectors overlap.
+- A built-in `builtin-default` filter is always prepended. It skips CPython standard-library frames (e.g. `asyncio`, `threading`, `importlib`) while re-enabling third-party packages under `site-packages` (except helpers such as `_virtualenv.py`), and redacts common secrets (`password`, `token`, API keys, etc.) across locals/globals/args/returns/attributes. Project filters can loosen or tighten these defaults as required.
 - Runtime metadata captures the active chain under `trace_metadata.json -> trace_filter`, including per-kind redaction counters. See `docs/onboarding/trace-filters.md` for the full DSL reference and examples.
 
 Example snippet:

codetracer-python-recorder/codetracer_python_recorder/cli.py

@@ -3,6 +3,7 @@
 
 import argparse
 import json
+import os
 import runpy
 import sys
 from dataclasses import dataclass
@@ -11,6 +12,7 @@
 from typing import Iterable, Sequence
 
 from . import flush, start, stop
+from .auto_start import ENV_TRACE_FILTER
 from .formats import DEFAULT_FORMAT, SUPPORTED_FORMATS, normalize_format
 
 
@@ -252,6 +254,9 @@ def main(argv: Iterable[str] | None = None) -> int:
     script_path = config.script
     script_args = config.script_args
     filter_specs = list(config.trace_filter)
+    env_filter = os.getenv(ENV_TRACE_FILTER)
+    if env_filter:
+        filter_specs.insert(0, env_filter)
     policy_overrides = config.policy_overrides if config.policy_overrides else None
 
     old_argv = sys.argv

codetracer-python-recorder/resources/trace_filters/builtin_default.toml

@@ -0,0 +1,63 @@
+[meta]
+name = "builtin-default"
+version = 1
+description = "Skip CPython stdlib internals and redact common sensitive identifiers."
+labels = ["builtin", "default"]
+
+[scope]
+default_exec = "trace"
+default_value_action = "allow"
+
+[[scope.rules]]
+selector = 'file:regex:.*[\\/](lib|Lib)[\\/]python\d+\.\d+[\/].*'
+exec = "skip"
+reason = "Skip Python standard library files"
+
+[[scope.rules]]
+selector = 'file:regex:.*[\\/](lib|Lib)[\\/]python\d+\.\d+[\/]site-packages[\/].*'
+exec = "trace"
+reason = "Allow third-party packages under site-packages"
+
+[[scope.rules]]
+selector = 'file:regex:.*[\\/]site-packages[\\/]_virtualenv\.py$'
+exec = "skip"
+reason = "Skip virtualenv bootstrap helper"
+
+[[scope.rules]]
+selector = 'pkg:regex:^(asyncio|selectors|concurrent|importlib|threading|multiprocessing)(\.|$)'
+exec = "skip"
+reason = "Skip noisy stdlib async/concurrency internals"
+
+[[scope.rules]]
+selector = 'pkg:literal:builtins'
+exec = "skip"
+reason = "Skip builtins module instrumentation"
+
+[[scope.rules]]
+selector = 'pkg:glob:*'
+value_default = "allow"
+
+[[scope.rules.value_patterns]]
+selector = 'local:regex:(?i).*(pass(word)?|passwd|pwd|secret|token|session|cookie|auth|credential|creds|bearer|ssn|credit|card|iban|cvv|cvc|pan|api[_-]?key|private[_-]?key|secret[_-]?key|ssh[_-]?key|jwt|refresh[_-]?token|access[_-]?token).*'
+action = "deny"
+reason = "Redact sensitive locals"
+
+[[scope.rules.value_patterns]]
+selector = 'global:regex:(?i).*(pass(word)?|passwd|pwd|secret|token|session|cookie|auth|credential|creds|bearer|ssn|credit|card|iban|cvv|cvc|pan|api[_-]?key|private[_-]?key|secret[_-]?key|ssh[_-]?key|jwt|refresh[_-]?token|access[_-]?token).*'
+action = "deny"
+reason = "Redact sensitive globals"
+
+[[scope.rules.value_patterns]]
+selector = 'arg:regex:(?i).*(pass(word)?|passwd|pwd|secret|token|session|cookie|auth|credential|creds|bearer|ssn|credit|card|iban|cvv|cvc|pan|api[_-]?key|private[_-]?key|secret[_-]?key|ssh[_-]?key|jwt|refresh[_-]?token|access[_-]?token).*'
+action = "deny"
+reason = "Redact sensitive arguments"
+
+[[scope.rules.value_patterns]]
+selector = 'ret:regex:(?i).*(pass(word)?|passwd|pwd|secret|token|session|cookie|auth|credential|creds|bearer|ssn|credit|card|iban|cvv|cvc|pan|api[_-]?key|private[_-]?key|secret[_-]?key|ssh[_-]?key|jwt|refresh[_-]?token|access[_-]?token).*'
+action = "deny"
+reason = "Redact sensitive return values"
+
+[[scope.rules.value_patterns]]
+selector = 'attr:regex:(?i).*(pass(word)?|passwd|pwd|secret|token|session|cookie|auth|credential|creds|bearer|ssn|credit|card|iban|cvv|cvc|pan|api[_-]?key|private[_-]?key|secret[_-]?key|ssh[_-]?key|jwt|refresh[_-]?token|access[_-]?token).*'
+action = "deny"
+reason = "Redact sensitive attributes"

codetracer-python-recorder/src/session/bootstrap.rs

@@ -33,6 +33,9 @@ pub struct TraceSessionBootstrap {
 
 const TRACE_FILTER_DIR: &str = ".codetracer";
 const TRACE_FILTER_FILE: &str = "trace-filter.toml";
+const BUILTIN_FILTER_LABEL: &str = "builtin-default";
+const BUILTIN_TRACE_FILTER: &str =
+    include_str!("../../resources/trace_filters/builtin_default.toml");
 
 impl fmt::Debug for TraceSessionBootstrap {
     fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
@@ -173,11 +176,10 @@ fn load_trace_filter(
         chain.extend(paths.iter().cloned());
     }
 
-    if chain.is_empty() {
-        return Ok(None);
-    }
-
-    let config = TraceFilterConfig::from_paths(&chain)?;
+    let config = TraceFilterConfig::from_inline_and_paths(
+        &[(BUILTIN_FILTER_LABEL, BUILTIN_TRACE_FILTER)],
+        &chain,
+    )?;
     Ok(Some(Arc::new(TraceFilterEngine::new(config))))
 }
 
@@ -234,6 +236,7 @@ mod tests {
     use super::*;
     use pyo3::types::PyList;
     use recorder_errors::ErrorCode;
+    use std::path::PathBuf;
     use tempfile::tempdir;
 
     #[test]
@@ -347,7 +350,7 @@ mod tests {
     }
 
     #[test]
-    fn prepare_bootstrap_ignores_missing_trace_filter() {
+    fn prepare_bootstrap_applies_builtin_trace_filter() {
         Python::with_gil(|py| {
             let tmp = tempdir().expect("tempdir");
             let trace_dir = tmp.path().join("out");
@@ -365,7 +368,13 @@ mod tests {
                 .expect("restore argv");
 
             let bootstrap = result.expect("bootstrap");
-            assert!(bootstrap.trace_filter().is_none());
+            let engine = bootstrap.trace_filter().expect("builtin filter");
+            let summary = engine.summary();
+            assert_eq!(summary.entries.len(), 1);
+            assert_eq!(
+                summary.entries[0].path,
+                PathBuf::from("<inline:builtin-default>")
+            );
         });
     }
 
@@ -416,8 +425,12 @@ mod tests {
             let bootstrap = result.expect("bootstrap");
             let engine = bootstrap.trace_filter().expect("filter engine");
             let summary = engine.summary();
-            assert_eq!(summary.entries.len(), 1);
-            assert_eq!(summary.entries[0].path, filter_path);
+            assert_eq!(summary.entries.len(), 2);
+            assert_eq!(
+                summary.entries[0].path,
+                PathBuf::from("<inline:builtin-default>")
+            );
+            assert_eq!(summary.entries[1].path, filter_path);
         });
     }
 
@@ -494,9 +507,13 @@ mod tests {
             let bootstrap = result.expect("bootstrap");
             let engine = bootstrap.trace_filter().expect("filter engine");
             let summary = engine.summary();
-            assert_eq!(summary.entries.len(), 2);
-            assert_eq!(summary.entries[0].path, default_filter_path);
-            assert_eq!(summary.entries[1].path, override_filter_path);
+            assert_eq!(summary.entries.len(), 3);
+            assert_eq!(
+                summary.entries[0].path,
+                PathBuf::from("<inline:builtin-default>")
+            );
+            assert_eq!(summary.entries[1].path, default_filter_path);
+            assert_eq!(summary.entries[2].path, override_filter_path);
         });
     }
 }

codetracer-python-recorder/src/trace_filter/config.rs

@@ -149,14 +149,27 @@ pub struct TraceFilterConfig {
 impl TraceFilterConfig {
     /// Load and compose filters from the provided paths.
     pub fn from_paths(paths: &[PathBuf]) -> RecorderResult<Self> {
-        if paths.is_empty() {
+        Self::from_inline_and_paths(&[], paths)
+    }
+
+    /// Load and compose filters from inline TOML sources combined with paths.
+    ///
+    /// Inline entries are ingested first in the order provided, followed by files.
+    pub fn from_inline_and_paths(
+        inline: &[(&str, &str)],
+        paths: &[PathBuf],
+    ) -> RecorderResult<Self> {
+        if inline.is_empty() && paths.is_empty() {
             return Err(usage!(
                 ErrorCode::InvalidPolicyValue,
-                "no trace filter paths supplied"
+                "no trace filter sources supplied"
             ));
         }
 
         let mut aggregator = ConfigAggregator::default();
+        for (label, contents) in inline {
+            aggregator.ingest_inline(label, contents)?;
+        }
         for path in paths {
             aggregator.ingest_file(path)?;
         }
@@ -225,8 +238,17 @@ impl ConfigAggregator {
             )
         })?;
 
-        let checksum = calculate_sha256(&contents);
-        let raw: RawFilterFile = toml::from_str(&contents).map_err(|err| {
+        self.ingest_source(path, &contents)
+    }
+
+    fn ingest_inline(&mut self, label: &str, contents: &str) -> RecorderResult<()> {
+        let pseudo_path = PathBuf::from(format!("<inline:{label}>"));
+        self.ingest_source(&pseudo_path, contents)
+    }
+
+    fn ingest_source(&mut self, path: &Path, contents: &str) -> RecorderResult<()> {
+        let checksum = calculate_sha256(contents);
+        let raw: RawFilterFile = toml::from_str(contents).map_err(|err| {
             usage!(
                 ErrorCode::InvalidPolicyValue,
                 "failed to parse trace filter '{}': {}",
@@ -771,6 +793,7 @@ const VALUE_SELECTOR_KINDS: [SelectorKind; 5] = [
 mod tests {
     use super::*;
     use std::io::Write;
+    use std::path::PathBuf;
     use tempfile::tempdir;
 
     #[test]
@@ -870,6 +893,30 @@ mod tests {
         Ok(())
     }
 
+    #[test]
+    fn from_inline_and_paths_parses_inline_only() -> RecorderResult<()> {
+        let inline_filter = r#"
+            [meta]
+            name = "inline"
+            version = 1
+
+            [scope]
+            default_exec = "trace"
+            default_value_action = "allow"
+        "#;
+
+        let config = TraceFilterConfig::from_inline_and_paths(&[("inline", inline_filter)], &[])?;
+
+        assert_eq!(config.default_exec(), ExecDirective::Trace);
+        assert_eq!(config.default_value_action(), ValueAction::Allow);
+        assert_eq!(config.rules().len(), 0);
+        let summary = config.summary();
+        assert_eq!(summary.entries.len(), 1);
+        assert_eq!(summary.entries[0].name, "inline");
+        assert_eq!(summary.entries[0].path, PathBuf::from("<inline:inline>"));
+        Ok(())
+    }
+
     #[test]
     fn rejects_unknown_keys() {
         let temp = tempdir().expect("temp dir");

codetracer-python-recorder/tests/python/test_cli_integration.py

@@ -137,6 +137,48 @@ def test_cli_honours_trace_filter_chain(tmp_path: Path) -> None:
     filters = trace_filter.get("filters", [])
     paths = [entry.get("path") for entry in filters if isinstance(entry, dict)]
     assert paths == [
+        "<inline:builtin-default>",
         str(default_filter.resolve()),
         str(override_filter.resolve()),
     ]
+
+
+def test_cli_honours_env_trace_filter(tmp_path: Path) -> None:
+    script = tmp_path / "program.py"
+    _write_script(script, "print('env filter test')\n")
+
+    filter_path = tmp_path / "env-filter.toml"
+    filter_path.write_text(
+        """
+        [meta]
+        name = "env-filter"
+        version = 1
+
+        [scope]
+        default_exec = "trace"
+        default_value_action = "allow"
+
+        [[scope.rules]]
+        selector = "pkg:program"
+        exec = "skip"
+        value_default = "allow"
+        """,
+        encoding="utf-8",
+    )
+
+    trace_dir = tmp_path / "trace"
+    env = _prepare_env()
+    env["CODETRACER_TRACE_FILTER"] = str(filter_path)
+
+    result = _run_cli(["--trace-dir", str(trace_dir), str(script)], cwd=tmp_path, env=env)
+    assert result.returncode == 0
+
+    metadata_file = trace_dir / "trace_metadata.json"
+    payload = json.loads(metadata_file.read_text(encoding="utf-8"))
+    trace_filter = payload.get("trace_filter", {})
+    filters = trace_filter.get("filters", [])
+    paths = [entry.get("path") for entry in filters if isinstance(entry, dict)]
+    assert paths == [
+        "<inline:builtin-default>",
+        str(filter_path.resolve()),
+    ]

design-docs/US0028 - Configurable Python trace filters.md

@@ -36,7 +36,7 @@ As a **Python team lead**, I want **a powerful configuration language to filter
 - [ ] Scenario: Default filter protects secrets  
   - Given no filter file is provided  
   - When the recorder starts  
-  - Then a built-in best-effort secret redaction policy is applied and the user is notified how to supply a project-specific filter
+  - Then a built-in best-effort secret redaction policy is applied, standard-library/asyncio frames are skipped, and the user is notified how to supply a project-specific filter
 - [ ] Scenario: Validate configuration errors  
   - Given I supply an invalid rule (e.g., circular include)  
   - When I launch the recorder  

design-docs/adr/0009-configurable-trace-filters.md

@@ -29,6 +29,7 @@ The solution has to load human-authored TOML, enforce schema validation, and add
    - Resolve `inherit` defaults while chaining multiple files (split on `::`). Later files append to the ordered rule list; `value_patterns` are likewise appended.
 2. **Expose filter loading at session bootstrap.**
    - Extend `TraceSessionBootstrap` to locate the default project filter (`<cwd>/.codetracer/trace-filter.toml` up the directory tree) and accept optional override specs from CLI, Python API, or env (`CODETRACER_TRACE_FILTER`).
+   - Prepend a bundled `builtin-default` filter that redacts common secrets and skips CPython standard-library/asyncio frames before applying project/user filters.
    - Parse each provided file once per `start_tracing` call. Propagate `RecorderError` on IO or schema failures with context about the offending selector.
 3. **Wire the engine into `RuntimeTracer`.**
    - Store `Arc<TraceFilterEngine>` plus a per-code cache of `ResolvedScope` decisions (`HashMap<usize, ScopeResolution>`). Each resolution records:

design-docs/configurable-trace-filters-implementation-plan.md

@@ -88,6 +88,7 @@ Related ADR: 0009 – Configurable Trace Filters for codetracer-python-recorder
 - Update `#[pyfunction] start_tracing` signature with `#[pyo3(signature = (path, format, activation_path=None, trace_filter=None))]`.
   - Parse `trace_filter` (string/path) into `FilterSpec`, split on `::`, resolve to absolute paths, and feed into loader. Map errors via `RecorderError`.
 - Extend `TraceSessionBootstrap` (or adjacent helper) to find the default `<project>/.codetracer/trace-filter.toml` by walking up from the script path when no explicit spec is provided.
+- Prepend a built-in default filter (shipped with the crate) that redacts common secrets and skips standard-library/asyncio frames before applying project/user filters.
 - Modify `session.start` and `.trace` to accept `trace_filter` keyword; wrap `pathlib.Path` inputs.
 - CLI:
   - Add `--trace-filter path` (repeatable). When multiple provided, respect CLI order; combine with default using `::`.

design-docs/configurable-trace-filters-implementation-plan.status.md

@@ -23,6 +23,7 @@
 - `codetracer-python-recorder/src/lib.rs`
 - `codetracer-python-recorder/benches/trace_filter.rs` *(WS6 microbench harness)*
 - `Justfile` *(WS6 bench automation)*
+- `codetracer-python-recorder/resources/trace_filters/builtin_default.toml` *(WS6 builtin defaults)*
 - Future stages: `codetracer-python-recorder/src/runtime/mod.rs`, Python surface files under `codetracer_python_recorder/`
 
 ## Stage Progress
@@ -31,7 +32,7 @@
 - ✅ **WS3 – Runtime Engine & Caching:** Implemented `trace_filter::engine` with `TraceFilterEngine::resolve` caching `ScopeResolution` entries per code id (DashMap), deriving module/object/file metadata, and compiling value policies with ordered pattern evaluation. Added `ValueKind` to align future runtime integration and unit tests proving caching, rule precedence (object > package/file), and relative path normalisation—all exercised via `just cargo-test`.
 - ✅ **WS4 – RuntimeTracer Integration:** `RuntimeTracer` now accepts an optional `Arc<TraceFilterEngine>`, caches `ScopeResolution` results per code id, and records `filter_scope_skip` when scopes are denied. Value capture helpers honour `ValuePolicy` with a reusable `<redacted>` sentinel, emit per-kind telemetry, and we persist the active filter summary plus skip/redaction counts into `trace_metadata.json`. Bootstrapping now discovers `.codetracer/trace-filter.toml`, instantiates `TraceFilterEngine`, and passes the shared `Arc` into `RuntimeTracer::new`; new `session::bootstrap` tests cover both presence/absence of the default filter and `just cargo-test` (nextest `--no-default-features`) confirms the flow end-to-end.
 - ✅ **WS5 – Python Surface, CLI, Metadata:** Session helpers normalise chained specs, auto-start honours `CODETRACER_TRACE_FILTER`, PyO3 merges explicit/default chains, CLI exposes `--trace-filter`, unit coverage exercises env auto-start filter chaining, and docs/CLI help now describe filter precedence and env wiring.
-- ✅ **WS6 – Hardening, Benchmarks & Documentation:** Completed selector error logging hardening, delivered Rust + Python benchmarking harnesses with `just bench` automation, refreshed the Nix dev shell (gnuplot) to keep Criterion plots available, and closed documentation gaps (README, onboarding guide). Follow-on benchmarking integration tasks are tracked under ADR 0010.
+- ✅ **WS6 – Hardening, Benchmarks & Documentation:** Completed selector error logging hardening, introduced a built-in default filter that redacts sensitive identifiers and skips stdlib/asyncio frames, delivered Rust + Python benchmarking harnesses with `just bench` automation, refreshed the Nix dev shell (gnuplot) to keep Criterion plots available, and closed documentation gaps (README, onboarding guide). Follow-on benchmarking integration tasks are tracked under ADR 0010.
 
 ## WS5 Progress Checklist
 1. ✅ Introduced Python-side helpers that normalise `trace_filter` inputs (strings, Paths, iterables) into absolute path chains, updated session API/context manager, and threaded env-driven auto-start.