Problem:
Currently pii_filter is not a subcrate and has plugin has PyO3 dependencies and macros embedded directly. This makes adding new plugins harder and couples plugin logic to Python bindings.

---

Context

As we expand support for both Rust and Python implementations, it would be good to align on testing strategy and plugin architecture to reduce duplication and long-term maintenance cost.

This issue proposes:

• Using Python for integration testing shared across Rust and Python implementations.
• Evaluating three possible plugin architecture approaches, with trade-offs for each.

---

Architecture

Below are three possible architectural approaches for plugins, listed from most Rust-centric to most Python-centric.

---

Option 1: Rust API Built on Top of the Python API

Description
Implement a complete Rust API that wraps or mirrors the Python API. Rust plugins would be written in pure Rust, without pyo3 or maturin.

Pros

• Pure Rust plugin development experience
• No Python bindings required for plugins
• Clean separation between core logic and Python runtime

Cons

• No ability to ship a pip package
• Rust API becomes a public contract that must be maintained indefinitely
• External users may depend on this Rust API (maintenance burden)

This can be seen as either a feature or a liability, depending on how much we want to support external Rust consumers.

---

Option 2: Fully Independent Plugin Crates

Description
Each plugin lives in its own crate and exposes its own #[pyfunction] / #[pymodule] for in-process usage, typically packaged and installed via maturin.
Optionally, a plugin may also expose a gRPC or HTTP interface instead of (or in addition to) Python bindings, if the implementer prefers an out-of-process model. Also helper crates can be developed gRPC crate for example.

Pros

• Clear ownership and isolation per plugin
• Straightforward Python integration for in-process use
• Natural fit for distribution via pip
• Plugin authors can choose between in-process (PyO3) or out-of-process (gRPC/HTTP) execution

Cons

• Each plugin must implement and maintain its own bindings or API surface
• Slightly more boilerplate per plugin
• Mixed execution models may increase overall system complexity

Binding complexity is relatively low based on initial experiments, but repetition remains.

---

Option 3: Hybrid Workspace with a Dedicated Adapter Crate

Description

• Plugins are written as pure Rust subcrates in a workspace
• One or more adapter crates provide integration layers (e.g. PyO3 adapter, gRPC/HTTP adapter)
• Adapters expose a unified interface while plugins remain runtime-agnostic

Pros

• Plugins remain clean, pure Rust
• Centralized and consistent integration layers
• Supports multiple adapters (in-process Python, gRPC/HTTP, etc.) without modifying plugins
• Avoids duplication while keeping plugin crates simple

Cons

• Adapter crates become critical integration points
• Slightly more complex workspace and dependency structure
• Requires careful API design between plugins and adapters

---

Option 4: Only use gRPC or HTTP

Description
Expose plugin functionality over a gRPC or HTTP API, potentially running in a separate process with its own resource management.

Pros

• Clear language and runtime separation
• Plugins can run in an isolated process with dedicated resources
• Easier to integrate with non-Python / non-Rust consumers
• Failure isolation (plugin crashes don’t bring down the host)

Cons

• Added latency compared to in-process calls
• More complex setup and deployment (network API, service lifecycle)
• Increased operational and debugging complexity
• Harder local development experience

Performance impact would need to be evaluated; in some cases isolation and resource control may outweigh the overhead.

---

Currently leaning toward options 2 or 3 and could have different adapter crates one for pyo3 and one for the Grpc giving flexibility.

---

Testing

Proposal

Use Python-based integration tests that are shared between Rust and Python implementations (when both are present).

Rationale

• Integration tests are focused on behavior, not implementation details.
• The code will not be run from native Rust directly anyway.
• A single test suite avoids duplicated effort and reduces drift between implementations.
• Python is already well-suited for higher-level integration testing and orchestration.

Benefits

• One source of truth for integration behavior
• Easier to add new test cases once
• Consistent validation across language boundaries

Potential Drawbacks

• Rust-only contributors may need minimal Python familiarity
• Debugging failures may be slightly less Rust-centric

---

From the Python side, there is also some repetition that could likely be removed. This is something we can revisit and simplify regardless of which architectural option we choose.