GIL Isolation Problem in Multi-threaded Node Simulation with PyO3

[cyrus937]

Context

I'm building a multi-node network simulation where each node runs independently and can execute Python applications through PyO3. The architecture involves:

• Multiple VNode instances running in separate Tokio tasks/threads
• Python applications registered on each node via PyObject instances
• Asynchronous message passing between nodes
• Periodic execution of Python methods (some_method, etc.)

#[derive(Clone)]
pub struct Application {
    pub app_instance: PyObject,
    // ... other fields
}

impl Application {
    pub fn call_periodic_run(&self, node: &VNode) -> PyResult<()> {
        Python::with_gil(|py| {
            let _ = py.allow_threads(|| {
                Python::with_gil(|py| {
                    if let Err(e) = self.app_instance
                        .call_method1(py, "some_method", (node.id.to_string(),)) {
                        error!("Error calling some_method on app {}: {}", self.id, e);
                    }
                })
            });
            Ok(())
        })
    }
}

The Problem

When multiple nodes execute their Python applications simultaneously across different threads, I'm experiencing:

1. GIL Contention: All Python calls are serialized through the GIL, creating a bottleneck
2. Potential Interference: Python applications from different nodes might interfere with each other's global state
3. Performance Degradation: The simulation doesn't scale with the number of nodes due to GIL serialization

Specific Questions

1. Is there a way to create isolated Python execution contexts for each node to prevent cross-contamination of Python state?
2. What's the recommended approach for calling Python methods from multiple Rust threads without creating a bottleneck at the GIL?
3. Should I consider using multiple Python sub-interpreters (if available in PyO3) to achieve better isolation?
4. Are there PyO3 patterns or best practices for this type of multi-node simulation where multiple Python processes need to run independently?

Current Workaround Attempt

I've tried using py.allow_threads() with nested Python::with_gil(), but this doesn't seem to provide the isolation I need:

Python::with_gil(|py| {
    let _ = py.allow_threads(|| {
        Python::with_gil(|py| {
            // Python method call here
        })
    });
    Ok(())
})

[davidhewitt]

To your questions:

1. "isolated Python contexts" it sounds like what you might want is subinterpreters, as per your (3). They have historically been an esoteric feature in CPython however getting more support, however we don't yet have a path to support them in PyO3 - Support sub-interpreters #576 . Can you make the state encapsulated onto a class so you don't need truly "global" state?

2. If you want to avoid the GIL you should look at Python 3.14's free threaded Python.

3. As per 1.

4. I'm not sure I have anything here I would define as "best practice", I think it's quite context depended. For example subprocesses might be acceptable to you if strong isolation is a hard requirement and the serialization costs are not prohibitive?