Calling python from rust threads

[dtheodor]

I'm trying to understand how and if rust threading works with passing a python object to different rust threads and calling the python object's methods.

I am not interested in bypassing the GIL, asyncio, or anything fancy. I just have an old school python object, which I instantiate once, and then I pass around to several threads, which use its methods. Exactly as I would do with pure python and the python threading module. It's up to the object to be thread-safe.

Only now my threads are rust threads, my python object is a Py<PyAny> which has been instantiated once and .clone()'d to pass to each thread, which call its methods with Python::with_gil(|py| ...).

In pure python land this would work fine (with parallelism achieved through Py_BEGIN_ALLOW_THREADS). Does this work the same way through pyo3 or am I missing something?

[adamreichold]

Generally speaking this works with PyO3 just the same where Python::allow_threads temporarily releases the GIL if you are currently holding it and Python::with_gil acquires it if you need to access a Python object from any of your threads. Of course, the amount of parallelism you can achieve this way depends on how often/long your Rust threads actually need to acquire/hold the GIL.

[adamreichold]

No, with_gil is just a safe Rust API on top of the usual Python mechanisms. The docs have some more details at https://docs.rs/pyo3/latest/pyo3/marker/struct.Python.html, but for example

let obj: Py<PyAny> = ...;

Python::with_gil(|py| {
   {
       // I have the GIL and can access the object
       let obj: &PyAny = obj.as_ref(py);
   }

   Python::allow_threads(move || {
       // I do not and hence cannot do much with `obj` because `py` cannot be passed in here
   });
});

illustrates that you can release the GIL from within with_gil. (The scope based API basically ensures proper nesting of acquiring/releasing the GIL which we need for automatically managing GIL-bound memory.)

[dtheodor]

Thank you.

[dtheodor]

You mention Python::allow_threads, which I understand is the equivalent of what Py_BEGIN_ALLOW_THREADS does when writing Python extensions in C. But this is relevant for when you call Rust from Python. My case is the opposite direction, I am only calling Python from Rust. I have several Rust threads which will in-parallel run Python::with_gil(...), and call some Python function. I know that the Python function internally will release the GIL, so I know I would have (some degree of) parallelism if this were pure Python threading. Does the fact that this happens from Rust threads all running Python::with_gil(...) hamper parallelism?

[adamreichold]

Does the fact that this happens from Rust threads all running Python::with_gil(...) hamper parallelism?

No, same as with_gil, allow_threads is just a safe Rust API around the same PyEval_Save/RestoreThread API backing Py_BEGIN_ALLOW_THREADS. I used allow_threads to keep the example short, but it does not matter whether Python or Rust or C code release the GIL inside with_gil (because that also just wraps the C API).