Rust call multiple python processes

[sun-rs]

I have succeed that create a python extension with Pyo3, and this could send a ptr of apache arrow data form rust to python.

Assume I have a main rust thread which has some big apache arrow data, and I want to share it to several Python interpreter for calculation. Can I use multi processes(GIL not work for thread) in rust to initialize multi independent Python interpreter and share the data ptr? Like a rust call multiple python

fn thread_fn1() {
         
    pyo3::prepare_freethreaded_python();
    for _ in 0..10 {
        Python::with_gil(|py| -> PyResult<()> {
           //get some data ref from rust main thread
        }).unwrap();
    }
}

fn thread_fn2() {
    
    pyo3::prepare_freethreaded_python();
    for _ in 0..10 {
        Python::with_gil(|py| -> PyResult<()> {
           //get some data ref from rust main thread
        }).unwrap();
    }
}

[adamreichold]

If these are single-shot computations, you could fork the main process after receiving the data and do the computations in the child processes. (Just spawning threads is not sufficient as Python will use process-wide data structures and locks like the GIL, so the work would done by thread_fn1 and thread_fn2 would run serially.)

[sun-rs]

If there is a way to fork process in rust, will it means that copy data from parent process to new child process? This could like python multiprocessing to run several pyo3 extension code?

[adamreichold]

There is no built-in safe way to do that, but e.g. the nix crate offers access to most Unix-specific API, including fork.

As with forking in general, the child processes will share the memory of the parent process in a copy-on-write manner, i.e. for read-only access the memory is physically shared but if the child processes write to the memory each one gets its own copy.

Note however, that if the parent has mapped a shared memory region, then the children will inherit this and that region will not be copy-on-write but truly shared between the original producer, the receiver/parent process and the computation/child processes. So you cannot change that shared memory while the child processes are still running.

[mejrs]

Note that rust and forking don't really work well (there is nothing in std to do forking, for example). You can't do multiprocessing with pyo3.

[adamreichold]

I have repeatedly used nix::fork in scientific code to save time on costly initialization procedures without apparent ill effect. It is certainly unsafe and care needs to be taken, but could you explain if there are any inherent limitations why it cannot be used or how using PyO3 in a process would prevent one from forking it?

[mejrs]

I can see how that was confusing. What I meant is that pyo3 doesn't offer an api for doing this.