Certain operations just spawn a new process

[SeanPedersen]

Hey,
I have been playing around with PyO3 a bit and enjoy it if it would not be so unstable. Certain Python module functions cause it to just spawn a new process...

I noticed it for: transformers model execution, tqdm and openTSNE

I also found this related issue: https://stackoverflow.com/questions/77522832/call-to-python-class-constructor-from-rust-with-pyo3-spawns-new-processes

Any idea how to deal / fix it gracefully?

Example code (resulting in permanent starting of new process...):

main.py:

from openTSNE import TSNE
import numpy as np

def go():
    try:
        x_train = np.random.rand(1000, 100)
        tsne = TSNE(
            perplexity=30,
            metric="euclidean",
            n_jobs=8,
            random_state=42,
            verbose=True,
        )
        embedding_train = tsne.fit(x_train)
        print(embedding_train.shape)
    except Exception as e:
        print(e)

if __name__ == "__main__":
    go()

main.rs:

use pyo3::{
    ffi::c_str,
    types::{PyAnyMethods, PyModule},
    Py, PyAny, PyResult, Python,
};

fn main() {
    println!("Hello, worldzz!!!aa");
    let py_server = c_str!(include_str!("../main.py"));

    let _from_python = Python::with_gil(|py| -> PyResult<Py<PyAny>> {
        let _ = py.import("numpy").expect("Error: import numpy");
        let app: Py<PyAny> = PyModule::from_code(py, py_server, c_str!(""), c_str!(""))?
            .getattr("go")?
            .into();
        app.call0(py)
    });
}

[SeanPedersen]

Found a fix, just add:

        let multiprocessing = py.import("multiprocessing")?;
        multiprocessing
            .getattr("set_start_method")?
            .call1(("fork",))?;

like this:

let _from_python = Python::with_gil(|py| -> PyResult<Py<PyAny>> {
        let multiprocessing = py.import("multiprocessing")?;
        multiprocessing
            .getattr("set_start_method")?
            .call1(("fork",))?;
        let _ = py.import("numpy").expect("Error: import numpy");
        let app: Py<PyAny> = PyModule::from_code(py, py_server, c_str!(""), c_str!(""))?
            .getattr("go")?
            .into();
        app.call0(py)
    });