Python's datasets load_dataset seems to start many threads which all keep running

[MarcWeber]

If I run the following code with python3.12 I expect to get back the dataset.
It seems to work but it feels like never ending and many threads returning continuing after the load_mnist() call which is unexpected. Any idea what's going on ? The return [] you see makes the problem go away but return no data. So the load_dataset feels like forking the thread many times and each continuing.

use pyo3::prelude::*;
use pyo3::types::IntoPyDict;
use pyo3::ffi::c_str;
use std::ffi::CString;

pub fn load_mnist() -> Vec<(i32, Vec<u8>)> {
    pyo3::prepare_freethreaded_python(); // Moved here
    Python::with_gil(|py| {
        let code = CString::new(r#"
from datasets import load_dataset
import pickle
def get_mnist():
    return []
    dataset = load_dataset("mnist", split="train")
    return [(item["label"], item["image"].tobytes()) for item in dataset]
"#).unwrap();
        let file = CString::new("mnist.py").unwrap();
        let module_name = CString::new("mnist").unwrap();
        
        let module = PyModule::from_code(py, &code, &file, &module_name).unwrap();
        let result: Vec<(i32, Vec<u8>)> = module
            .getattr("get_mnist").unwrap()
            .call0().unwrap()
            .extract().unwrap();
        result
    })
}


Can there be a simple workaround like if therad_id is different kill the thread or is there a solution I missed ?