Race condition with concurrent calls to Python::attach

[alex]

Bug Description

(Found by claude, verified by me)

There is a race condition where a thread can begin executing Python code before Py_InitializeEx() has finished importing site.py, resulting in an incomplete Python environment (missing sys.path entries, etc.).

Root Cause

CPython sets runtime->initialized = 1 (which makes Py_IsInitialized() return true) before importing site.py:

// cpython/Python/pylifecycle.c, init_interp_main()

        interp->runtime->initialized = 1;   // line 1333

    if (config->site_import) {
        status = init_import_site();         // line 1337 — AFTER initialized=1

PyO3's try_attach() (src/internal/state.rs:83-116) trusts Py_IsInitialized() to mean the interpreter is fully initialized and ready for use. It bypasses the std::sync::Once guard in initialize() (src/interpreter_lifecycle.rs:7-20) that serializes the actual Py_InitializeEx() call.

Race Window

1. Thread A calls Python::attach() → try_attach() sees Py_IsInitialized() == 0 → returns NotInitialized → calls ensure_initialized() → enters call_once_force → calls Py_InitializeEx(0).

2. Inside Py_InitializeEx(0), CPython sets runtime->initialized = 1 (line 1333), then begins importing site.py (line 1337).

3. During the site.py import, the GIL can be temporarily released, particularly if site.py executes for long periods (e.g., many .pth files).

4. Thread B calls Python::attach() → try_attach() checks Py_IsInitialized() at state.rs:98 — it returns 1. Thread B proceeds to do_attach_unchecked() → PyGILState_Ensure() → acquires the GIL while site.py has released it.

5. Thread B is now executing Python code while site.py hasn't finished. sys.path is incomplete, , .pth files are only partially processed, etc.

6. Eventually Thread B releases the GIL, site.py resumes and finishes, Py_InitializeEx(0) returns, and Thread A calls PyEval_SaveThread() — but by then Thread B may have already observed or acted on the incomplete state.

Steps to Reproduce

I haven't written a minimal reproducer yet, I've just observed the symptoms and stared hard (with Claude).

Backtrace

Your operating system and version

n/a

Your Python version (python --version)

n/a

Your Rust version (rustc --version)

n/a

Your PyO3 version

masater

How did you install python? Did you use a virtualenv?

n/a

Additional Info

No response

[alex]

Ok, here's a test case:

#![cfg(not(any(PyPy, GraalPy)))]
#![cfg(not(target_arch = "wasm32"))]

//! Regression test: `Python::try_attach()` on a second thread can acquire the
//! GIL while `Py_InitializeEx` is still importing `site.py`, because CPython
//! sets `Py_IsInitialized()` to true before the site import finishes.

use pyo3::prelude::*;

#[test]
fn test_concurrent_init_site_race() {
    let tmpdir = std::env::temp_dir().join("pyo3_test_init_race");
    let _ = std::fs::remove_dir_all(&tmpdir);
    std::fs::create_dir_all(&tmpdir).unwrap();

    std::fs::write(
        tmpdir.join("sitecustomize.py"),
        "import sys, time\n\
         sys._pyo3_site_started = True\n\
         time.sleep(2)\n\
         sys._pyo3_site_done = True\n",
    )
    .unwrap();

    std::env::set_var("PYTHONPATH", &tmpdir);
    std::env::set_var("PYTHONDONTWRITEBYTECODE", "1");

    std::thread::scope(|s| {
        s.spawn(|| {
            Python::initialize();
        });

        s.spawn(|| {
            loop {
                let result = Python::try_attach(|py| {
                    let done = py
                        .import("sys")
                        .unwrap()
                        .getattr("_pyo3_site_done")
                        .ok();
                    assert!(
                        done.is_some(),
                        "Python::try_attach() acquired the GIL while site.py \
                         was still importing — sys._pyo3_site_done was not yet set."
                    );
                });
                if result.is_some() {
                    break;
                }
                std::hint::spin_loop();
            }
        });
    });

    let _ = std::fs::remove_dir_all(&tmpdir);
}

[ngoldbaum]

@alex this was closed by #5903 right?