Making wrapped modules available in a potentially pre-initialised interpreter

[MusicalNinjaDad]

I've been working on testing wrapped functions in rust, without resorting to constantly compiling and installing the python extension module to run tests via pytest.

I hit on snag along the road which forced me to write a little unsafe code and I was hoping that someone with more experience of the workings of pyo3 would be able and willing to take a quick look and let me know your thoughts ...

The problem

When running multiple tests on wrapped modules, interpreter initialisation becomes a problem:

• If the tests are run with auto-initialize on then the interpreter is guaranteed to be initialised and append_to_inittab!() will fail - in this case all tests are guaranteed to panic.
• If the tests are run without auto-initialize then there is the risk that the interpreter will already be initialized by a different test - in this case the tests are simply flaky and will panic at random.

I experienced this issue continually with different fail points but no easy way to guarantee failure.

The solution I found

  Python::with_gil(|py| {
      let sys = PyModule::import_bound(py, "sys").unwrap();
      let py_modules: Bound<'_, PyDict> =
          sys.getattr("modules").unwrap().downcast_into().unwrap();
      let py_adders_pymodule = unsafe { Bound::from_owned_ptr(py, py_adders::__pyo3_init()) };
      py_modules
          .set_item("adders", py_adders_pymodule)
          .expect("Failed to import adders");
...

This is based on the example in the Guide section 3.4 plus looking through the results of cargo expand and the main codebase:

1. sys is imported and sys.modules assigned to a variable
2. the module is initialised (unsafe call to init)
3. the module is inserted into sys.modules

I believe that the lifetime of py_adders_pymodule is safe as:

• py_adders::__pyo3_init() is called with the GIL lock in place.
• Therefore the GIL handler will hand out the same 'py lifetime as the rest of the code is using.
• Bound::from_owned_ptr() is passed the py token from this GIL lock, binding the initialised module to the correct lock
  and the whole process is performed as a single action.

Have I correctly understood how the GIL lock is handled w.r.t. the lifetime of the PyModule created by __pyo3_init?

Note:

This sadly also means that compatibility is limited to cPython>=3.9 if I understand make_module correctly.

Thanks for any advice or comments.

[mejrs]

Can you use https://docs.rs/pyo3/latest/pyo3/macro.append_to_inittab.html ?

[MusicalNinjaDad]

Sadly not, as it requires the interpreter to be un-initialised, otherwise it deliberately panics at Line 209

when running multiple test cases via cargo test there are no guarantees as to when the interpreter is already initialised.

[mejrs]

I see two solutions:

• Run them as integration tests

• Have a global Lazy/Once that calls append_to_inittab and prepare_freethreaded_python, and refer to it in every test. Then only the first test will actually execute it.

[MusicalNinjaDad]

Thanks, That's the conclusion I came to as well. Given the context it's not an option, the unsafe is well protected from the end users - and I think it is safe (hoping for confirmation here)

The context is pyo3-testing which provides a test attribute #[pyo3test] which allows you to shorten your tests to:

#[pyo3test]
#[pyo3import(py_adders: from adders import addone)]
fn test_pyo3test_simple_case() {
    let result: isize = addone!(1);
    assert_eq!(result, 2);
}

and a with_py_raises! macro modelled on pytest's with raises context manager to test for expected Exceptions:

# use pyo3_testing::{pyo3test, with_py_raises};
#[pyo3test]
#[allow(unused_macros)]
#[pyo3import(py_adders: from adders import addone)]
fn test_raises() {
    with_py_raises!(PyTypeError, { addone.call1(("4",)) });
}

[jochemhoorneman]

I cannot answer your question whether you understand correctly that this should be safe. But I just want to say I encountered the same issue and applied your solution, which seems to work perfectly fine.