Module definition abstraction

[emmatyping]

Python modules rely on a static definition of a module. To avoid needing to have a static, threadsafe PyObject, I propose we use PyModExport, which is the new way of defining modules, and does not rely on the safety of a static, threadsafe PyObject.

An example of a module definition, taken from the above PEP:

static PyModuleDef_Slot examplemodule_slots[] = {
    {Py_mod_name, "examplemodule"},
    {Py_mod_doc, (char*)examplemodule_doc},
    {Py_mod_methods, examplemodule_methods},
    {Py_mod_state_size, (void*)sizeof(examplemodule_state)},
    {Py_mod_exec, (void*)examplemodule_exec},
    {0}
};

PyMODEXPORT_FUNC
PyModExport_examplemodule(void)
{
    return examplemodule_slots;
}

PyMODEXPORT_FUNC is equivalent to *mut PyModuleDef_Slot.

One design I have in mind is something along the lines of:

pub enum PyModuleDefSlot {
    Name(&'static CStr),
    Doc(&'static CStr),
    MultipleInterpreters(*mut c_void),
    GIL(*mut c_void),
    StateSize(usize),
    // ... etc.
}

pub struct PyModuleDef<const N: usize>(UnsafeCell<[ffi::PyModuleDef_Slot; N]>);

impl<const N: usize> PyModuleDef<N> {
    pub const fn new(slots: [PyModuleDefSlot; N]) -> Self {
        todo!()
    }

    pub fn as_mut_ptr(&'static self) -> *mut PyModuleDef_Slot {
        self.0.get() as *mut PyModuleDef_Slot
    }
}

unsafe impl<const N: usize> Sync for PyModuleDef<N> {}

static MODULE: PyModuleDef<5> = PyModule::new([
    PyModuleDefSlot::Name(c"mymodule"),
    PyModuleDefSlot::Doc(c"An example module for a proof of concept"),
    // etc.
]);

#[unsafe(no_mangle)]
unsafe extern "C" fn PyModExport_mymodule() -> *mut PyModuleDef_Slot {
    MODULE.as_mut_ptr()
}

I think this isn't currently sound because we probably need a thread state-aware lock to prevent races on MODULE.as_mut_ptr()?

We will likely also want a design for specifying methods that are part of the module, which are listed in a static array similar to the PyModExport static.

[davidhewitt]

I think this isn't currently sound because we probably need a thread state-aware lock to prevent races on MODULE.as_mut_ptr()?

Given the PyModExport API is brand new, can we ask it to export as *const PyModuleDef_Slot instead? Immutable static data is generally much nicer and it might be the case that it's mutable only because that's less work in C.

cc @encukou

[encukou]

Per the docs, “This array must remain valid and constant until interpreter shutdown.”
Can you treat it as immutable?

[davidhewitt]

We can pretend and cast the constness away, I guess. Is there a reason you didn't define the API to return const PyModuleDef_Slot *?

[encukou]

In C, const is such a weak annotation that I don't really think about it...

[ngoldbaum]

Unless someone beats me to it, I’ll be implementing PEP 793 support in PyO3 sometime early next year to enable support for the new stable ABI. PyO3/pyo3#5644

[emmatyping]

We can pretend and cast the constness away, I guess.

Yeah, I think this is the best solution.

Since the contents will be immutable/constant, we also don't need a lock to be sound.