What's the correct way to get dotted import paths working with declarative submodules?

[j-g00da]

I'm working on a project a that uses nested declarative modules.
The project is generally all-rust but uses mixed layout in order to provide .pyi stubs for nested modules (with python sources under python/a).

Here's a minimal example that demonstrates the problem:

// lib.rs
#[pymodule(name = "a")]
mod py_a {
    #[pymodule_export]
    use crate::b::py_b;
}

// b.rs
#[pyclass]
struct Example;

#[pymethods]
impl Example {
    // ...
}

#[pymodule(name = "b", submodule)]
pub mod py_b {
    #[pymodule_export]
    use super::Example;
}

A resulting class Example can be successfully imported this way:

from a import b

b.Example()

But cannot be imported directly using a dotted path to submodule:

from a.b import Example

My current workaround is to explicitly add module contents to globals in a python file for each submodule, e.g. b.py:

from .a import b as _mod

__all__ = _mod.__all__
for name in getattr(_mod, "__all__"):
    globals()[name] = getattr(_mod, name)

This is less than ideal.

I see that in maturin-starter example, this is achieved this way:

    // Inserting to sys.modules allows importing submodules nicely from Python
    // e.g. from maturin_starter.submodule import SubmoduleClass

    let sys = PyModule::import(py, "sys")?;
    let sys_modules: Bound<'_, PyDict> = sys.getattr("modules")?.cast_into()?;
    sys_modules.set_item("maturin_starter.submodule", m.getattr("submodule")?)?;

Is something similar possible using declarative approach?

(There's a similar discussion about submodule issues already but it's two years old.)

[Icxolu]

Yes, you can use #[pymodule_init] to apply the same trick.

#[pymodule_init]
fn init(m: &Bound<'_, PyModule>) -> PyResult<()> {
    let modules = PyModule::import(m.py(), "sys")?.getattr("modules")?;
    modules.set_item("maturin_starter.submodule", m.getattr("submodule")?)?;
    Ok(())
}

[j-g00da]

Thanks, btw. is there a reason this is not the default? Are there some plans to make it an attribute or simplify it in any way?

I think that having to add 6 lines of boilerplate to each submodule in order to make imports work properly is less than ideal.

[davidhewitt]

I completely agree this should work as the default; the module system is somewhat due a refactor in its internals to make this work. The first step along that way is to catch up to CPython best practice in #5142