How to share Python code between multiple packages without needing to publish the code as a library

[gaborbernat]

So this question is a variation of the monorepo support. Imagine you have a project that generates 2 or more (e.g. A, B) distinct maturin projects. However, there's some common code you want to share between them:

.
├──  Cargo.lock
├──  Cargo.toml
├── rust-common
│   ├──  Cargo.toml
│   ├──  src
│   └── target
├── py-common
│   ├──  CHANGELOG.md
│   ├──  pyproject.toml
│   └──  src
├── A
│   ├──  Cargo.toml
│   ├──  pyproject.toml
│   ├── rust
│   └── src
├── B
│   ├──  Cargo.toml
│   ├──  pyproject.toml
│   ├── rust
│   └── src
└──  rust-toolchain.toml

Sharing the rust code (hosted here in rust-common without needing to publish the crate is possible via cargo allowing relative paths for dependencies and rust has a virtual workspace feature. The question is: How can you share the Python code? The Python part of PEP-621 does not allow relative paths in dependencies.

Ideally, I would be able to take the content of src from py-common and mount it (copy it, symlink it) under src/a/py-common. Something like:

[tool.maturin]
python-dependency-local = ['../py-common']

Which would then cause:

• sdist (no op?)
• wheel - build a wheel of py-common, take the non dist-info part, and copy it within the projects package,
• editable - mark the project as an editable dependency.

Things I have tried:

• using python-packages with a relative path, but this does not make sense because is unclear where it should live.
• creating symlink (maturin seems to ignore the symlink part of the wheel build (but not sdist)).

Example project tox-dev/toml-fmt#12, in this case trying to share py-common under pyproject-fmt, without needing to publish the py-common to PyPI.

[messense]

Does any other general purpose python packaging tools support this kind of setup?

[messense]

Since there is no standard, maybe it'd be good enough if we can make the symlink solution work for starters?

[gaborbernat]

That can be a reasonable stop gap, but would mean Windows support would be buggy out of box; so for that reason would suggest the python-dependency-local solution that is more portable, and we can use PTH files for editable wheels with just normal copy for wheel builds.

[konstin]

Python is not really local dependency friendly (other than requirement files perhaps).

uv, pdm and hatch all have some form local python dependency support, would that work for you or am i misunderstanding your problem?

[gaborbernat]

It would work. But I need it for maturin.

[konstin]

We're running into a combination of two major problems i think:

• editables and relative paths missing in PEP 621
• missing options in PEP 517 / Lack of a plugin interface for package managers, so it's inconvenient to use maturin through another frontend.