A single library crate for Rust and Python

[rgreinho]

We have a Rust workspace with several projects in it. One of the projects is named "core" and contains all the common pieces between the different projects, including our models. We want to perform some data science using pandas, etc. Therefore we decided to use PyO3 to export our models to a Python module.

Following the guides and tutorial, everything worked like a charm by simply adding annotations to the models with the appropriate proc macros, and we now have a Python module built with maturin that allows us to work with the Rust models in Python!

However, now the "core" crate does not compile with cargo build anymore, AND neither does any of the other crates from the workspace. We are hitting the linking problem described in some issues in this repo. The tests and examples don't work either anymore.

Looking as other discussions and going through the issues, we thought we would be able to make it work with the workarounds like specifying 2 crate types, or splitting the features:

[lib]
name = "core"
crate-type = ["cdylib", "rlib"]

[features]
default = ["extension-module"]
extension-module = ["pyo3/extension-module"]

But it did not help.

Therefore we are wondering if we are doing correctly? 🤔 Is it possible to have just one crate for both the rust lib and the python extensions? Or are we supposed to create different 2 crates?

The closest answer we found was this one: https://users.rust-lang.org/t/wrap-rust-library-for-python-without-changing-rust-library/69285/4. But like the author of the question mentioned in his last message, we though that recreating wrapping structs and wrapping functions was not the ideal path.

We would appreciate if anybody could provide some guidance/best practices for this kind of effort.

[messense]

Try remove default = ["extension-module"] and specify the feature in pyproject.toml instead:

[tool.maturin]
cargo-extra-args = "--features extension-module"

[rgreinho]

Fantastic! That did the trick and now everything works again! 🎉 Thank you very much for the quick answer.

[rgreinho]

Actually I was a bit too quick to answer :( While this did solve the problems for the "core" crate, the other members of the workspace still don't compile due to the linking error.

[messense]

Can you share the linking error? And a minimal reproducer would be very helpful.

[rgreinho]

I think we had a bad configuration option somewhere. I am not entirely sure what the problem was exactly, but after a deep clean of the repo (git clean -ffdx) and removing/adjusting some configuration flags here and there we got it to work!

So I am putting our reproducer here, as a small example which works.

Project structure:

.
├── Cargo.toml
├── compute
│   ├── Cargo.toml
│   └── src
└── core
    ├── Cargo.toml
    ├── pyproject.toml
    └── src

Cargo.toml:

[workspace]
members = [
  "compute",
  "core",
]

core/Cargo.toml:

[package]
name = "core"
version = "0.1.0"
edition = "2021"

[lib]
name = "core"
crate-type = ["cdylib","rlib"]

[dependencies]
pyo3 = { version = "0.16.3" }

[features]
extension-module = ["pyo3/extension-module"]

core/pyproject.toml:

[build-system]
requires = ["maturin>=0.12,<0.13"]
build-backend = "maturin"

[project]
name = "core"
requires-python = ">=3.6"
classifiers = [
    "Programming Language :: Rust",
    "Programming Language :: Python :: Implementation :: CPython",
    "Programming Language :: Python :: Implementation :: PyPy",
]

[tool.maturin]
cargo-extra-args = "--features extension-module"

core/src/lib.rs:

use pyo3::prelude::*;

#[pyclass]
#[derive(Debug)]
pub struct User {
    #[pyo3(get, set)]
    name: String,
    #[pyo3(get, set)]
    age: u8,
}

#[pymethods]
impl User {
    #[new]
    pub fn new(name: &str, age: u8) -> Self {
        User {
            name: name.into(),
            age,
        }
    }
}

#[pymodule]
fn core(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_class::<User>()?;
    Ok(())
}

compute/Cargo.toml:

[package]
name = "compute"
version = "0.1.0"
edition = "2021"

[dependencies]
core = { path = "../core"}

compute/src/main.rs:

use core::User;

fn main() {
    let u = User::new("tester".into(), 42);
    println!("{:?}", &u);
}

Now let's use it! So the first step is to build the python package:

$ cd core
$ python3 -m venv .env && pip install -U pip && pip install maturin && maturin develop

[...setup and build stuff...]

📦 Built wheel for CPython 3.10 to /var/folders/fp/rcs_97l57rqb28vb6qls5m180000gn/T/.tmpnUdplB/core-0.1.0-cp310-cp310-macosx_11_0_arm64.whl
🛠  Installed core-0.1.0

Let's try it:

$ python
Python 3.10.2 (main, Feb  2 2022, 05:51:25) [Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import core
>>> u = core.User("tester", 42)
>>> u
<builtins.User object at 0x1033c0cf0>
>>> u.name
'tester'
>>> u.age
42

And we can also build a Rust library from it:

$ cargo build
   Compiling pyo3-build-config v0.16.3
   Compiling pyo3-ffi v0.16.3
   Compiling pyo3 v0.16.3
   Compiling core v0.1.0 (/private/tmp/multilib/core)
    Finished dev [unoptimized + debuginfo] target(s) in 3.58s

Now try to run the compute binary:

$ cd ../compute/
$ cargo run -q
    User { name: "tester", age: 42 }

Everything works like a charm 🎉

[dclong]

What happens to functions exported to Python? Taking the following code as example, it seems that there's no way to keep the Rust function multiply untouched as to export it to Python, we have to wrap the result into PyResult. This means that we add a little overhead into Rust code if we want to use the function multiply in Rust.

use pyo3::prelude::*;

#[pyfunction]
fn multiply(a: isize, b: isize) -> PyResult<isize> {
    Ok(a * b)
}

#[pymodule]
fn rust(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(multiply, m)?)?;
    Ok(())
}

[dclong]

And this also means that the Rust library has to depend on the PyO3 crate since we use the PyResult type in Rust code?

[davidhewitt]

There may be some confusion here... this is also allowed (without PyResult):

#[pyfunction]
fn multiply(a: isize, b: isize) -> isize {
    a * b
}

[dclong]

It's awesome to know! I was referring to the blog https://saidvandeklundert.net/learn/2021-11-18-calling-rust-from-python-using-pyo3/ was wondering why do we have to manually wrap Rust output into PyResult. It seems to me that it's something should be done automatically by PyO3.

[dclong]

Any thoughts on Rust code which read data files? What is the best practice for the corresponding Python package to handle the data files? The safest way seems to make copies of data files into the Python package. This means that on the same machine which uses both the Rust library and the Python package, the data files are potentially duplicated.

[dclong]

Or maybe the Python package can install data files into standard locations?

[davidhewitt]

Perhaps you can leave it up to your users to specify where to load the data files? Or you could use include_str! / include_bytes! macros to include them inside the Rust code, if they're not too large.