TypeError("'PyExpr' object cannot be converted to 'PyExpr'")

[sun-rs]

There is a #[pyclass] named PyExpr in crate Polars, and I want to get this obj in rust from python code. So I create a python script as below:

#code.py
import polars as pl
from polars.polars import PyExpr

def get_expr() -> PyExpr:
    expr: pl.Expr = pl.col('a').cumsum()
    print(f'I am in python code and expr is: {expr}')
    
    return expr._pyexpr

and then run this script in rust with pyo3 PyModule::from_code:

mod tests {
    
    use pyo3::prelude::*;
    use polars::prelude::*;
    use crate::lazy::dsl::PyExpr;
    use polars::lazy::dsl::Expr;
    
#[test]
fn test_new() {
    let df = DataFrame::new(vec![
        Series::new("a", &[1, 2, 3]),
        Series::new("b", &[4, 5, 6]),
    ]);
    println!("{:?}", df);
    
    let py_code = include_str!("../examples/code.py");
    //println!("{}", py_code);
    
    pyo3::prepare_freethreaded_python();
    
    Python::with_gil(|py|  { 
        let app: Py<PyAny> = PyModule::from_code(py, py_code, "code", "code").unwrap()
            .getattr("get_expr")
            .unwrap()
            .into();  
        let _pyexpr = app.call0(py).unwrap();
        let _pyexpr = _pyexpr.extract::<PyExpr>(py).unwrap();
    })
}

}

I run this newly added test within Polars source code, then I got the error:

thread 'tests::test_new' panicked at 'called `Result::unwrap()` on an `Err` value: PyErr { type: <class 'TypeError'>, value: TypeError("'PyExpr' object cannot be converted to 'PyExpr'"), traceback: None }', src\lib.rs:708:53
stack backtrace:
   0: std::panicking::begin_panic_handler
             at /rustc/935dc07218b4bf6e20231e44eb9263b612fd649b/library\std\src\panicking.rs:575
   1: core::panicking::panic_fmt
             at /rustc/935dc07218b4bf6e20231e44eb9263b612fd649b/library\core\src\panicking.rs:64
   2: core::result::unwrap_failed
             at /rustc/935dc07218b4bf6e20231e44eb9263b612fd649b/library\core\src\result.rs:1790
   3: enum2$<core::result::Result<polars::lazy::dsl::PyExpr,pyo3::err::PyErr> >::unwrap<polars::lazy::dsl::PyExpr,pyo3::err::PyErr>
             at /rustc/935dc07218b4bf6e20231e44eb9263b612fd649b\library\core\src\result.rs:1112
   4: polars::tests::test_new::closure$0
             at .\src\lib.rs:708
   5: pyo3::marker::Python::with_gil<polars::tests::test_new::closure_env$0,tuple$<> >
             at C:\Users\cceva\.cargo\registry\src\mirrors.tuna.tsinghua.edu.cn-df7c3c540f42cdbd\pyo3-0.16.6\src\marker.rs:314
   6: polars::tests::test_new
             at .\src\lib.rs:702
   7: polars::tests::test_new::closure$0
             at .\src\lib.rs:690
   8: core::ops::function::FnOnce::call_once<polars::tests::test_new::closure_env$0,tuple$<> >
             at /rustc/935dc07218b4bf6e20231e44eb9263b612fd649b\library\core\src\ops\function.rs:507
   9: core::ops::function::FnOnce::call_once
             at /rustc/935dc07218b4bf6e20231e44eb9263b612fd649b/library\core\src\ops\function.rs:507
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
test tests::test_new ... FAILED

It looks like I do got an obj named PyExpr, why it is not equal to PyExpr?

[adamreichold]

I suspect that you installed polars and your own code as separate extension? If so, then you most likely hit a fundamental limitation of how PyO3 currently works. The type objects backing #[pyclass] are stored as statics. Hence, each extension built using PyO3 has their own type objects and specifically the type object backing PyExpr in your extension and in the prebuilt polar extension you installed are equal but not identical which explain the strange error message you are getting.

Note that while the statically stored type objects are the current blocker, this whole issue not easy to resolve due to the lack of ABI stability for Rust generally. I.e. even if your extension and the polars extension were able to share type objects, accessing Rust internals of another extension could only work if they were both built with the exact same compiler in the first place.

As to what you can do about it: Usually the best course of action is to treat these objects as ordinary Python objects and ignore their Rust internals and work with them using their Python API via PyAny. If you control both the involved extensions, there is the possibility of using capsule-based API but that is limited to a C API/ABI and similar to FFI usage in Rust with the additional complications of the Python capsules.

[adamreichold]

If I compile the source code to generate a local version of polars and install this extension. Will this works?

Only if you merge them into a single extension providing a common API, so this is quite an intrusive change.

Generally people write an expression like "a+b" and then send this as PyExpr on a dataframe then get result. But my data and expr are not in the same process,so I need to catch this expr and let the dataframe which is held in rust use it as rust strict.

Since you want to send this to another process anyway, you will probably want to serialize the PyExpr anyway. Maybe you can do the serialization using just the Python object and similarly deserialize using the Python API on the receiver side?

[sun-rs]

Thanks. PyExpr can't be serialization by python because some parts would call a error with pickle. Looks like I could only build a single extension.

[nikita240]

Note that while the statically stored type objects are the current blocker, this whole issue not easy to resolve due to the lack of ABI stability for Rust generally. I.e. even if your extension and the polars extension were able to share type objects, accessing Rust internals of another extension could only work if they were both built with the exact same compiler in the first place.

Can you elaborate on why this is? If the type object is stored with repr(C) I don't see why we can't share it across binaries. I think the challenge is in ensuring that the memory layout of the structs in the cdynlib that python imports matches the layout of the local library (since python could have got that library from anywhere, potentially different version). We could force all of the python/rust structs to be repr(C) and hash their fields and args using a macro to allow us to validate for a matching memory layout when python imports the lib.

I'm toying around with this in a local branch, but I wanted to ask if there is more to this than my eye can see @adamreichold

[mejrs]

This means all of their fields have to be #[repr(c)] as well, which means you wouldn't be able to use most of the standard library types like Vec and String and so on.

[adamreichold]

As @mejrs explained, thinking the #[repr(C)] approach through means the whole API that the two extensions want to share between each other needs to use the stable C ABI which means you basically end up at the existing capsule API approach.

And doing this is certainly possible (we do this for the cross-extension dynamic borrow checking in rust-numpy for example), it is just rather tedious and PyO3 provides little support for implementing this beyond the PyCapsule type.