Importing module from conda or venv when calling python from rust

[Nohac]

Hi.

Is it possible to use modules/packages installed into a venv or conda environment when calling python from rust?

Example:

#[pyfunction]
fn rust_function(n: i64) -> i64 {
    n + 1
}

#[pymodule]
fn foo(_py: Python<'_>, foo_module: &PyModule) -> PyResult<()> {
    foo_module.add_function(wrap_pyfunction!(rust_function, foo_module)?)?;
    Ok(())
}

fn main() {
    pyo3::append_to_inittab!(foo);
    let res = Python::with_gil(|py| -> PyResult<Py<PyAny>> {
        let syspath: &PyList = py.import("sys")?.getattr("path")?.downcast()?;
        syspath.append("/path/to/my/module")?;
        let app: Py<PyAny> = PyModule::import(py, "my_module")?
            .getattr("python_callback")?
            .into();

        app.call0(py)
    });

    println!("res: {res:?}");
}

# /path/to/my/module/my_module.py

from foo import rust_function
from conda.module import some_function

def python_callback():
    rust_num = rust_function(42);
    some_function(rust_num);
    print("Printing rsult from rust_function: ", rust_num)
    return rust_num;

This works fine if I don't include the "conda.module", otherwise I'm getting the following error message:

res: Err(PyErr { type: <class 'ModuleNotFoundError'>, value: ModuleNotFoundError("No module named 'conda'"), traceback: Some(<traceback object at 0x7f539f34bfc0>) })

I'm running the executable inside of an already activated conda environment, importing the module works fine if I call python directly.

[davidhewitt]

You might want to check PyO3's issue tracker. Venvs are known to work well in Linux but have pain points on Windows and Mac.

I think we will need to make improvements here to support Python 3.12, but I haven't got a clear proposal in my head yet.

[Nohac]

I see. I looked at the issues, but didn't find anything directly related to imports, only interpreter issues, which I don't have. I'm also running this on a linux machine, I can test out venv and see if that works better.
I thought venv and conda worked the same way regarding setting up an environment that python would understand. Python having specific logic to work with conda doesn't sound right, but I guess it wouldn't be a surprise if it did...

[Nohac]

I finally managed to run and import modules inside a conda environment.
The two main things I had to do was:

• compile the bin/lib inside of the conda environment.
• Specify: LD_LIBRARY_PATH=$CONDA_PREFIX/lib/ command-to-run

An additional step I had to do when compiling as a lib and using it with dlopen is to specify the RTLD_GLOBAL flag to make sure that any python module that also uses dlopen gets access to LD_LIBRARY_PATH set by the parent process.

This is obviously not very usable, and definitively not portable. I also don't see a way to make this easier without PyO3 itself using dlopen or something similar to find and "link" the correct (conda) python library at runtime.