How to configure PyO3 to respect my venv ( ModuleNotFoundError ) when calling Python from Rust

[ZizoAdam]

Python 3.11.7
Windows 10 and Ubuntu 22.04 via Docker/WSL2

I have tried using pyenv (both on Windows and Ubuntu) and currently Windows which is ultimately where I'd like things to be running but an Ubuntu dev container was convenient at the time as I could just globally install for the time being until I resolved the issue. I have also tried this with Poetry but had the same issue and I couldn't find any way to sort this via manually setting PYO3_PYTHON or using PYO3_CONFIG_FILE

I repeatedly ran into the issue of getting ModuleNotFoundError when importing third party dependencies in my venv so printing some debug information to the console as so

            let sysconfig = PyModule::import(py, "sysconfig").unwrap();
            let python_version = sysconfig.call_method0("get_python_version").unwrap();
            log::debug!("Using python version: {}", python_version);
            let python_lib = sysconfig
                .call_method("get_config_var", ("LIBDEST",), None)
                .unwrap();
            log::debug!("Using python lib: {}", python_lib);
            let python_site_packages = sysconfig
                .call_method("get_path", ("purelib",), None)
                .unwrap();
            log::debug!("Using python site-packages: {}", python_site_packages);

I get

2024-01-04T12:37:42.187Z DEBUG [demo] Using python version: 3.11
2024-01-04T12:37:42.188Z DEBUG [demo] Using python lib: C:\Users\Adam\AppData\Local\Programs\Python\Python311\Lib
2024-01-04T12:37:42.188Z DEBUG [demo] Using python site-packages: C:\Users\Adam\AppData\Local\Programs\Python\Python311\Lib\site-packages

It is at least partially respecting my venv as it's using Python 3.11 rather than 3.12 when I try to run via poetry run but notably seems to be pointing to the wrong site-packages.

How can I point PyO3 to the right place?

[ZizoAdam]

A bodgy workaround that is doing the job at the moment is manually adding the site packages via

import sys
sys.path.append('my\\venv\\site-packages') # obviously not my real filepath

[isonao]

Thank you. When I was about to give up, I found this post and tried it, and it worked. (Windows 10)

Python::with_gil(|py| {
    let sys = py.import_bound("sys")?;
    let path = sys.getattr("path")?;
    path.call_method1("append", ("myvenv\\Lib\\site-packages",))?;  // append my venv path
    
    let requests = py.import_bound("requests")?;    //not error occurred!!
    ......

[allenwyma]

thank you, this saved me for a demo to show Pythonists how to call Python from Rust :)

[davidhewitt]

This detail may help point you in the right direction: #3717 (comment)

[ZizoAdam]

Activating the virtual environment only seems to point it towards the correct python executable (at least on Windows) and it never picks up the correct path for the third party dependencies. I think the more relevant environment variable would be PYTHONPATH but my current bodge job seems to work for now

[BeastLe9enD]

any news here how to setup the python env properly?

[glevco]

I got everything working correctly by setting both the PYTHONPATH (to where your dependencies are installed) and PYTHONHOME (to your python executable).

I'm on a Mac using with .venv with uv, so on the project directory with the venv activated, that would be:

# maps to /Users/user/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none
$ export PYTHONHOME="$(dirname $(dirname $(realpath $(which python))))"

# maps to .venv/lib/python3.12/site-packages
$ export PYTHONPATH=$(echo .venv/lib/python*/site-packages)

Just adapt it to your setup accordingly.

[BeastLe9enD]

I got everything working correctly by setting both the PYTHONPATH (to where your dependencies are installed) and PYTHONHOME (to your python executable).

I'm on a Mac using with .venv with uv, so on the project directory with the venv activated, that would be:

# maps to /Users/user/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none
$ export PYTHONHOME="$(dirname $(dirname $(realpath $(which python))))"

# maps to .venv/lib/python3.12/site-packages
$ export PYTHONPATH=$(echo .venv/lib/python*/site-packages)

Just adapt it to your setup accordingly.

have you tried setting the environment variables at runtime? Would be interesting if it works for you. Because in my case, I need the rust application to load the venv from a path relativ to the exe, so running a script before is no alternative.

[glevco]

@BeastLe9enD yep, setting both of them with env::set_var before calling Python::with_gil (or any other method that interacts with Python, I suppose), works for me.

[BeastLe9enD]

@glevco can you share your example? And if you did it before pyo3::init_freethreaded_python. Or are u using auto init?