How do I configure pyo3 to work with Homebrew installed python3?

[l-monninger]

I'm new to pyo3 trying to figure out what the best set up for developing using pyo3 on an M1 mac would be. I'd prefer to not have to develop inside a VM or container.

I'm trying to run the example provided in the documentation:

#[cfg(test)]
use super::*;
use std::env;
use pyo3::prelude::*;
use pyo3::types::IntoPyDict;

fn main() -> PyResult<()> {
    Python::with_gil(|py| {
        let sys = py.import("sys")?;
        let version: String = sys.getattr("version")?.extract()?;

        let locals = [("os", py.import("os")?)].into_py_dict(py);
        let code = "os.getenv('USER') or os.getenv('USERNAME') or 'Unknown'";
        let user: String = py.eval(code, None, Some(&locals))?.extract()?;

        println!("Hello {}, I'm Python {}", user, version);
        Ok(())
    })
}

#[tokio::test(flavor = "multi_thread")]
pub async fn pyo3_installed(){

    main();

}

However, it seems pyo3 is looking for Python3 as a framework which is not how it is installed by Homebrew.

Reason: tried: '/Library/Frameworks/Python3.framework/Versions/3.8/Python3' (no such file), '/System/Library/Frameworks/Python3.framework/Versions/3.8/Python3' (no such file)

What do I need to configure?

[davidhewitt]

Sorry for the long delay in response, I had only limited time recently and was swamped by a huge backlog of issues etc.

Can you run a build with PYO3_PRINT_CONFIG=1 environment variable and post the result here? We can look at the output of that to confirm whether PyO3 is picking up your homebrew python or a system one.

[l-monninger]

error: failed to run custom build command for `pyo3-ffi v0.16.5`

Caused by:
  process didn't exit successfully: `/xx/target/debug/build/pyo3-ffi-8accaf139e500424/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=PYO3_CROSS
  cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR
  cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION
  cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_IMPLEMENTATION
  cargo:rerun-if-env-changed=PYO3_PRINT_CONFIG

  -- PYO3_PRINT_CONFIG=1 is set, printing configuration and halting compile --
  implementation=CPython
  version=3.8
  shared=true
  abi3=false
  lib_name=python3.8
  lib_dir=/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib
  executable=/Applications/Xcode.app/Contents/Developer/usr/bin/python3
  pointer_width=64
  build_flags=
  suppress_build_script_link_lines=false

[aganders3]

Can you try creating and activating a virtual environment with your homebrew Python, then building with the environment activated?

% /usr/local/bin/python3 -m venv venv
% source venv/bin/activate

[l-monninger]

Yeah, that works fine. (That's how I've been using pyo3 since.) But, I ran into the above when trying to run the demo originally, so I'd thought I'd raise it.

Do you want to see some kind of verbose output from the above to compare?

[l-monninger]

It would also be nice just to have a better understanding of how the build configuration works.

  cargo:rerun-if-env-changed=PYO3_CROSS
  cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR
  cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION
  cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_IMPLEMENTATION
  cargo:rerun-if-env-changed=PYO3_PRINT_CONFIG

That is, are those all the relevant environment variables? Why?

[l-monninger]

And, I guess also, why do venvs just work?

[aganders3]

This page goes over how the build config looks for an appropriate Python 3 installation:
https://pyo3.rs/main/building_and_distribution.html#configuring-the-python-version

For whatever reason the version from XCode (I have it too at /Applications/Xcode.app/Contents/Developer/usr/bin/python3) is first on your path without the venv and not suitable for building.

I think if you're not cross compiling and don't want to use a venv it's probably most reliable to use PYO3_PYTHON to set a specific interpreter, print the config, and save it to a config file.

[bazhenov]

You can run pyo3 with homebrew installed python even if it has been linked agains standard one. Just export following variables:

DYLD_FALLBACK_FRAMEWORK_PATH=/usr/local/Frameworks

But, I should note this workaround has some issues. DYLD_*_PATH variables are not effective for binaries protected by macOS System Integrity Protection. This could be a problem if executable is called using intermediate sh/bash process. More robust way is to use PYO3_CONFIG_FILE.

[bazhenov]

I ran into the same issue on github actions runners. Standard python library in located in /Library/Frameworks and homebrew installed python in /usr/local/Frameworks. I printed a config for a homebrew python:

implementation=CPython
version=3.11
shared=true
abi3=false
lib_name=python3.11
lib_dir=/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.11/lib
executable=/usr/local/opt/[email protected]/bin/python3.11
pointer_width=64
build_flags=
suppress_build_script_link_lines=false

Next I add this config under VCS and build a project using following command command:

$ PYO3_CONFIG=ci/macos/python-config cargo build --locked --release

But for whatever reason resulting binary is still linked agains /Library/Framework/Python.framework.

$ otool -L app
app:
        /System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60157.60.19)
        /Library/Frameworks/Python.framework/Versions/3.11/Python (compatibility version 3.11.0, current version 3.11.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1856.105.0)
        /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

How can I force pyo3-build-config to use homebrew installed python or at least investigate this issue further?

[davidhewitt]

Looks like you're using the wrong env var - should be PYO3_CONFIG_FILE ?

[bazhenov]

Yes, indeed. It works. My bad, I somehow didn't notice it. Thank you!