Cross compilation fails #3554

rgreinho · 2023-10-27T02:34:00Z

rgreinho
Oct 27, 2023

Bug Description

We have a project with several crates. They all depend on a common crate named "core". The "core" crate contains structs and functions that we share for both Rust and Python project.

We recently started to create our new AWS Lambda functions in Rust, and we wanted to use structs from the "core" crate.

However, since cargo lambda must cross compile the binaries, the process fails during the compilation.

Steps to Reproduce

Here is a minimal reproducible example:

project tree:

├── Cargo.lock
├── Cargo.toml
├── core
│   ├── Cargo.toml
│   └── src
│       └── lib.rs
└── lambdas
    ├── Cargo.toml
    └── src
        └── lambda-city.rs

Cargo.toml:

[workspace]
members = ["core", "lambdas"]
resolver = "2"

core/Cargo.toml:

[package]
name = "core"
version = "0.1.0"
edition = "2021"

[lib]
name = "core"
crate-type = ["cdylib", "rlib"]

[dependencies]
pyo3 = "0.20.0"
serde = { version = "1.0.190", features = ["derive"] }
serde_json = "1.0.107"

[features]
extension-module = ["pyo3/extension-module"]

core/src/lib.rs:

use pyo3::prelude::*;
use serde::{Deserialize, Serialize};

/// Represents a city.
#[pyclass]
#[derive(Debug, Deserialize, Serialize, Clone)]
pub struct City {
    /// City name.
    #[pyo3(get, set)]
    pub name: String,
    /// Country where the city is located.
    #[pyo3(get, set)]
    pub country: String,
    /// State/region where the city is located.
    #[pyo3(get, set)]
    pub state: String,
}

lambdas/Cargo.toml:

[package]
name = "lambdas"
version = "0.1.0"
edition = "2021"

[dependencies]
aws_lambda_events = { version = "0.11.1", default-features = false, features = [
  "sqs",
] }
core = { path = "../core" }
lambda_runtime = "0.8.1"
tokio = { version = "1", features = ["macros"] }
tracing = { version = "0.1", features = ["log"] }
tracing-subscriber = { version = "0.3", default-features = false, features = [
  "fmt",
] }

[[bin]]
name = "lambda-city"
path = "src/lambda-city.rs"

lambdas/src/lambda-city.rs:

use aws_lambda_events::sqs::SqsEventObj;
use core::City;
use lambda_runtime::{run, service_fn, Error, LambdaEvent};

async fn function_handler(event: LambdaEvent<SqsEventObj<City>>) -> Result<(), Error> {
    let _first_city = &event.payload.records[0].body;

    Ok(())
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    tracing_subscriber::fmt()
        .with_max_level(tracing::Level::INFO)
        .with_target(false)
        .without_time()
        .init();

    run(service_fn(function_handler)).await
}

$ cargo lambda build --release
   Compiling pyo3-build-config v0.20.0
   Compiling serde v1.0.190
   Compiling futures-util v0.3.29
   Compiling tracing v0.1.40
   Compiling darling_macro v0.20.3
   Compiling tokio v1.33.0
   Compiling heck v0.4.1
   Compiling pin-project-internal v1.1.3
   Compiling pyo3-macros-backend v0.20.0
   Compiling darling v0.20.3
   Compiling serde_with_macros v3.4.0
   Compiling memoffset v0.9.0
   Compiling parking_lot v0.12.1
   Compiling pin-project v1.1.3
   Compiling async-stream-impl v0.3.5
   Compiling lazy_static v1.4.0
   Compiling unindent v0.2.3
   Compiling tower-layer v0.3.2
   Compiling indoc v2.0.4
   Compiling pyo3-ffi v0.20.0
   Compiling pyo3 v0.20.0
   Compiling async-stream v0.3.5
   Compiling sharded-slab v0.1.7
   Compiling thread_local v1.1.7
   Compiling base64 v0.21.5
error: failed to run custom build command for `pyo3-ffi v0.20.0`

Caused by:
  process didn't exit successfully: `/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/release/build/pyo3-ffi-d091e3cfbb2b2707/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=PYO3_CROSS
  cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR
  cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION
  cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_IMPLEMENTATION
  cargo:rerun-if-env-changed=PYO3_NO_PYTHON

  --- stderr
  error: PYO3_CROSS_PYTHON_VERSION or an abi3-py3* feature must be specified when cross-compiling and PYO3_CROSS_LIB_DIR is not set.
warning: build failed, waiting for other jobs to finish...

Backtrace

No response

Your operating system and version

macOS Sonoma 14.0 (23A344) - Apple silicon - aarch64

Your Python version (`python --version`)

Python 3.11.6

Your Rust version (`rustc --version`)

rustc 1.73.0 (cc66ad468 2023-10-03)

Your PyO3 version

0.20.0

How did you install python? Did you use a virtualenv?

brew install [email protected]

Additional Info

No response

davidhewitt · 2023-10-27T06:40:04Z

davidhewitt
Oct 27, 2023
Maintainer

Have you tried to set the PYO3_CROSS_PYTHON_VERSION environment variable, as instructed by the error message?

There is more detail on how to cross compile at https://pyo3.rs/v0.20.0/building_and_distribution#cross-compiling

0 replies

rgreinho · 2023-10-27T13:52:33Z

rgreinho
Oct 27, 2023
Author

I did try to set the variables mentioned in the error message, but I was not successful...

$ export PYO3_CROSS_PYTHON_VERSION=3.11
$ cargo lambda build --release

[...]

error: linking with `/Users/rgreinhofer/Library/Caches/cargo-zigbuild/0.17.3/zigcc-x86_64-unknown-linux-gnu.sh` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/rgreinhofer/.nvm/versions/node/v18.16.0/bin:/opt/homebrew/opt/openssl@3/bin:/opt/local/bin:/Users/rgreinhofer/.cargo/bin:/Users/rgreinhofer/.pyenv/shims:/Users/rgreinhofer/.pyenv/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X11/bin:/Library/Apple/usr/bin:/Users/rgreinhofer/go/bin" VSLANG="1033" "/Users/rgreinhofer/Library/Caches/cargo-zigbuild/0.17.3/zigcc-x86_64-unknown-linux-gnu.sh" "-Wl,--version-script=/var/folders/fp/rcs_97l57rqb28vb6qls5m180000gn/T/rustceMXApy/list" "-Wl,--no-undefined-version" "-m64" "/var/folders/fp/rcs_97l57rqb28vb6qls5m180000gn/T/rustceMXApy/symbols.o" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps/core.core.e680254edd87feb-cgu.0.rcgu.o" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps/core.core.e680254edd87feb-cgu.1.rcgu.o" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps/core.core.e680254edd87feb-cgu.2.rcgu.o" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps/core.52kbm1nft4slh0gv.rcgu.o" "-Wl,--as-needed" "-L" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps" "-L" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/release/deps" "-L" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps/libserde-26df56b7b2473ce8.rlib" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps/libpyo3-ee7865e6d17d7936.rlib" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps/libmemoffset-d4f51b749cd3baf7.rlib" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps/libparking_lot-797f78e76482fead.rlib" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps/libparking_lot_core-d203c418574c6ba7.rlib" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps/libcfg_if-92c71c502b5a0dde.rlib" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps/libsmallvec-5a7f6d43c37e8216.rlib" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps/liblock_api-d987550080366c9f.rlib" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps/libscopeguard-6455c5a9aaea911c.rlib" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps/libpyo3_ffi-5f7152356cb2b8f9.rlib" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps/liblibc-7cf16d8d1f7d178f.rlib" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps/libunindent-86e6841b8055a4d0.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-6498d8891e016dca.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-3debdee1a9058d84.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-8339c5bd5cbc92bf.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-160ebcebb54c11ba.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-95c75789f1b65e37.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-7e8094f2d6258832.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-bac9783ef1b45db0.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-a1cd87df2f2d8e76.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-7fd06d468d7dba16.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-5ac19487656e05bf.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-c7c35d32cf825c11.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-c523f1571362e70b.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-85f17c92b770a911.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-598d3ba148dadcea.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-a58ec2dab545caa4.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-f9dda8cca149f0fc.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-7ba4c315dd7a3503.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-5ac2993e19124966.rlib" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-df2fb7f50dec519a.rlib" "-Wl,-Bdynamic" "-lpython3.11" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps/libcore.so" "-Wl,--gc-sections" "-shared" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-Wl,--strip-all" "-nodefaultlibs"
  = note: error: unable to find Dynamic system library 'python3.11' using strategy 'no_fallback'. searched paths:
            /Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/x86_64-unknown-linux-gnu/release/deps/libpython3.11.so
            /Users/rgreinhofer/projects/PeopleForBikes/labs/pyo3ccissue/target/release/deps/libpython3.11.so
            /Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpython3.11.so
            /Users/rgreinhofer/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpython3.11.so


error: could not compile `core` (lib) due to previous error

0 replies

rgreinho · 2023-10-27T23:13:43Z

rgreinho
Oct 27, 2023
Author

I found a solution to my problem!

I took the libpython package from Debian, extracted its content, and symlinked the libpython3.11.so to the appropriate version of my rust toolchain.

It is easy to implement, but it was a bit tricky to find. So I feel like this could be documented or even automated. If you give me some pointers where it would be best to put this information, I could definitely take a stab at it.

Here are the steps.

cd /tmp
mdir libpython
cd libpython
curl -lO http://http.us.debian.org/debian/pool/main/p/python3.11/libpython3.11_3.11.6-3_amd64.deb
ar x libpython3.11_3.11.6-3_amd64.deb
tar xvJf data.tar.xz
ln usr/lib/x86_64-linux-gnu/libpython3.11.so.1.0 ~/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpython3.11.so

Make sure to export the Python version:

export PYO3_CROSS_PYTHON_VERSION=3.11

Then compile again:

cargo lambda build --release

And it works!

$ file target/lambda/lambda-city/bootstrap
target/lambda/lambda-city/bootstrap: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.0.0, stripped

1 reply

adamreichold Oct 28, 2023
Maintainer

I took the libpython package from Debian, extracted its content, and symlinked the libpython3.11.so to the appropriate version of my rust toolchain.

It is easy to implement, but it was a bit tricky to find. So I feel like this could be documented or even automated. If you give me some pointers where it would be best to put this information, I could definitely take a stab at it.

How do you know that the Python library you extracted from Debian matches the AWS Lambda target environment?

The cargo lambda subcommand seems to rely on cargo zigbuild behind the scenes to create a rather hermetic build that does not depend on shared libraries present on your system (because they won't be on the Lambda machine). So I suspect you need to investigate how to change the environment used cargo lambda and specifically how to make it use one that includes a Python interpreter.

rgreinho · 2023-10-28T14:58:02Z

rgreinho
Oct 28, 2023
Author

cargo lambda build compiles for the x86_64 architecture by default, which is why I chose this Python version. It can be controlled with the --arm64 or --x86-64 cli flags.

You may be right, there may be something to configure for zigbuild.

Also to be noted that with the trick I used, I cannot run the tests anymore, as they do not compile anymore:

$ cargo t

[...]

error: linking with `cc` failed: exit status: 1
[...]
  = note: ld: library 'python3.11' not found
          clang: error: linker command failed with exit code 1 (use -v to see invocation)
 
error: could not compile `core` (lib test) due to previous error
warning: build failed, waiting for other jobs to finish...
error: could not compile `core` (lib) due to previous error

So, my workaround is just to unblock me and allow me to deploy for now. 😭 Thus I removed the " Mark as answer" check mark.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Cross compilation fails #3554

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 4 comments 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Cross compilation fails #3554

Uh oh!

Uh oh!

rgreinho Oct 27, 2023

Bug Description

Steps to Reproduce

Backtrace

Your operating system and version

Your Python version (python --version)

Your Rust version (rustc --version)

Your PyO3 version

How did you install python? Did you use a virtualenv?

Additional Info

Replies: 4 comments · 1 reply

Uh oh!

davidhewitt Oct 27, 2023 Maintainer

Uh oh!

rgreinho Oct 27, 2023 Author

Uh oh!

rgreinho Oct 27, 2023 Author

Uh oh!

adamreichold Oct 28, 2023 Maintainer

Uh oh!

rgreinho Oct 28, 2023 Author

rgreinho
Oct 27, 2023

Your Python version (`python --version`)

Your Rust version (`rustc --version`)

Replies: 4 comments 1 reply

davidhewitt
Oct 27, 2023
Maintainer

rgreinho
Oct 27, 2023
Author

rgreinho
Oct 27, 2023
Author

adamreichold Oct 28, 2023
Maintainer

rgreinho
Oct 28, 2023
Author