cross compile from linux to win produces a wheel that crashes silently at runtime #2297
Replies: 6 comments 4 replies
-
|
Have you tried reproduce it directly on Windows w/o cross compilation? |
Beta Was this translation helpful? Give feedback.
-
|
I did try NOTE: it's a corp windows machine it is quite constrained with no admin rights. This is one reason, besides convenience, I try to buid all wheels from linux. For example Btw thx for the fast reply. |
Beta Was this translation helpful? Give feedback.
-
|
I keep searching and just found that if I cross compile without
mod black_scholes;
mod implied_vol;
// use pyo3_polars::PolarsAllocator;
use pyo3::types::{PyModule, PyModuleMethods};
use pyo3::{pymodule, Bound, PyResult};
#[pymodule]
fn _rust(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add("__version__", env!("CARGO_PKG_VERSION"))?;
Ok(())
}
// #[global_allocator]
// static ALLOC: PolarsAllocator = PolarsAllocator::new();It seems PolarAllocator is defined there in repo pyo3-polars. What could be so toxic in it that produces a silent crash ? |
Beta Was this translation helpful? Give feedback.
-
|
Ref issue pola-rs/pyo3-polars#117 |
Beta Was this translation helpful? Give feedback.
-
|
Now I'm trying to cross compile from linux to windows with target I do it inside a docker container. Compilation fails with error: This container runs inside a corp network and internet is only accessible through a proxy. How to let cargo to go fetch data through the proxy defined by these env variables ? if possible ? |
Beta Was this translation helpful? Give feedback.
-
|
I just tried run the docker container outside corp: FROM python:3.12-slim-bullseye
# ref https://learn.microsoft.com/en-us/vcpkg/users/platforms/mingw#mingw-cross
RUN apt-get update -y && \
apt-get install -y curl wget vim build-essential
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH=/root/.cargo/bin:$PATH
RUN rustup target add x86_64-pc-windows-msvc
RUN python -m pip install --no-cache-dir -U pip && \
python -m pip install --no-cache-dir cargo-xwin && \
python -m pip install --no-cache-dir maturin
ENV XWIN_CACHE_DIR=/io/cache
WORKDIR /io
COPY ./maturin-build-win-msvc.sh .
ENTRYPOINT [ "/io/maturin-build-win-msvc.sh" ]where #! /bin/bash
maturin build -m /io/Cargo.toml --release --target x86_64-pc-windows-msvc --out /io/dist/docker-msvcBut I get this error: docker run --rm -v "$(pwd)":/io builder-win-msvc:local
Updating crates.io index
Downloading crates ...
Downloaded ahash v0.8.9
Downloaded allocator-api2 v0.2.16
Downloaded array-init-cursor v0.2.0
Downloaded atoi_simd v0.15.6
Downloaded autocfg v1.1.0
Downloaded argminmax v0.6.2
Downloaded base64 v0.22.1
Downloaded aho-corasick v1.1.2
Downloaded cc v1.1.31
Downloaded bitflags v1.3.2
Downloaded bytemuck_derive v1.5.0
Downloaded byteorder v1.5.0
Downloaded regex-automata v0.4.5
Downloaded shlex v1.3.0
Downloaded streaming-decompression v0.1.2
Downloaded equivalent v1.0.1
Downloaded unindent v0.2.3
Downloaded windows-targets v0.48.5
Downloaded wasm-bindgen-shared v0.2.91
Downloaded windows-targets v0.52.6
Downloaded scopeguard v1.2.0
Downloaded home v0.5.9
Downloaded glob v0.3.1
Downloaded fallible-streaming-iterator v0.1.9
Downloaded heck v0.5.0
Downloaded crossbeam-deque v0.8.5
Downloaded cfg-if v1.0.0
Downloaded strength_reduce v0.2.4
Downloaded iter-read v0.3.1
Downloaded indoc v2.0.4
Downloaded dyn-clone v1.0.16
Downloaded castaway v0.2.3
Downloaded rustversion v1.0.14
Downloaded itoa v1.0.10
Downloaded num-integer v0.1.46
Downloaded target-features v0.1.5
Downloaded static_assertions v1.1.0
Downloaded thiserror v1.0.57
Downloaded version_check v0.9.4
Downloaded bumpalo v3.15.1
Downloaded either v1.13.0
Downloaded thiserror-impl v2.0.0
Downloaded wasm-bindgen-macro v0.2.91
Downloaded wasm-bindgen-macro-support v0.2.91
Downloaded option-pricing v0.1.4
Downloaded streaming-iterator v0.1.9
Downloaded stacker v0.1.15
Downloaded polars-error v0.43.1
Downloaded thiserror-impl v1.0.57
Downloaded polars-schema v0.43.1
Downloaded recursive v0.1.1
Downloaded polars-ffi v0.43.1
Downloaded xxhash-rust v0.8.10
Downloaded pyo3-polars-derive v0.11.0
Downloaded recursive-proc-macro-impl v0.1.1
Downloaded memoffset v0.9.0
Downloaded simdutf8 v0.1.4
Downloaded thiserror v2.0.0
Downloaded getrandom v0.2.12
Downloaded lock_api v0.4.11
Downloaded multiversion-macros v0.7.3
Downloaded log v0.4.20
Downloaded wasi v0.11.0+wasi-snapshot-preview1
Downloaded multiversion v0.7.3
Downloaded bitflags v2.4.2
Downloaded memmap2 v0.7.1
Downloaded bytes v1.8.0
Downloaded wasm-bindgen-backend v0.2.91
Downloaded smallvec v1.13.1
Downloaded pyo3-polars v0.17.0
Downloaded polars-row v0.43.1
Downloaded psm v0.1.23
Downloaded zerocopy-derive v0.7.32
Downloaded unicode-ident v1.0.12
Downloaded strum_macros v0.26.4
Downloaded ryu v1.0.17
Downloaded target-lexicon v0.12.16
Downloaded serde_derive v1.0.214
Downloaded serde v1.0.214
Downloaded rand_chacha v0.3.1
Downloaded pyo3-macros v0.22.5
Downloaded bytemuck v1.14.3
Downloaded once_cell v1.19.0
Downloaded crossbeam-utils v0.8.20
Downloaded fast-float v0.2.0
Downloaded crossbeam-epoch v0.9.18
Downloaded percent-encoding v2.3.1
Downloaded planus v0.3.1
Downloaded polars-compute v0.43.1
Downloaded polars-mem-engine v0.43.1
Downloaded polars-utils v0.43.1
Downloaded ppv-lite86 v0.2.17
Downloaded rand_core v0.6.4
Downloaded pyo3-build-config v0.22.5
Downloaded zerocopy v0.7.32
Downloaded wasm-bindgen v0.2.91
Downloaded compact_str v0.8.0
Downloaded redox_syscall v0.4.1
Downloaded ethnum v1.5.0
Downloaded num-traits v0.2.18
Downloaded parking_lot_core v0.9.9
Downloaded parking_lot v0.12.1
Downloaded polars-arrow-format v0.1.0
Downloaded polars-expr v0.43.1
Downloaded quote v1.0.35
Downloaded parquet-format-safe v0.2.4
Downloaded polars-lazy v0.43.1
Downloaded proc-macro2 v1.0.89
Downloaded python3-dll-a v0.2.9
Downloaded rand_distr v0.4.3
Downloaded rayon-core v1.12.1
Downloaded indexmap v2.2.3
Downloaded pyo3-macros-backend v0.22.5
Downloaded js-sys v0.3.68
Downloaded memchr v2.7.1
Downloaded pyo3-ffi v0.22.5
Downloaded rand v0.8.5
Downloaded num-bigint v0.4.4
Downloaded raw-cpuid v11.0.1
Downloaded polars v0.43.1
Downloaded libm v0.2.8
Downloaded itertools v0.13.0
Downloaded hashbrown v0.14.5
Downloaded syn v1.0.109
Downloaded syn v2.0.87
Downloaded polars-io v0.43.1
Downloaded serde-pickle v1.1.1
Downloaded polars-ops v0.43.1
Downloaded portable-atomic v1.9.0
Downloaded rayon v1.9.0
Downloaded polars-parquet v0.43.1
Downloaded regex-syntax v0.8.5
Downloaded chrono v0.4.34
Downloaded regex v1.10.3
Downloaded polars-plan v0.43.1
Downloaded polars-core v0.43.1
Downloaded polars-arrow v0.43.1
Downloaded windows_aarch64_gnullvm v0.52.6
Downloaded windows_aarch64_gnullvm v0.48.5
Downloaded windows_x86_64_gnullvm v0.52.6
Downloaded windows_x86_64_gnullvm v0.48.5
Downloaded pyo3 v0.22.5
Downloaded libc v0.2.153
Downloaded windows_i686_gnullvm v0.52.6
Downloaded windows_aarch64_msvc v0.48.5
Downloaded windows_aarch64_msvc v0.52.6
Downloaded windows_x86_64_msvc v0.52.6
Downloaded windows_x86_64_msvc v0.48.5
Downloaded windows_x86_64_gnu v0.48.5
Downloaded windows_i686_gnu v0.52.6
Downloaded windows_x86_64_gnu v0.52.6
Downloaded windows_i686_msvc v0.48.5
Downloaded windows_i686_gnu v0.48.5
Downloaded windows_i686_msvc v0.52.6
Downloaded winapi v0.3.9
Downloaded windows-sys v0.52.0
Downloaded winapi-i686-pc-windows-gnu v0.4.0
Downloaded winapi-x86_64-pc-windows-gnu v0.4.0
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.8
🐍 Not using a specific python interpreter (Automatically generating windows import library)
Compiling cfg-if v1.0.0
Compiling libm v0.2.8
Compiling serde v1.0.214
Compiling num-traits v0.2.18
Compiling crossbeam-utils v0.8.20
Compiling libc v0.2.153
Compiling rayon-core v1.12.1
Compiling either v1.13.0
Compiling once_cell v1.19.0
Compiling memchr v2.7.1
Compiling winapi v0.3.9
Compiling psm v0.1.23
Compiling ahash v0.8.9
Compiling windows_x86_64_msvc v0.48.5
The following warnings were emitted during compilation:
warning: psm@0.1.23: Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is `clang-cl` installed?
warning: psm@0.1.23: Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is `clang-cl` installed?
error: failed to run custom build command for `psm v0.1.23`
Caused by:
process didn't exit successfully: `/io/target/release/build/psm-627b4451f7640b8d/build-script-build` (exit status: 1)
--- stdout
OPT_LEVEL = Some(3)
TARGET = Some(x86_64-pc-windows-msvc)
OUT_DIR = Some(/io/target/x86_64-pc-windows-msvc/release/build/psm-fabedde2f730ef42/out)
cargo:rerun-if-env-changed=VCINSTALLDIR
VCINSTALLDIR = None
HOST = Some(x86_64-unknown-linux-gnu)
cargo:rerun-if-env-changed=CC_x86_64-pc-windows-msvc
CC_x86_64-pc-windows-msvc = None
cargo:rerun-if-env-changed=CC_x86_64_pc_windows_msvc
CC_x86_64_pc_windows_msvc = Some(clang-cl)
cargo:rerun-if-env-changed=CC_KNOWN_WRAPPER_CUSTOM
CC_KNOWN_WRAPPER_CUSTOM = None
RUSTC_WRAPPER = None
cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
cargo:warning=Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is `clang-cl` installed?
cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
CRATE_CC_NO_DEFAULTS = None
CARGO_CFG_TARGET_FEATURE = Some(cmpxchg16b,fxsr,sse,sse2,sse3)
DEBUG = Some(false)
cargo:rerun-if-env-changed=CFLAGS_x86_64-pc-windows-msvc
CFLAGS_x86_64-pc-windows-msvc = None
cargo:rerun-if-env-changed=CFLAGS_x86_64_pc_windows_msvc
CFLAGS_x86_64_pc_windows_msvc = Some(--target=x86_64-pc-windows-msvc -Wno-unused-command-line-argument -fuse-ld=lld-link /imsvc/root/.cache/cargo-xwin/xwin/crt/include /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/ucrt /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/um /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/shared )
cargo:rerun-if-env-changed=CC_SHELL_ESCAPED_FLAGS
CC_SHELL_ESCAPED_FLAGS = None
cargo:rustc-check-cfg=cfg(switchable_stack,asm,link_asm)
cargo:rustc-cfg=asm
cargo:rustc-cfg=link_asm
cargo:warning=Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is `clang-cl` installed?
--- stderr
error occurred: Failed to find tool. Is `clang-cl` installed?
warning: build failed, waiting for other jobs to finish...
💥 maturin failed
Caused by: Failed to build a native library through cargo
Caused by: Cargo build finished with "exit status: 101": `env -u CARGO AR_x86_64_pc_windows_msvc="llvm-lib" BINDGEN_EXTRA_CLANG_ARGS_x86_64_pc_windows_msvc="-I/root/.cache/cargo-xwin/xwin/crt/include -I/root/.cache/cargo-xwin/xwin/sdk/include/ucrt -I/root/.cache/cargo-xwin/xwin/sdk/include/um -I/root/.cache/cargo-xwin/xwin/sdk/include/shared" CARGO_ENCODED_RUSTFLAGS="-C\u{1f}linker-flavor=lld-link\u{1f}-Lnative=/root/.cache/cargo-xwin/xwin/crt/lib/x86_64\u{1f}-Lnative=/root/.cache/cargo-xwin/xwin/sdk/lib/um/x86_64\u{1f}-Lnative=/root/.cache/cargo-xwin/xwin/sdk/lib/ucrt/x86_64" CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER="lld-link" CC_x86_64_pc_windows_msvc="clang-cl" CFLAGS_x86_64_pc_windows_msvc="--target=x86_64-pc-windows-msvc -Wno-unused-command-line-argument -fuse-ld=lld-link /imsvc/root/.cache/cargo-xwin/xwin/crt/include /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/ucrt /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/um /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/shared " CL_FLAGS="--target=x86_64-pc-windows-msvc -Wno-unused-command-line-argument -fuse-ld=lld-link /imsvc/root/.cache/cargo-xwin/xwin/crt/include /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/ucrt /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/um /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/shared " CMAKE_GENERATOR="Ninja" CMAKE_SYSTEM_NAME="Windows" CMAKE_TOOLCHAIN_FILE_x86_64_pc_windows_msvc="/root/.cache/cargo-xwin/cmake/x86_64-pc-windows-msvc-toolchain.cmake" CXXFLAGS_x86_64_pc_windows_msvc="--target=x86_64-pc-windows-msvc -Wno-unused-command-line-argument -fuse-ld=lld-link /imsvc/root/.cache/cargo-xwin/xwin/crt/include /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/ucrt /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/um /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/shared " CXX_x86_64_pc_windows_msvc="clang-cl" PATH="/root/.cargo/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/.cache/cargo-xwin" RCFLAGS="-I/root/.cache/cargo-xwin/xwin/crt/include -I/root/.cache/cargo-xwin/xwin/sdk/include/ucrt -I/root/.cache/cargo-xwin/xwin/sdk/include/um -I/root/.cache/cargo-xwin/xwin/sdk/include/shared" TARGET_AR="llvm-lib" TARGET_CC="clang-cl" TARGET_CXX="clang-cl" "cargo" "rustc" "--target" "x86_64-pc-windows-msvc" "--message-format" "json-render-diagnostics" "--manifest-path" "/io/Cargo.toml" "--release" "--lib"`
/io/maturin-build-win-msvc.sh: line 4: docker-msvc: command not foundI'm puzzled as I read in the cargo-xwin prerequisite that for a Rust only project (my case) no clang or llvm is necessary. BTW: Unsurprising the corp proxy issue has gone (Cf comment above). So I need find a way to solve these 2 separate issues. |
Beta Was this translation helpful? Give feedback.



Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Bug Description
The cross compile from linux to windows of package polars_plugin_option_pricing produces a wheel.
This wheel can be pip installed on windows.
So far no apparent error or warning.
But at runtime, the program exits silently without error message when reaching the compiled module:
polars_plugin_option_pricing._rust.I tried to build the wheel on a linux host (ubuntu) and from inside a docker container (reproducible below).
I build the docker container on a ubuntu host and a redhat8 host.
Same result :-(
How to debug this ?
Is there a tested template of docker based cross compilation from linux to win somewhere in the docs ?
Note: the native and manylinux compilations work and produce working wheels.
Your maturin version (
maturin --version)1.7.4
Your Python version (
python -V)3.12
Your pip version (
pip -V)24.3.1
What bindings you're using
pyo3
Does
cargo buildwork?If on windows, have you checked that you aren't accidentally using unix path (those with the forward slash
/)?Steps to Reproduce
Commands on linux host:
docker run logs:
Commands on win host, in conda env with py3.12:
pip install polars_plugin_option_pricing-0.1.0-cp38-abi3-win_amd64.whl # from repo root python test-run-bs.pyThis command outputs:
The program should continue (cf demo notebook run-bs.ipynb), but exits without showing any indication what the error was...
Beta Was this translation helpful? Give feedback.
All reactions