compile for musl target issues

[stutxo]

heyo, im trying to use bitcoinkernel in an aws nitro enclave, but having some trouble compiling the library to x86_64-unknown-linux-musl or aarch64-unknown-linux-musl, which are the two only supported targets for nitro enclaves.

any suggestions for what i should try? i did get it working once with a modified build.rs but i dont think it worked, had to use the following, which i do not think is supported in nitro encalves.

export RUSTFLAGS="-C link-arg=-no-pie"

thanks!

[TheCharlatan]

This is definitely something I'd like to support. I probably won't be able to get to it the next few days though.

[stutxo]

i managed to get something working
with minimal changes to build.rs using https://github.com/rust-cross/cargo-zigbuild

shall i open a PR with the changes so you can review and see if that is a good way or not to do this?

thanks!

[TheCharlatan]

shall i open a PR with the changes so you can review and see if that is a good way or not to do this?

Sure! But I think we should also have a CI job here for every target we support.

[stutxo]

Here is my dockerfile that lets me compile this library to aarch64-unknown-linux-musl

The only change i had to make to build.rs to get this to work was adding .arg("-DCMAKE_CXX_FLAGS_RELWITHDEBINFO=-UNDEBUG")

I posted the error logs below for when i dont have this enabled. It seems like i shouldnt add this but im not sure how else to get around this error. Let me know if you have any thoughts. thanks

FROM ubuntu:24.04 AS builder

ARG RUST=1.89.0
ARG ZIG_VER=0.14.1
ARG TARGET=aarch64-unknown-linux-musl

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
    curl xz-utils cmake libboost1.83-dev libclang-17-dev \
 && rm -rf /var/lib/apt/lists/*

RUN curl -fSL "https://ziglang.org/download/${ZIG_VER}/zig-aarch64-linux-${ZIG_VER}.tar.xz" -o /tmp/zig.tar.xz \
 && tar -C /opt -xJf /tmp/zig.tar.xz \
 && mv /opt/zig-aarch64-linux-* /opt/zig \
 && ln -sf /opt/zig/zig /usr/local/bin/zig \
 && rm /tmp/zig.tar.xz

RUN curl -sSf https://sh.rustup.rs | RUSTUP_INIT_SKIP_PATH_CHECK=yes sh -s -- -y --profile minimal --default-toolchain "${RUST}" \
 && /root/.cargo/bin/rustup target add "${TARGET}" \
 && /root/.cargo/bin/cargo install --locked cargo-zigbuild
ENV PATH="/root/.cargo/bin:${PATH}"

ENV ZIG_TARGET=aarch64-linux-musl
ENV CC="zig cc -target ${ZIG_TARGET}"
ENV CXX="zig c++ -target ${ZIG_TARGET}"
ENV CMAKE_C_COMPILER="zig"
ENV CMAKE_C_COMPILER_ARG1="cc"
ENV CMAKE_CXX_COMPILER="zig"
ENV CMAKE_CXX_COMPILER_ARG1="c++"

WORKDIR /app
COPY . ./

RUN set -eux; \
  BOOST_DIR="$(dirname "$(find /usr -name BoostConfig.cmake -print -quit)")"; \
  echo "Detected Boost_DIR=$BOOST_DIR"; \
  test -n "$BOOST_DIR"; \
  CMAKE_PREFIX_PATH=/usr \  
  Boost_DIR="$BOOST_DIR" \
  cargo zigbuild --release --target ${TARGET} -p examples --bin silentpaymentscanner

FROM alpine:3.22.1
COPY --from=builder /app/target/*/release/silentpaymentscanner /silentpaymentscanner
CMD ["/silentpaymentscanner"]

Without DCMAKE_CXX_FLAGS_RELWITHDEBINFO=-UNDEBUG i get the following error when building to musl target

7.68   [ 41%] Built target leveldb
67.68   [ 43%] Linking C static library ../lib/libsecp256k1.a
67.68   [ 43%] Built target secp256k1
67.68   -- Installing: /app/target/aarch64-unknown-linux-musl/release/build/libbitcoinkernel-sys-ff17b63e14a3c2ca/out/install/lib/pkgconfig/libbitcoinkernel.pc
67.68   cargo:rustc-link-search=native=/app/target/aarch64-unknown-linux-musl/release/build/libbitcoinkernel-sys-ff17b63e14a3c2ca/out/install/lib
67.68   cargo:rustc-link-lib=static=bitcoinkernel
67.68
67.68   --- stderr
67.68
67.68   Configuring secp256k1 subtree...
67.68
67.68
67.68   secp256k1 configure summary
67.68   ===========================
67.68   Build artifacts:
67.68     library type ........................ Static
67.68   Optional modules:
67.68     ECDH ................................ OFF
67.68     ECDSA pubkey recovery ............... ON
67.68     extrakeys ........................... ON
67.68     schnorrsig .......................... ON
67.68     musig ............................... ON
67.68     ElligatorSwift ...................... ON
67.68   Parameters:
67.68     ecmult window size .................. 15
67.68     ecmult gen table size ............... 86 KiB
67.68   Optional features:
67.68     assembly ............................ OFF
67.68     external callbacks .................. OFF
67.68   Optional binaries:
67.68     benchmark ........................... OFF
67.68     noverify_tests ...................... OFF
67.68     tests ............................... OFF
67.68     exhaustive tests .................... OFF
67.68     ctime_tests ......................... OFF
67.68     examples ............................ OFF
67.68
67.68   Cross compiling ....................... FALSE
67.68   API visibility attributes ............. ON
67.68   Valgrind .............................. OFF
67.68   Preprocessor defined macros ........... ECMULT_WINDOW_SIZE=15 COMB_BLOCKS=43 COMB_TEETH=6
67.68   C compiler ............................ Clang 19.1.7, /usr/local/bin/zig
67.68   CFLAGS ................................
67.68   Compile options ....................... -Wall -pedantic -Wcast-align -Wconditional-uninitialized -Wextra -Wnested-externs -Wno-long-long -Wno-overlength-strings -Wno-unused-function -Wreserved-identifier -Wshadow -Wstrict-prototypes -Wundef
67.68   Build type:
67.68    - CMAKE_BUILD_TYPE ................... RelWithDebInfo
67.68    - CFLAGS ............................. -O2 -g
67.68    - LDFLAGS for executables ............
67.68    - LDFLAGS for shared libraries .......
67.68
67.68
67.68
67.68   Configure summary
67.68   =================
67.68   Executables:
67.68     bitcoin ............................. OFF
67.68     bitcoind ............................ OFF
67.68     bitcoin-node (multiprocess) ......... OFF
67.68     bitcoin-qt (GUI) .................... OFF
67.68     bitcoin-gui (GUI, multiprocess) ..... OFF
67.68     bitcoin-cli ......................... OFF
67.68     bitcoin-tx .......................... OFF
67.68     bitcoin-util ........................ OFF
67.68     bitcoin-wallet ...................... OFF
67.68     bitcoin-chainstate (experimental) ... OFF
67.68     libbitcoinkernel (experimental) ..... ON
67.68     kernel-test (experimental) .......... ON
67.68   Optional features:
67.68     wallet support ...................... OFF
67.68     external signer ..................... OFF
67.68     ZeroMQ .............................. OFF
67.68     IPC ................................. OFF
67.68     USDT tracing ........................ OFF
67.68     QR code (GUI) ....................... OFF
67.68     DBus (GUI) .......................... OFF
67.68   Tests:
67.68     test_bitcoin ........................ OFF
67.68     test_bitcoin-qt ..................... OFF
67.68     bench_bitcoin ....................... OFF
67.68     fuzz binary ......................... OFF
67.68
67.68   Cross compiling ....................... FALSE
67.68   C++ compiler .......................... Clang 19.1.7, /usr/local/bin/zig
67.68   CMAKE_BUILD_TYPE ...................... RelWithDebInfo
67.68   Preprocessor defined macros ...........
67.68   C++ compiler flags .................... c++ -target aarch64-linux-musl -O2 -g -std=c++20 -fPIC -fdebug-prefix-map=/app/libbitcoinkernel-sys/bitcoin/src=. -fmacro-prefix-map=/app/libbitcoinkernel-sys/bitcoin/src=. -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -Wstack-protector -fstack-protector-all -fstack-clash-protection -mbranch-protection=standard -Wall -Wextra -Wgnu -Wformat -Wformat-security -Wvla -Wshadow-field -Wthread-safety -Wloop-analysis -Wredundant-decls -Wunused-member-function -Wdate-time -Wconditional-uninitialized -Woverloaded-virtual -Wsuggest-override -Wimplicit-fallthrough -Wunreachable-code -Wdocumentation -Wself-assign -Wundef -Wno-unused-parameter
67.68   Linker flags .......................... -O2 -g -fstack-protector-all -fstack-clash-protection -Wl,--dynamicbase -Wl,--nxcompat -Wl,--high-entropy-va -Wl,-z,relro -Wl,-z,now -fPIE -pie
67.68
67.68   NOTE: The summary above may not exactly match the final applied build flags
67.68         if any additional CMAKE_* or environment variables have been modified.
67.68         To see the exact flags applied, build with the --verbose option.
67.68
67.68   Treat compiler warnings as errors ..... OFF
67.68   Use ccache for compiling .............. OFF
67.68
67.68
67.68     ******
67.68
67.68   CMake Warning at CMakeLists.txt:711 (message):
67.68     Minimum required Python not found.
67.68
67.68
67.68     ******
67.68
67.68   In file included from /app/libbitcoinkernel-sys/bitcoin/src/crypto/muhash.cpp:10:
67.68   /app/libbitcoinkernel-sys/bitcoin/src/util/check.h:68:2: error: "Cannot compile without assertions!"
67.68      68 | #error "Cannot compile without assertions!"
67.68         |  ^
67.68   1 error generated.
67.68   gmake[2]: *** [src/crypto/CMakeFiles/bitcoin_crypto.dir/build.make:174: src/crypto/CMakeFiles/bitcoin_crypto.dir/muhash.cpp.o] Error 1
67.68   gmake[2]: *** Waiting for unfinished jobs....
67.68   gmake[1]: *** [CMakeFiles/Makefile2:648: src/crypto/CMakeFiles/bitcoin_crypto.dir/all] Error 2
67.68   gmake[1]: *** Waiting for unfinished jobs....
67.68   /app/libbitcoinkernel-sys/bitcoin/src/leveldb/util/cache.cc:129:14: warning: variable 'count' set but not used [-Wunused-but-set-variable]
67.68     129 |     uint32_t count = 0;
67.68         |              ^
67.68   1 warning generated.
67.68   gmake: *** [Makefile:136: all] Error 2
67.68   CMake Error at /app/target/aarch64-unknown-linux-musl/release/build/libbitcoinkernel-sys-ff17b63e14a3c2ca/out/bitcoin/src/kernel/cmake_install.cmake:50 (file):
67.68     file INSTALL cannot find
67.68     "/app/target/aarch64-unknown-linux-musl/release/build/libbitcoinkernel-sys-ff17b63e14a3c2ca/out/bitcoin/lib/libbitcoinkernel.a":
67.68     No such file or directory.
67.68   Call Stack (most recent call first):
67.68     /app/target/aarch64-unknown-linux-musl/release/build/libbitcoinkernel-sys-ff17b63e14a3c2ca/out/bitcoin/src/cmake_install.cmake:67 (include)
67.68     /app/target/aarch64-unknown-linux-musl/release/build/libbitcoinkernel-sys-ff17b63e14a3c2ca/out/bitcoin/cmake_install.cmake:57 (include)
67.68
67.68
67.68
67.68   thread 'main' panicked at libbitcoinkernel-sys/build.rs:86:10:
67.68   Unable to generate bindings: NotExist("/app/target/aarch64-unknown-linux-musl/release/build/libbitcoinkernel-sys-ff17b63e14a3c2ca/out/install/include/bitcoinkernel.h")
67.68   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
------

 1 warning found (use docker --debug to expand):
 - UndefinedVar: Usage of undefined variable '$TARGET' (line 42)
Dockerfile:33
--------------------
  32 |
  33 | >>> RUN set -eux; \
  34 | >>>   BOOST_DIR="$(dirname "$(find /usr -name BoostConfig.cmake -print -quit)")"; \
  35 | >>>   echo "Detected Boost_DIR=$BOOST_DIR"; \
  36 | >>>   test -n "$BOOST_DIR"; \
  37 | >>>   CMAKE_PREFIX_PATH=/usr \
  38 | >>>   Boost_DIR="$BOOST_DIR" \
  39 | >>>   cargo zigbuild --release --target ${TARGET} -p examples --bin silentpaymentscanner
  40 |
--------------------
ERROR: failed to solve: process "/bin/sh -c set -eux;   BOOST_DIR=\"$(dirname \"$(find /usr -name BoostConfig.cmake -print -quit)\")\";   echo \"Detected Boost_DIR=$BOOST_DIR\";   test -n \"$BOOST_DIR\";   CMAKE_PREFIX_PATH=/usr   Boost_DIR=\"$BOOST_DIR\"   cargo zigbuild --release --target ${TARGET} -p examples --bin silentpaymentscanner" did not complete successfully: exit code: 101

[stutxo]

Think this is because cargo-zigbuild is injecting DNDEBUG when it runs, so thats why i have to set -UNDEBUG Will see if i can stop that from happening

[stutxo]

added ENV CXXFLAGS="-UNDEBUG" to the dockerfile instead and its working now. here is my working dockerfile for aarch64-unknown-linux-musl target

Would it be useful for me to add this to the CI? or just have it here if someone else needs it?

FROM ubuntu:24.04 AS builder

ARG RUST=1.89.0
ARG ZIG_VER=0.14.1
ARG TARGET=aarch64-unknown-linux-musl

RUN apt-get update && apt-get install -y \
    curl xz-utils cmake libboost1.83-dev libclang-17-dev \
 && rm -rf /var/lib/apt/lists/*

RUN curl -fSL "https://ziglang.org/download/${ZIG_VER}/zig-aarch64-linux-${ZIG_VER}.tar.xz" -o /tmp/zig.tar.xz \
 && tar -C /opt -xJf /tmp/zig.tar.xz \
 && mv /opt/zig-aarch64-linux-* /opt/zig \
 && ln -sf /opt/zig/zig /usr/local/bin/zig \
 && rm /tmp/zig.tar.xz

RUN curl -sSf https://sh.rustup.rs | RUSTUP_INIT_SKIP_PATH_CHECK=yes sh -s -- -y --profile minimal --default-toolchain "${RUST}" \
 && /root/.cargo/bin/rustup target add "${TARGET}" \
 && /root/.cargo/bin/cargo install --locked cargo-zigbuild
ENV PATH="/root/.cargo/bin:${PATH}"

ENV ZIG_TARGET=aarch64-linux-musl
ENV CC="zig cc -target ${ZIG_TARGET}"
ENV CXX="zig c++ -target ${ZIG_TARGET}"
ENV CXXFLAGS="-UNDEBUG"

WORKDIR /app
COPY . ./

ENV Boost_DIR=/usr/lib/aarch64-linux-gnu/cmake/Boost-1.83.0

RUN set -eux; \
  cargo zigbuild --release --target aarch64-unknown-linux-musl \
    -p examples --bin silentpaymentscanner

FROM alpine:3.22.1
COPY --from=builder /app/target/*/release/silentpaymentscanner /silentpaymentscanner
CMD ["/silentpaymentscanner"]

[joshdoman]

@stutxo I was able to compile to aarch64-unknown-linux-musl using Nix. Here's the relevant portion of the flake file:

pkgsMusl = import nixpkgs {
  inherit system overlays;
  crossSystem = {
    config = "aarch64-unknown-linux-musl";
  };
};

myApp = pkgsMusl.rustPlatform.buildRustPackage {
  pname = "myApp";
  version = "0.1.0";
  src = ./.;
  cargoLock.lockFile = ./Cargo.lock;
  doCheck = false;
  
  # Build dependencies for libbitcoinkernel
  nativeBuildInputs = with pkgsMusl.pkgsBuildHost; [
    cmake
    pkg-config
    git
    clang
    llvm
  ];
  
  buildInputs = with pkgsMusl; [
    boost
    libevent
    openssl
    sqlite
  ];
  
  # Environment variables
  RUSTFLAGS = "-C target-feature=+crt-static";
  RUST_LOG = "info";
  RUST_BACKTRACE = "1";
  
  # Bindgen environment variables
  LIBCLANG_PATH = "${pkgsMusl.pkgsBuildHost.libclang.lib}/lib";
  BINDGEN_EXTRA_CLANG_ARGS = pkgs.lib.concatStringsSep " " [
    "-I${pkgsMusl.pkgsBuildHost.clang}/resource-root/include"
    "-I${pkgsMusl.glibc.dev}/include"
  ];
};

[TheCharlatan]

Thanks for all the work posted here.

It seems like i shouldnt add this but im not sure how else to get around this error. Let me know if you have any thoughts

For the time being, defning UNDEBUG seems acceptable to me too. The environment should not be allowed to override this option. Maybe it would be better to be explicit about this though by warning in the build script if the environment value is set.

Would it be useful for me to add this to the CI? or just have it here if someone else needs it?

Yes, definitely useful. Will also try to get around to cleaning up the CI a bit by allowing the user to run CI jobs like this one locally through nix.