feat: make build script more general

[josibake]

Currently, the build script sets stdc++ as the C++ standard library if using gcc as a compiler, and libc++ if using clang. This is based on the assumption that a user is building the crate on linux if using a gnu style compiler and building on macos if using clang.

However, if using clang on linux, the build will fail due since stdc++ is used but the build script assumes libc++.

steps to reproduce

# use clang on linux
export CC=/usr/bin/clang
export CXX=/usr/bin/clang++

cargo test

Result:

error: linking with `cc` failed: exit status: 1
  |
  = note:  "cc" "-m64" "/tmp/rustc6De2yx/symbols.o" "<6 object files omitted>" "-Wl,--as-needed" "-Wl,-Bstatic" "<sysroot>/lib64/rustlib/x86_64-unknown-linux-gnu/lib/{libtest-*,libgetopts-*,libunicode_width-*,librustc_std_workspace_std-*}.rlib" "/root/rust-bitcoinkernel/target/debug/deps/{liblibbitcoinkernel_sys-fa55f36743b3a8c6.rlib}.rlib" "<sysroot>/lib64/rustlib/x86_64-unknown-linux-gnu/lib/{libstd-*,libpanic_unwind-*,libobject-*,libmemchr-*,libaddr2line-*,libgimli-*,librustc_demangle-*,libstd_detect-*,libhashbrown-*,librustc_std_workspace_alloc-*,libminiz_oxide-*,libadler2-*,libunwind-*,libcfg_if-*,liblibc-*,liballoc-*,librustc_std_workspace_core-*,libcore-*,libcompiler_builtins-*}.rlib" "-Wl,-Bdynamic" "-lc++" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/root/rust-bitcoinkernel/target/debug/build/secp256k1-sys-fe87fd57724954db/out" "-L" "/root/rust-bitcoinkernel/target/debug/build/libbitcoinkernel-sys-f9af07807abfaa57/out/install/lib" "-L" "<sysroot>/lib64/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/root/rust-bitcoinkernel/target/debug/deps/bitcoinkernel-dda3e1a9baf10ae0" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: /usr/bin/ld: cannot find -lc++: No such file or directory
          collect2: error: ld returned 1 exit status


error: could not compile `bitcoinkernel` (lib test) due to 1 previous error
warning: build failed, waiting for other jobs to finish...

Ideas going forward

#6 improves on this by only using libc++ if we using clang on macos. This works for now, but I wanted to open this issue to track alternatives and iterate on the idea a bit more. One idea to explore further is leveraging the cc crate more to infer the tool chain on the system "automagically." Another idea would be explicitly setting sane defaults for the platforms we expect, while allow the user to override these if they are in a non-typical set up.

[stevenroose]

In my nix devshell it also doesn't build at first go. I'm investigating and will report what I had to set to make it work.

You're definitely encouraged to enlighten me:

error: failed to run custom build command for `libbitcoinkernel-sys v0.0.23`
note: To improve backtraces for build dependencies, set the CARGO_PROFILE_TEST_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation.

Caused by:
  process didn't exit successfully: `/var/cargo-target/debug/build/libbitcoinkernel-sys-71ce947d3dc5d00a/build-script-build` (exit status: 101)
  --- stdout
  bitcoin /var/cargo-target/debug/build/libbitcoinkernel-sys-45c29d360e2ba436/out/bitcoin
  cargo:rerun-if-changed=bitcoin
  -- The CXX compiler identification is GNU 14.3.0
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /nix/store/kaj8d1zcn149m40s9h0xi0khakibiphz-gcc-wrapper-14.3.0/bin/g++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Performing Test CXX_SUPPORTS__WERROR
  -- Performing Test CXX_SUPPORTS__WERROR - Success
  -- Performing Test CXX_SUPPORTS__G3
  -- Performing Test CXX_SUPPORTS__G3 - Success
  -- Performing Test LINKER_SUPPORTS__G3
  -- Performing Test LINKER_SUPPORTS__G3 - Success
  -- Performing Test CXX_SUPPORTS__FTRAPV
  -- Performing Test CXX_SUPPORTS__FTRAPV - Success
  -- Performing Test LINKER_SUPPORTS__FTRAPV
  -- Performing Test LINKER_SUPPORTS__FTRAPV - Success
  -- Performing Test LINKER_SUPPORTS__WL___FATAL_WARNINGS
  -- Performing Test LINKER_SUPPORTS__WL___FATAL_WARNINGS - Success
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
  -- Found Threads: TRUE
  -- Configuring incomplete, errors occurred!
  cargo:rustc-link-search=native=/var/cargo-target/debug/build/libbitcoinkernel-sys-45c29d360e2ba436/out/install/lib
  cargo:rustc-link-lib=static=bitcoinkernel

  --- stderr
  CMake Warning (dev) at cmake/module/AddBoostIfNeeded.cmake:32 (find_package):
    Policy CMP0144 is not set: find_package uses upper-case <PACKAGENAME>_ROOT
    variables.  Run "cmake --help-policy CMP0144" for policy details.  Use the
    cmake_policy command to set the policy and suppress this warning.

    Environment variable BOOST_ROOT is set to:

      /nix/store/i7gmw5s133iv0s1gwx7ylydiyr3drzfx-boost-1.87.0-dev

    For compatibility, find_package is ignoring the variable, but code in a
    .cmake module might still use it.
  Call Stack (most recent call first):
    CMakeLists.txt:416 (add_boost_if_needed)
  This warning is for project developers.  Use -Wno-dev to suppress it.

  CMake Error at cmake/module/AddBoostIfNeeded.cmake:32 (find_package):
    Could not find a package configuration file provided by "Boost" (requested
    version 1.73.0) with any of the following names:

      BoostConfig.cmake
      boost-config.cmake

    Add the installation prefix of "Boost" to CMAKE_PREFIX_PATH or set
    "Boost_DIR" to a directory containing one of the above files.  If "Boost"
    provides a separate development package or SDK, be sure it has been
    installed.
  Call Stack (most recent call first):
    CMakeLists.txt:416 (add_boost_if_needed)


  make: Makefile: No such file or directory
  make: *** No rule to make target 'Makefile'.  Stop.
  CMake Error: Not a file: /var/cargo-target/debug/build/libbitcoinkernel-sys-45c29d360e2ba436/out/bitcoin/cmake_install.cmake
  CMake Error: Error processing file: /var/cargo-target/debug/build/libbitcoinkernel-sys-45c29d360e2ba436/out/bitcoin/cmake_install.cmake

  thread 'main' panicked at /home/steven/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/libbitcoinkernel-sys-0.0.23/build.rs:88:10:
  Unable to generate bindings: NotExist("/var/cargo-target/debug/build/libbitcoinkernel-sys-45c29d360e2ba436/out/install/include/bitcoinkernel.h")
  stack backtrace:
     0: __rustc::rust_begin_unwind
               at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/panicking.rs:697:5
     1: core::panicking::panic_fmt
               at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/panicking.rs:75:14
     2: core::result::unwrap_failed
               at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/core/src/result.rs:1765:5
     3: core::result::Result<T,E>::expect
     4: build_script_build::main
     5: core::ops::function::FnOnce::call_once
  note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
warning: build failed, waiting for other jobs to finish...

[stevenroose]

The build works for me on 0.0.21.

[TheCharlatan]

Can you configure Bitcoin Core by itself from the same devshell? If not, I think this should be reported upstream.

[stevenroose]

Ok, I found it (ok tbh claude did), I needed pkgs.boost.dev in my nix devShell's packages, that's all :)

[stevenroose]

Also, you can't have the bitcoinconsensus library in your dependency tree because there is a symbol collision. This might also cause issues when people have different versions of bitcoinkernel in their dependency tree. For rust-secp we have some script that prefixes all symbols with the crate name and version to avoid collisions.

[TheCharlatan]

For rust-secp we have some script that prefixes all symbols with the crate name and version to avoid collisions.

Thanks for the heads up, will try to do the same here.

[stevenroose]

Got it to work :) Can now also replace my bitcoinconsensus usage elsewhere.

Now the fight begins... tx verification failed: ScriptVerify(Invalid) 😅