./src/ci/docker/run.sh dist-x86_64-netbsd fails with: cargo:warning=llvm-wrapper/LLVMWrapper.h:6:10: fatal error: llvm/Config/llvm-config.h: No such file or directory

[psumbera]

I'm not able to run docker build on Ubuntu 24:

$ git clone --recurse-submodules https://github.com/rust-lang/rust.git rust
$ cd rust
$ ./src/ci/docker/run.sh dist-x86_64-netbsd
..
   Compiling rustc_symbol_mangling v0.0.0 (/checkout/compiler/rustc_symbol_mangling)
[RUSTC-TIMING] cc test:false 12.591
[RUSTC-TIMING] sharded_slab test:false 2.321
[RUSTC-TIMING] nu_ansi_term test:false 1.186
[RUSTC-TIMING] wasm_encoder test:false 5.171
   Compiling rustc_llvm v0.0.0 (/checkout/compiler/rustc_llvm)
[RUSTC-TIMING] build_script_build test:false 0.654
[RUSTC-TIMING] thorin test:false 5.010
[RUSTC-TIMING] matchers test:false 0.218
   Compiling rustc_incremental v0.0.0 (/checkout/compiler/rustc_incremental)
warning: rustc_llvm@0.0.0: In file included from llvm-wrapper/PassWrapper.cpp:1:0:
warning: rustc_llvm@0.0.0: llvm-wrapper/LLVMWrapper.h:6:10: fatal error: llvm/Config/llvm-config.h: No such file or directory
warning: rustc_llvm@0.0.0:  #include "llvm/Config/llvm-config.h"  // LLVM_VERSION_MAJOR, LLVM_VERSION_MINOR
warning: rustc_llvm@0.0.0:           ^~~~~~~~~~~~~~~~~~~~~~~~~~~
warning: rustc_llvm@0.0.0: compilation terminated.
error: failed to run custom build command for `rustc_llvm v0.0.0 (/checkout/compiler/rustc_llvm)`

Caused by:
  process didn't exit successfully: `/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/release/build/rustc_llvm-30e73e0cc4e18b0d/build-script-build` (exit status: 1)
  --- stdout
  cargo:rustc-check-cfg=cfg(llvm_component,values("ipo"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("bitreader"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("bitwriter"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("linker"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("asmparser"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("lto"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("coverage"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("instrumentation"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("x86"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("arm"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("aarch64"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("amdgpu"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("avr"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("loongarch"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("m68k"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("csky"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("mips"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("powerpc"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("systemz"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("jsbackend"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("webassembly"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("msp430"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("sparc"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("nvptx"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("hexagon"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("riscv"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("xtensa"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("bpf"))
  cargo:rerun-if-env-changed=RUST_CHECK
  cargo:rerun-if-env-changed=REAL_LIBRARY_PATH_VAR
  cargo:rerun-if-env-changed=REAL_LIBRARY_PATH
  cargo:rerun-if-env-changed=LLVM_CONFIG
  cargo:rerun-if-changed=/checkout/obj/build/x86_64-unknown-linux-gnu/ci-llvm/bin/llvm-config
  cargo:rustc-cfg=llvm_component="aarch64"
  cargo:rustc-cfg=llvm_component="arm"
  cargo:rustc-cfg=llvm_component="asmparser"
  cargo:rustc-cfg=llvm_component="avr"
  cargo:rustc-cfg=llvm_component="bitreader"
  cargo:rustc-cfg=llvm_component="bitwriter"
  cargo:rustc-cfg=llvm_component="bpf"
  cargo:rustc-cfg=llvm_component="coverage"
  cargo:rustc-cfg=llvm_component="csky"
  cargo:rustc-cfg=llvm_component="hexagon"
  cargo:rustc-cfg=llvm_component="instrumentation"
  cargo:rustc-cfg=llvm_component="ipo"
  cargo:rustc-cfg=llvm_component="linker"
  cargo:rustc-cfg=llvm_component="loongarch"
  cargo:rustc-cfg=llvm_component="lto"
  cargo:rustc-cfg=llvm_component="m68k"
  cargo:rustc-cfg=llvm_component="mips"
  cargo:rustc-cfg=llvm_component="msp430"
  cargo:rustc-cfg=llvm_component="nvptx"
  cargo:rustc-cfg=llvm_component="powerpc"
  cargo:rustc-cfg=llvm_component="riscv"
  cargo:rustc-cfg=llvm_component="sparc"
  cargo:rustc-cfg=llvm_component="systemz"
  cargo:rustc-cfg=llvm_component="webassembly"
  cargo:rustc-cfg=llvm_component="x86"
  cargo:rustc-cfg=llvm_component="xtensa"
  cargo:rerun-if-env-changed=LLVM_RUSTLLVM
  cargo:rerun-if-env-changed=LLVM_ASSERTIONS
  cargo:rerun-if-changed=llvm-wrapper/RustWrapper.cpp
  cargo:rerun-if-changed=llvm-wrapper/CoverageMappingWrapper.cpp
  cargo:rerun-if-changed=llvm-wrapper/ArchiveWrapper.cpp
  cargo:rerun-if-changed=llvm-wrapper/PassWrapper.cpp
  cargo:rerun-if-changed=llvm-wrapper/SymbolWrapper.cpp
  cargo:rerun-if-changed=llvm-wrapper/README
  cargo:rerun-if-changed=llvm-wrapper/Linker.cpp
  cargo:rerun-if-changed=llvm-wrapper/.editorconfig
  cargo:rerun-if-changed=llvm-wrapper/LLVMWrapper.h
  cargo:rerun-if-changed=llvm-wrapper/SuppressLLVMWarnings.h
  OUT_DIR = Some(/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-netbsd/release/build/rustc_llvm-d77db8633b6b4a82/out)
  OPT_LEVEL = Some(3)
  TARGET = Some(x86_64-unknown-netbsd)
  HOST = Some(x86_64-unknown-linux-gnu)
  cargo:rerun-if-env-changed=CXX_x86_64-unknown-netbsd
  CXX_x86_64-unknown-netbsd = None
  cargo:rerun-if-env-changed=CXX_x86_64_unknown_netbsd
  CXX_x86_64_unknown_netbsd = Some(sccache x86_64--netbsd-g++-sysroot)
  cargo:rerun-if-env-changed=CC_KNOWN_WRAPPER_CUSTOM
  CC_KNOWN_WRAPPER_CUSTOM = None
  cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some(false)
  CARGO_CFG_TARGET_FEATURE = Some(fxsr,sse,sse2,x87)
  cargo:rerun-if-env-changed=CXXFLAGS_x86_64-unknown-netbsd
  CXXFLAGS_x86_64-unknown-netbsd = None
  cargo:rerun-if-env-changed=CXXFLAGS_x86_64_unknown_netbsd
  CXXFLAGS_x86_64_unknown_netbsd = Some(-ffunction-sections -fdata-sections -fPIC -m64)
  cargo:rerun-if-env-changed=CC_SHELL_ESCAPED_FLAGS
  CC_SHELL_ESCAPED_FLAGS = None
  CARGO_ENCODED_RUSTFLAGS = Some(--cfg=windows_raw_dylib-Csymbol-mangling-version=v0-Zunstable-options--check-cfg=cfg(bootstrap)--check-cfg=cfg(llvm_enzyme)-Zdual-proc-macros-Zmacro-backtrace-Csplit-debuginfo=off-Wrustc::internal-Drustc::symbol_intern_string_literal-Wkeyword_idents_2024-Wunsafe_op_in_unsafe_fn-Clink-args=-Wl,-z,origin-Clink-args=-Wl,-rpath,$ORIGIN/../lib-Alinker-messages-Zon-broken-pipe=kill)
  cargo:warning=In file included from llvm-wrapper/PassWrapper.cpp:1:0:
  cargo:warning=llvm-wrapper/LLVMWrapper.h:6:10: fatal error: llvm/Config/llvm-config.h: No such file or directory
  cargo:warning= #include "llvm/Config/llvm-config.h"  // LLVM_VERSION_MAJOR, LLVM_VERSION_MINOR
  cargo:warning=          ^~~~~~~~~~~~~~~~~~~~~~~~~~~
  cargo:warning=compilation terminated.

  --- stderr


  error occurred in cc-rs: Command LC_ALL="C" "sccache" "x86_64--netbsd-g++-sysroot" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I/checkout/obj/build/x86_64-unknown-netbsd/ci-llvm/include" "-std=c++17" "-fno-exceptions" "-funwind-tables" "-fno-rtti" "-D_GNU_SOURCE" "-D_DEBUG" "-D_GLIBCXX_ASSERTIONS" "-D__STDC_CONSTANT_MACROS" "-D__STDC_FORMAT_MACROS" "-D__STDC_LIMIT_MACROS" "-DLLVM_COMPONENT_AARCH64" "-DLLVM_COMPONENT_ARM" "-DLLVM_COMPONENT_ASMPARSER" "-DLLVM_COMPONENT_AVR" "-DLLVM_COMPONENT_BITREADER" "-DLLVM_COMPONENT_BITWRITER" "-DLLVM_COMPONENT_BPF" "-DLLVM_COMPONENT_COVERAGE" "-DLLVM_COMPONENT_CSKY" "-DLLVM_COMPONENT_HEXAGON" "-DLLVM_COMPONENT_INSTRUMENTATION" "-DLLVM_COMPONENT_IPO" "-DLLVM_COMPONENT_LINKER" "-DLLVM_COMPONENT_LOONGARCH" "-DLLVM_COMPONENT_LTO" "-DLLVM_COMPONENT_M68K" "-DLLVM_COMPONENT_MIPS" "-DLLVM_COMPONENT_MSP430" "-DLLVM_COMPONENT_NVPTX" "-DLLVM_COMPONENT_POWERPC" "-DLLVM_COMPONENT_RISCV" "-DLLVM_COMPONENT_SPARC" "-DLLVM_COMPONENT_SYSTEMZ" "-DLLVM_COMPONENT_WEBASSEMBLY" "-DLLVM_COMPONENT_X86" "-DLLVM_COMPONENT_XTENSA" "-DLLVM_RUSTLLVM" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-netbsd/release/build/rustc_llvm-d77db8633b6b4a82/out/ef10e86dc40538c1-PassWrapper.o" "-c" "llvm-wrapper/PassWrapper.cpp" with args x86_64--netbsd-g++-sysroot did not execute successfully (status code exit status: 1).


warning: build failed, waiting for other jobs to finish...
[RUSTC-TIMING] rustc_symbol_mangling test:false 7.363
[RUSTC-TIMING] rustc_incremental test:false 10.839
[RUSTC-TIMING] rustc_mir_dataflow test:false 14.557
[RUSTC-TIMING] rustc_infer test:false 21.240
[RUSTC-TIMING] rustc_metadata test:false 25.954
Build completed unsuccessfully in 0:00:29
  local time: Wed Jan 29 14:24:36 UTC 2025
  network time: Wed, 29 Jan 2025 14:24:36 GMT

Any suggestion please?

[jieyouxu]

Can you try if this repros for you if you clone without --recurse-submodules, i.e. just plain git clone?

[psumbera]

Can you try if this repros for you if you clone without --recurse-submodules, i.e. just plain git clone?

It fails like this then. I think I'm missing some documentation for this.

..
   Compiling termcolor v1.4.1
   Compiling build_helper v0.1.0 (/checkout/src/build_helper)
    Finished `dev` profile [unoptimized] target(s) in 21.05s
WARN: currently no CI rustc builds have rustc debug assertions enabled. Please either set `rust.debug-assertions` to `false` if you want to use download CI rustc or set `rust.download-rustc` to `false`.
WARN: `rust.debug-assertions = true` will prevent downloading CI rustc as alt CI rustc is not currently built with debug assertions.
WARNING: The `change-id` is missing in the `config.toml`. This means that you will not be able to track the major changes made to the bootstrap configurations.
NOTE: to silence this warning, add `change-id = 135832` at the top of `config.toml`
submodule library/backtrace does not appear to be checked out, but it is required for this step
Consider setting `build.submodules = true` or manually initializing the submodules.
The submodule is required for the standard library and the main Cargo workspace.
Build completed unsuccessfully in 0:00:43
make: *** [Makefile:98: prepare] Error 1
Command failed. Attempt 2/5:
Building bootstrap
    Finished `dev` profile [unoptimized] target(s) in 0.03s
WARN: currently no CI rustc builds have rustc debug assertions enabled. Please either set `rust.debug-assertions` to `false` if you want to use download CI rustc or set `rust.download-rustc` to `false`.
WARN: `rust.debug-assertions = true` will prevent downloading CI rustc as alt CI rustc is not currently built with debug assertions.
WARNING: The `change-id` is missing in the `config.toml`. This means that you will not be able to track the major changes made to the bootstrap configurations.
NOTE: to silence this warning, add `change-id = 135832` at the top of `config.toml`
submodule library/backtrace does not appear to be checked out, but it is required for this step
Consider setting `build.submodules = true` or manually initializing the submodules.
The submodule is required for the standard library and the main Cargo workspace.
Build completed unsuccessfully in 0:00:00
make: *** [Makefile:98: prepare] Error 1
Command failed. Attempt 3/5:
Building bootstrap
    Finished `dev` profile [unoptimized] target(s) in 0.04s
WARN: currently no CI rustc builds have rustc debug assertions enabled. Please either set `rust.debug-assertions` to `false` if you want to use download CI rustc or set `rust.download-rustc` to `false`.
WARN: `rust.debug-assertions = true` will prevent downloading CI rustc as alt CI rustc is not currently built with debug assertions.
WARNING: The `change-id` is missing in the `config.toml`. This means that you will not be able to track the major changes made to the bootstrap configurations.
NOTE: to silence this warning, add `change-id = 135832` at the top of `config.toml`
submodule library/backtrace does not appear to be checked out, but it is required for this step
Consider setting `build.submodules = true` or manually initializing the submodules.
The submodule is required for the standard library and the main Cargo workspace.
Build completed unsuccessfully in 0:00:00
make: *** [Makefile:98: prepare] Error 1
Command failed. Attempt 4/5:
Building bootstrap
    Finished `dev` profile [unoptimized] target(s) in 0.03s
WARN: currently no CI rustc builds have rustc debug assertions enabled. Please either set `rust.debug-assertions` to `false` if you want to use download CI rustc or set `rust.download-rustc` to `false`.
WARN: `rust.debug-assertions = true` will prevent downloading CI rustc as alt CI rustc is not currently built with debug assertions.
WARNING: The `change-id` is missing in the `config.toml`. This means that you will not be able to track the major changes made to the bootstrap configurations.
NOTE: to silence this warning, add `change-id = 135832` at the top of `config.toml`
submodule library/backtrace does not appear to be checked out, but it is required for this step
Consider setting `build.submodules = true` or manually initializing the submodules.
The submodule is required for the standard library and the main Cargo workspace.
Build completed unsuccessfully in 0:00:00
make: *** [Makefile:98: prepare] Error 1
Command failed. Attempt 5/5:
Building bootstrap
    Finished `dev` profile [unoptimized] target(s) in 0.03s
WARN: currently no CI rustc builds have rustc debug assertions enabled. Please either set `rust.debug-assertions` to `false` if you want to use download CI rustc or set `rust.download-rustc` to `false`.
WARN: `rust.debug-assertions = true` will prevent downloading CI rustc as alt CI rustc is not currently built with debug assertions.
WARNING: The `change-id` is missing in the `config.toml`. This means that you will not be able to track the major changes made to the bootstrap configurations.
NOTE: to silence this warning, add `change-id = 135832` at the top of `config.toml`
submodule library/backtrace does not appear to be checked out, but it is required for this step
Consider setting `build.submodules = true` or manually initializing the submodules.
The submodule is required for the standard library and the main Cargo workspace.
Build completed unsuccessfully in 0:00:00
make: *** [Makefile:98: prepare] Error 1
The command has failed after 5 attempts.
  local time: Wed Jan 29 15:46:45 UTC 2025
  network time: Wed, 29 Jan 2025 15:46:45 GMT

[onur-ozkan]

Bootstrap should be able to checkout submodules automatically. We pre-fetch all the submodules on CI, that's why we never seen this error before.

[psumbera]

Bootstrap should be able to checkout submodules automatically. We pre-fetch all the submodules on CI, that's why we never seen this error before.

I have tried without --recurse-submodules but it seems it was needed. As can be seen above.

Any idea what should I do differently?

[onur-ozkan]

Bootstrap should be able to checkout submodules automatically. We pre-fetch all the submodules on CI, that's why we never seen this error before.

I have tried without --recurse-submodules but it seems it was needed. As can be seen above.

Any idea what should I do differently?

I will send out a fix PR soon.

[onur-ozkan]

Bootstrap should be able to checkout submodules automatically. We pre-fetch all the submodules on CI, that's why we never seen this error before.

I have tried without --recurse-submodules but it seems it was needed. As can be seen above.
Any idea what should I do differently?

I will send out a fix PR soon.

It turns out that if we allow updating submodules, we should also disable read-only volume binding, which is not a good idea. I will try to figure out the first problem with git clone --recurse-submodules.

[onur-ozkan]

Using precompiled LLVM seems to cause this problem. You can set NO_DOWNLOAD_CI_LLVM=1 as a workaround until we fix the bootstrapping issue.

[nikic]

The dist-* docker images only work if you set DEPLOY=1.

[onur-ozkan]

#136294 should fix this.

[Kobzol]

Hi, if you want to execute a Linux CI job locally, please use the ci.py script (see https://rustc-dev-guide.rust-lang.org/tests/docker.html at the very bottom).

Try this:

$ python src/ci/github-actions/ci.py run-local dist-x86_64-netbsd