Crash while trying to BOLT Clang/LLD 198c3eecee50d90cdff4b7840cfa39eef5613870

[ms178]

This crash was seen while trying to BOLT Clang/LLD 198c3ee.

-- Check for working C compiler: /home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/clang-19 - broken
CMake Error at /usr/share/cmake/Modules/CMakeTestCCompiler.cmake:67 (message):
  The C compiler

    "/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/clang-19"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: '/home/marcus/toolchain/llvm/stage3-without-sampling/CMakeFiles/CMakeScratch/TryCompile-EXhick'
    
    Run Build Command(s): /usr/bin/ninja -v cmTC_ba85d
    [1/2] /home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/clang-19   -O3 -march=native -mtune=native -maes -mbmi2 -mpclmul -mllvm -inline-threshold=500 -mllvm -extra-vectorizer-passes -mllvm -enable-interleaved-mem-accesses -mllvm -enable-masked-interleaved-mem-accesses -mllvm -enable-cond-stores-vec -mllvm -slp-vectorize-hor-store -mllvm -enable-loopinterchange -mllvm -enable-loop-distribute -mllvm -enable-unroll-and-jam -mllvm -enable-loop-flatten -mllvm -unroll-runtime-multi-exit -mllvm -aggressive-ext-opt -fno-math-errno -fno-trapping-math -falign-functions=32 -fno-semantic-interposition -fno-omit-frame-pointer -fcf-protection=none -mharden-sls=none -flto=thin -fwhole-program-vtables -mllvm -adce-remove-loops -mllvm -enable-ext-tsp-block-placement=1 -mllvm -enable-gvn-hoist -mllvm -enable-dfa-jump-thread -MD -MT CMakeFiles/cmTC_ba85d.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_ba85d.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_ba85d.dir/testCCompiler.c.o -c /home/marcus/toolchain/llvm/stage3-without-sampling/CMakeFiles/CMakeScratch/TryCompile-EXhick/testCCompiler.c
    [2/2] : && /home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/clang-19 -O3 -march=native -mtune=native -maes -mbmi2 -mpclmul -mllvm -inline-threshold=500 -mllvm -extra-vectorizer-passes -mllvm -enable-interleaved-mem-accesses -mllvm -enable-masked-interleaved-mem-accesses -mllvm -enable-cond-stores-vec -mllvm -slp-vectorize-hor-store -mllvm -enable-loopinterchange -mllvm -enable-loop-distribute -mllvm -enable-unroll-and-jam -mllvm -enable-loop-flatten -mllvm -unroll-runtime-multi-exit -mllvm -aggressive-ext-opt -fno-math-errno -fno-trapping-math -falign-functions=32 -fno-semantic-interposition -fno-omit-frame-pointer -fcf-protection=none -mharden-sls=none -flto=thin -fwhole-program-vtables -mllvm -adce-remove-loops -mllvm -enable-ext-tsp-block-placement=1 -mllvm -enable-gvn-hoist -mllvm -enable-dfa-jump-thread -Wl,--lto-CGO3 -Wl,--gc-sections -Wl,--icf=all -Wl,--lto-O3,-O3,-Bsymbolic-functions,--as-needed -march=native -mtune=native -maes -mbmi2 -mpclmul -fuse-ld=lld -fcf-protection=none -mharden-sls=none -flto=thin -fwhole-program-vtables CMakeFiles/cmTC_ba85d.dir/testCCompiler.c.o -o cmTC_ba85d   && :
    FAILED: cmTC_ba85d 
    : && /home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/clang-19 -O3 -march=native -mtune=native -maes -mbmi2 -mpclmul -mllvm -inline-threshold=500 -mllvm -extra-vectorizer-passes -mllvm -enable-interleaved-mem-accesses -mllvm -enable-masked-interleaved-mem-accesses -mllvm -enable-cond-stores-vec -mllvm -slp-vectorize-hor-store -mllvm -enable-loopinterchange -mllvm -enable-loop-distribute -mllvm -enable-unroll-and-jam -mllvm -enable-loop-flatten -mllvm -unroll-runtime-multi-exit -mllvm -aggressive-ext-opt -fno-math-errno -fno-trapping-math -falign-functions=32 -fno-semantic-interposition -fno-omit-frame-pointer -fcf-protection=none -mharden-sls=none -flto=thin -fwhole-program-vtables -mllvm -adce-remove-loops -mllvm -enable-ext-tsp-block-placement=1 -mllvm -enable-gvn-hoist -mllvm -enable-dfa-jump-thread -Wl,--lto-CGO3 -Wl,--gc-sections -Wl,--icf=all -Wl,--lto-O3,-O3,-Bsymbolic-functions,--as-needed -march=native -mtune=native -maes -mbmi2 -mpclmul -fuse-ld=lld -fcf-protection=none -mharden-sls=none -flto=thin -fwhole-program-vtables CMakeFiles/cmTC_ba85d.dir/testCCompiler.c.o -o cmTC_ba85d   && :
    PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
     #0 0x00005626b2356327 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld+0x7956327)
     #1 0x00005626b2356afd SignalHandler(int) Signals.cpp:0:0
     #2 0x00007f1120c96830 (/usr/lib/libc.so.6+0x296830)
     #3 0x00005626b1d4cf14 llvm::TargetPassConfig::addPass(llvm::Pass*) (/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld+0x734cf14)
     #4 0x00005626b1d50f0b llvm::TargetPassConfig::addRegAssignAndRewriteOptimized() (/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld+0x7350f0b)
     #5 0x00005626b1feda28 (anonymous namespace)::X86PassConfig::addRegAssignAndRewriteOptimized() (.14f7859427a5b994a1575b581a8e1564) X86TargetMachine.cpp:0:0
     #6 0x00005626b1d4dd8f llvm::TargetPassConfig::addOptimizedRegAlloc() (/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld+0x734dd8f)
     #7 0x00005626b1d4ac9e llvm::TargetPassConfig::addMachinePasses() (/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld+0x734ac9e)
     #8 0x00005626b1d4a3f8 addPassesToGenerateCode(llvm::LLVMTargetMachine&, llvm::legacy::PassManagerBase&, bool, llvm::MachineModuleInfoWrapperPass&) LLVMTargetMachine.cpp:0:0
     #9 0x00005626b1ef5e7e llvm::LLVMTargetMachine::addPassesToEmitFile(llvm::legacy::PassManagerBase&, llvm::raw_pwrite_stream&, llvm::raw_pwrite_stream*, llvm::CodeGenFileType, bool, llvm::MachineModuleInfoWrapperPass*) (/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld+0x74f5e7e)
    #10 0x00005626b1d96d3d codegen(llvm::lto::Config const&, llvm::TargetMachine*, std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>, unsigned int, llvm::Module&, llvm::ModuleSummaryIndex const&) LTOBackend.cpp:0:0
    #11 0x00005626b1d968a1 llvm::lto::thinBackend(llvm::lto::Config const&, unsigned int, std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>, llvm::Module&, llvm::ModuleSummaryIndex const&, llvm::DenseMap<llvm::StringRef, std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>>> const&, llvm::DenseMap<unsigned long, llvm::GlobalValueSummary*, llvm::DenseMapInfo<unsigned long, void>, llvm::detail::DenseMapPair<unsigned long, llvm::GlobalValueSummary*>> const&, llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int>>, llvm::SmallVector<std::pair<llvm::StringRef, llvm::BitcodeModule>, 0u>>*, std::vector<unsigned char, std::allocator<unsigned char>> const&)::$_1::operator()(llvm::Module&, llvm::TargetMachine*, std::unique_ptr<llvm::ToolOutputFile, std::default_delete<llvm::ToolOutputFile>>) const LTOBackend.cpp:0:0
    #12 0x00005626b1d92697 llvm::lto::thinBackend(llvm::lto::Config const&, unsigned int, std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>, llvm::Module&, llvm::ModuleSummaryIndex const&, llvm::DenseMap<llvm::StringRef, std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>>> const&, llvm::DenseMap<unsigned long, llvm::GlobalValueSummary*, llvm::DenseMapInfo<unsigned long, void>, llvm::detail::DenseMapPair<unsigned long, llvm::GlobalValueSummary*>> const&, llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int>>, llvm::SmallVector<std::pair<llvm::StringRef, llvm::BitcodeModule>, 0u>>*, std::vector<unsigned char, std::allocator<unsigned char>> const&) (/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld+0x7392697)
    #13 0x00005626b20d1f76 (anonymous namespace)::InProcessThinBackend::runThinLTOBackendThread(std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>, std::function<llvm::Expected<std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>> (unsigned int, llvm::StringRef, llvm::Twine const&)>, unsigned int, llvm::BitcodeModule, llvm::ModuleSummaryIndex&, llvm::DenseMap<llvm::StringRef, std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>>> const&, llvm::DenseSet<llvm::ValueInfo, llvm::DenseMapInfo<llvm::ValueInfo, void>> const&, std::map<unsigned long, llvm::GlobalValue::LinkageTypes, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, llvm::GlobalValue::LinkageTypes>>> const&, llvm::DenseMap<unsigned long, llvm::GlobalValueSummary*, llvm::DenseMapInfo<unsigned long, void>, llvm::detail::DenseMapPair<unsigned long, llvm::GlobalValueSummary*>> const&, llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int>>, llvm::SmallVector<std::pair<llvm::StringRef, llvm::BitcodeModule>, 0u>>&)::'lambda'(std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>)::operator()(std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>) const LTO.cpp:0:0
    #14 0x00005626b20d1018 std::_Function_handler<void (), std::_Bind<(anonymous namespace)::InProcessThinBackend::start(unsigned int, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>>> const&, llvm::DenseSet<llvm::ValueInfo, llvm::DenseMapInfo<llvm::ValueInfo, void>> const&, std::map<unsigned long, llvm::GlobalValue::LinkageTypes, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, llvm::GlobalValue::LinkageTypes>>> const&, llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int>>, llvm::SmallVector<std::pair<llvm::StringRef, llvm::BitcodeModule>, 0u>>&)::'lambda'(llvm::BitcodeModule, llvm::ModuleSummaryIndex&, llvm::DenseMap<llvm::StringRef, std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>>> const&, llvm::DenseSet<llvm::ValueInfo, llvm::DenseMapInfo<llvm::ValueInfo, void>> const&, std::map<unsigned long, llvm::GlobalValue::LinkageTypes, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, llvm::GlobalValue::LinkageTypes>>> const&, llvm::DenseMap<unsigned long, llvm::GlobalValueSummary*, llvm::DenseMapInfo<unsigned long, void>, llvm::detail::DenseMapPair<unsigned long, llvm::GlobalValueSummary*>> const&, llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int>>, llvm::SmallVector<std::pair<llvm::StringRef, llvm::BitcodeModule>, 0u>>&) (llvm::BitcodeModule, std::reference_wrapper<llvm::ModuleSummaryIndex>, std::reference_wrapper<llvm::DenseMap<llvm::StringRef, std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>>> const>, std::reference_wrapper<llvm::DenseSet<llvm::ValueInfo, llvm::DenseMapInfo<llvm::ValueInfo, void>> const>, std::reference_wrapper<std::map<unsigned long, llvm::GlobalValue::LinkageTypes, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, llvm::GlobalValue::LinkageTypes>>> const>, std::reference_wrapper<llvm::DenseMap<unsigned long, llvm::GlobalValueSummary*, llvm::DenseMapInfo<unsigned long, void>, llvm::detail::DenseMapPair<unsigned long, llvm::GlobalValueSummary*>> const>, std::reference_wrapper<llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int>>, llvm::SmallVector<std::pair<llvm::StringRef, llvm::BitcodeModule>, 0u>>>)>>::_M_invoke(std::_Any_data const&) LTO.cpp:0:0
    #15 0x00005626adb59438 (/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld+0x3159438)
    #16 0x00005626b23dce31 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) Driver.cpp:0:0
    #17 0x00007f1120cf54f8 (/usr/lib/libc.so.6+0x2f54f8)
    #18 0x00007f1120cf544f __pthread_once (/usr/lib/libc.so.6+0x2f544f)
    #19 0x00005626b20558e6 void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) Parallel.cpp:0:0
    #20 0x00005626b2055791 std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) Parallel.cpp:0:0
    #21 0x00005626ad5d659a (/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld+0x2bd659a)
    #22 0x00005626b202c69d std::__future_base::_State_baseV2::wait() Parallel.cpp:0:0
    #23 0x00005626b20578ad llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld+0x76578ad)
    #24 0x00005626b205749f void* llvm::thread::ThreadProxy<std::tuple<llvm::StdThreadPool::grow(int)::$_0>>(void*) ThreadPool.cpp:0:0
    #25 0x00007f1120cf12dc (/usr/lib/libc.so.6+0x2f12dc)
    #26 0x00007f1120db77b8 (/usr/lib/libc.so.6+0x3b77b8)
    clang-19: error: unable to execute command: Segmentation fault

[aaupov]

Can you please provide a repro? I.e. how clang was built, what profile was used, etc

[ms178]

I cannot tell if it is still worth it to hunt that one down. The issue was seen while using the llvm-built-scripts from my repo at that time: https://github.com/ms178/archpkgbuilds/tree/main/toolchain-experimental/llvm-bolt-scripts-master

I can't tell if that particular crash still reproduces on llvm-git main as I have refined the scripts since reporting the bug and the used CFLAGS/LDFLAGS might have changed in the newer version. You might need to adjust the flags in the stage3-without-sampling-script with the ones mentioned in the crash report. I could build a newer revision on September 2nd just fine.

If you still want to pursue this, you'd need:

1. build_stage1.bash
2. build_stage2-prof-generate.bash
3. build_stage2-prof-use-lto.bash
4. build_stage3-bolt-without-sampling.bash
5. build_stage3-train.bash
6. For convenience, this one uses all of the ones needed in the correct order: full_workflow.bash

Just download all of them into a folder, make them executable and then use full_workload.bash. You'd need to alter the build_stage1-script to exclude the locally used PGO-file: -fprofile-use=/home/marcus/Downloads/clang-20.profdata which I use to build a more optimized stage1 compiler. Note that the crash reproduced with the stage3-without-sampling-script which doesn't use perf on LBR-compatible CPUs but uses the classic instrumentation method. If you have perf installed on a LBR-compatible machine, just delete perf before trying it out as having perf installed triggers the usage of a different script instead.

I am also fine to close this one if you don't see any value in this report any longer.

[ms178]

Closing in protest due to (https://seylaw.blogspot.com/2025/05/when-compiler-engineers-act-as-judges.html)