Crash in wasm-ld (20.0.0) with Full LTO

[fs-eire]

I try to upgrade emsdk to 3.1.74 for onnxruntime and got this crash with build. The build was fine with emsdk 3.1.59.

Error logs:

2025-01-08 12:09:00,886 build [INFO] - C:\Program Files\CMake\bin\cmake.EXE --build C:\code\onnxruntime\\build\Windows\Release --config Release --target onnxruntime_webassembly
[0/2] Re-checking globbed directories...
[1/1] Linking CXX executable ort-wasm-simd-threaded.mjs
FAILED: ort-wasm-simd-threaded.mjs
C:\Windows\system32\cmd.exe /C "cd . && C:\code\onnxruntime\cmake\external\emsdk\upstream\emscripten\em++.bat -ffunction-sections -fdata-sections -flto -msimd128 -pthread -Wno-pthreads-mem-growth -O3 -DNDEBUG -s ALLOW_UNIMPLEMENTED_SYSCALLS=1 -s DEFAULT_TO_CXX=1    --post-js C:/code/onnxruntime/onnxruntime/wasm/js_post_js.js -s EXPORTED_RUNTIME_METHODS=[stackAlloc,stackRestore,stackSave,UTF8ToString,stringToUTF8,lengthBytesUTF8,getValue,setValue] -s EXPORTED_FUNCTIONS=_malloc,_free -s MAXIMUM_MEMORY=4294967296 -s EXIT_RUNTIME=0 -s ALLOW_MEMORY_GROWTH=1 -s MODULARIZE=1 -s EXPORT_ALL=0 -s VERBOSE=0 -s FILESYSTEM=0 -s INCOMING_MODULE_JS_API=[locateFile,instantiateWasm,wasmBinary] -s WASM_BIGINT=1 "-s DISABLE_EXCEPTION_CATCHING=0" --no-entry --pre-js C:/code/onnxruntime/onnxruntime/wasm/pre.js -s MALLOC=dlmalloc -s STACK_SIZE=5242880 -s ASSERTIONS=0 -s SAFE_HEAP=0 -s STACK_OVERFLOW_CHECK=0 --closure 1 -s DISABLE_EXCEPTION_THROWING=0 -s EXPORT_NAME=ortWasmThreaded -s DEFAULT_PTHREAD_STACK_SIZE=131072 -s PTHREAD_POOL_SIZE=Module[\"numThreads\"]-1 @CMakeFiles\onnxruntime_webassembly.rsp -o ort-wasm-simd-threaded.mjs && cd ."
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: C:/code/onnxruntime/cmake/external/emsdk/upstream/bin\\wasm-ld.exe -o ort-wasm-simd-threaded.wasm -LC:\\code\\onnxruntime\\cmake\\external\\emsdk\\upstream\\emscripten\\cache\\sysroot\\lib\\wasm32-emscripten\\lto CMakeFiles/onnxruntime_webassembly.dir/C_/code/onnxruntime/onnxruntime/wasm/api.cc.o _deps/protobuf-build/libprotobuf-lite.a _deps/onnx-build/libonnx.a _deps/onnx-build/libonnx_proto.a libonnxruntime_common.a libonnxruntime_lora.a libonnxruntime_flatbuffers.a libonnxruntime_framework.a libonnxruntime_graph.a libonnxruntime_mlas.a libonnxruntime_optimizer.a libonnxruntime_providers.a libonnxruntime_session.a libonnxruntime_util.a _deps/re2-build/libre2.a _deps/protobuf-build/libprotobuf-lite.a libonnxruntime_lora.a libonnxruntime_framework.a lib/libortcustomops.a lib/libocos_operators.a lib/libnoexcep_operators.a _deps/re2-build/libre2.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_reflection.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_private_handle_accessor.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_internal.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_commandlineflag.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_commandlineflag_internal.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_marshalling.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_config.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_program_name.a _deps/abseil_cpp-build/absl/strings/libabsl_cord.a _deps/abseil_cpp-build/absl/strings/libabsl_cordz_info.a _deps/abseil_cpp-build/absl/strings/libabsl_cord_internal.a _deps/abseil_cpp-build/absl/strings/libabsl_cordz_functions.a _deps/abseil_cpp-build/absl/strings/libabsl_cordz_handle.a _deps/abseil_cpp-build/absl/crc/libabsl_crc_cord_state.a _deps/abseil_cpp-build/absl/crc/libabsl_crc32c.a _deps/abseil_cpp-build/absl/crc/libabsl_crc_internal.a _deps/abseil_cpp-build/absl/crc/libabsl_crc_cpu_detect.a _deps/abseil_cpp-build/absl/container/libabsl_raw_hash_set.a _deps/abseil_cpp-build/absl/container/libabsl_hashtablez_sampler.a _deps/abseil_cpp-build/absl/profiling/libabsl_exponential_biased.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_conditions.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_check_op.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_message.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_format.a _deps/abseil_cpp-build/absl/strings/libabsl_str_format_internal.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_log_sink_set.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_globals.a _deps/abseil_cpp-build/absl/log/libabsl_log_sink.a _deps/abseil_cpp-build/absl/log/libabsl_log_entry.a _deps/abseil_cpp-build/absl/log/libabsl_log_globals.a _deps/abseil_cpp-build/absl/hash/libabsl_hash.a _deps/abseil_cpp-build/absl/hash/libabsl_city.a _deps/abseil_cpp-build/absl/hash/libabsl_low_level_hash.a _deps/abseil_cpp-build/absl/types/libabsl_bad_variant_access.a _deps/abseil_cpp-build/absl/log/libabsl_vlog_config_internal.a _deps/abseil_cpp-build/absl/synchronization/libabsl_synchronization.a _deps/abseil_cpp-build/absl/synchronization/libabsl_kernel_timeout_internal.a _deps/abseil_cpp-build/absl/time/libabsl_time.a _deps/abseil_cpp-build/absl/time/libabsl_time_zone.a _deps/abseil_cpp-build/absl/time/libabsl_civil_time.a _deps/abseil_cpp-build/absl/synchronization/libabsl_graphcycles_internal.a _deps/abseil_cpp-build/absl/types/libabsl_bad_optional_access.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_fnmatch.a _deps/abseil_cpp-build/absl/debugging/libabsl_examine_stack.a _deps/abseil_cpp-build/absl/debugging/libabsl_symbolize.a _deps/abseil_cpp-build/absl/base/libabsl_malloc_internal.a _deps/abseil_cpp-build/absl/debugging/libabsl_demangle_internal.a _deps/abseil_cpp-build/absl/debugging/libabsl_demangle_rust.a _deps/abseil_cpp-build/absl/debugging/libabsl_decode_rust_punycode.a _deps/abseil_cpp-build/absl/debugging/libabsl_utf8_for_code_point.a _deps/abseil_cpp-build/absl/debugging/libabsl_stacktrace.a _deps/abseil_cpp-build/absl/debugging/libabsl_debugging_internal.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_proto.a _deps/abseil_cpp-build/absl/base/libabsl_strerror.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_nullguard.a _deps/abseil_cpp-build/absl/strings/libabsl_strings.a _deps/abseil_cpp-build/absl/strings/libabsl_strings_internal.a _deps/abseil_cpp-build/absl/numeric/libabsl_int128.a _deps/abseil_cpp-build/absl/strings/libabsl_string_view.a _deps/abseil_cpp-build/absl/base/libabsl_base.a _deps/abseil_cpp-build/absl/base/libabsl_spinlock_wait.a _deps/abseil_cpp-build/absl/base/libabsl_throw_delegate.a _deps/abseil_cpp-build/absl/base/libabsl_raw_logging_internal.a _deps/abseil_cpp-build/absl/base/libabsl_log_severity.a C:\\code\\onnxruntime\\cmake\\external\\emsdk\\upstream\\emscripten\\cache\\sysroot\\lib\\wasm32-emscripten\\lto\\crtbegin.o -lGL-mt-getprocaddr -lal -lhtml5 -lbulkmemory -lstubs -lnoexit -lc-mt -ldlmalloc-mt -lcompiler_rt-mt -lc++-mt-noexcept -lc++abi-mt-noexcept -lsockets-mt -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr C:\\Users\\yulongw\\AppData\\Local\\Temp\\tmpul8tk14rlibemscripten_js_symbols.so --import-memory --shared-memory --strip-debug -u__cxa_atexit --export=malloc --export=free --export=_emscripten_stack_alloc --export=_emscripten_thread_free_data --export=_emscripten_thread_crashed --export=__wasm_call_ctors --export=_emscripten_tls_init --export=_emscripten_thread_init --export=emscripten_stack_get_current --export=_emscripten_stack_restore --export=emscripten_stack_set_limits --export=_emscripten_thread_exit --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-table -z stack-size=5242880 --max-memory=4294967296 --initial-memory=16777216 --no-entry --table-base=1 --global-base=1024
1.      Running pass "cgscc(function-attrs)" on module "ld-temp.o"
Exception Code: 0xC0000005
 #0 0x00007ff63b177ea9 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x707ea9)
 #1 0x00007ff63b17952c (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x70952c)
 #2 0x00007ff63b178aab (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x708aab)
 #3 0x00007ff63b688f8b (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0xc18f8b)
 #4 0x00007ff63b9091de (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0xe991de)
 #5 0x00007ff63b68ab31 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0xc1ab31)
 #6 0x00007ff63b474837 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0xa04837)
 #7 0x00007ff63b0f2fd0 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x682fd0)
 #8 0x00007ff63b0f5116 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x685116)
 #9 0x00007ff63ac1d8d6 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x1ad8d6)
#10 0x00007ff63af24b27 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x4b4b27)
#11 0x00007ff63aef40f4 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x4840f4)
#12 0x00007ff63ab3b396 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0xcb396)
#13 0x00007ff63ab355d5 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0xc55d5)
#14 0x00007ff63aa9ca1e (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x2ca1e)
#15 0x00007ff63aa71225 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x1225)
#16 0x00007ff63aa71dd4 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x1dd4)
#17 0x00007ff63d2f19f8 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x28819f8)
#18 0x00007ffdbc34e8d7 (C:\Windows\System32\KERNEL32.DLL+0x2e8d7)
#19 0x00007ffdbe3dfbcc (C:\Windows\SYSTEM32\ntdll.dll+0xdfbcc)
em++: error: 'C:/code/onnxruntime/cmake/external/emsdk/upstream/bin\wasm-ld.exe -o ort-wasm-simd-threaded.wasm -LC:\code\onnxruntime\cmake\external\emsdk\upstream\emscripten\cache\sysroot\lib\wasm32-emscripten\lto CMakeFiles/onnxruntime_webassembly.dir/C_/code/onnxruntime/onnxruntime/wasm/api.cc.o _deps/protobuf-build/libprotobuf-lite.a _deps/onnx-build/libonnx.a _deps/onnx-build/libonnx_proto.a libonnxruntime_common.a libonnxruntime_lora.a libonnxruntime_flatbuffers.a libonnxruntime_framework.a libonnxruntime_graph.a libonnxruntime_mlas.a libonnxruntime_optimizer.a libonnxruntime_providers.a libonnxruntime_session.a libonnxruntime_util.a _deps/re2-build/libre2.a _deps/protobuf-build/libprotobuf-lite.a libonnxruntime_lora.a libonnxruntime_framework.a lib/libortcustomops.a lib/libocos_operators.a lib/libnoexcep_operators.a _deps/re2-build/libre2.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_reflection.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_private_handle_accessor.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_internal.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_commandlineflag.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_commandlineflag_internal.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_marshalling.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_config.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_program_name.a _deps/abseil_cpp-build/absl/strings/libabsl_cord.a _deps/abseil_cpp-build/absl/strings/libabsl_cordz_info.a _deps/abseil_cpp-build/absl/strings/libabsl_cord_internal.a _deps/abseil_cpp-build/absl/strings/libabsl_cordz_functions.a _deps/abseil_cpp-build/absl/strings/libabsl_cordz_handle.a _deps/abseil_cpp-build/absl/crc/libabsl_crc_cord_state.a _deps/abseil_cpp-build/absl/crc/libabsl_crc32c.a _deps/abseil_cpp-build/absl/crc/libabsl_crc_internal.a _deps/abseil_cpp-build/absl/crc/libabsl_crc_cpu_detect.a _deps/abseil_cpp-build/absl/container/libabsl_raw_hash_set.a _deps/abseil_cpp-build/absl/container/libabsl_hashtablez_sampler.a _deps/abseil_cpp-build/absl/profiling/libabsl_exponential_biased.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_conditions.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_check_op.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_message.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_format.a _deps/abseil_cpp-build/absl/strings/libabsl_str_format_internal.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_log_sink_set.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_globals.a _deps/abseil_cpp-build/absl/log/libabsl_log_sink.a _deps/abseil_cpp-build/absl/log/libabsl_log_entry.a _deps/abseil_cpp-build/absl/log/libabsl_log_globals.a _deps/abseil_cpp-build/absl/hash/libabsl_hash.a _deps/abseil_cpp-build/absl/hash/libabsl_city.a _deps/abseil_cpp-build/absl/hash/libabsl_low_level_hash.a _deps/abseil_cpp-build/absl/types/libabsl_bad_variant_access.a _deps/abseil_cpp-build/absl/log/libabsl_vlog_config_internal.a _deps/abseil_cpp-build/absl/synchronization/libabsl_synchronization.a _deps/abseil_cpp-build/absl/synchronization/libabsl_kernel_timeout_internal.a _deps/abseil_cpp-build/absl/time/libabsl_time.a _deps/abseil_cpp-build/absl/time/libabsl_time_zone.a _deps/abseil_cpp-build/absl/time/libabsl_civil_time.a _deps/abseil_cpp-build/absl/synchronization/libabsl_graphcycles_internal.a _deps/abseil_cpp-build/absl/types/libabsl_bad_optional_access.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_fnmatch.a _deps/abseil_cpp-build/absl/debugging/libabsl_examine_stack.a _deps/abseil_cpp-build/absl/debugging/libabsl_symbolize.a _deps/abseil_cpp-build/absl/base/libabsl_malloc_internal.a _deps/abseil_cpp-build/absl/debugging/libabsl_demangle_internal.a _deps/abseil_cpp-build/absl/debugging/libabsl_demangle_rust.a _deps/abseil_cpp-build/absl/debugging/libabsl_decode_rust_punycode.a _deps/abseil_cpp-build/absl/debugging/libabsl_utf8_for_code_point.a _deps/abseil_cpp-build/absl/debugging/libabsl_stacktrace.a _deps/abseil_cpp-build/absl/debugging/libabsl_debugging_internal.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_proto.a _deps/abseil_cpp-build/absl/base/libabsl_strerror.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_nullguard.a _deps/abseil_cpp-build/absl/strings/libabsl_strings.a _deps/abseil_cpp-build/absl/strings/libabsl_strings_internal.a _deps/abseil_cpp-build/absl/numeric/libabsl_int128.a _deps/abseil_cpp-build/absl/strings/libabsl_string_view.a _deps/abseil_cpp-build/absl/base/libabsl_base.a _deps/abseil_cpp-build/absl/base/libabsl_spinlock_wait.a _deps/abseil_cpp-build/absl/base/libabsl_throw_delegate.a _deps/abseil_cpp-build/absl/base/libabsl_raw_logging_internal.a _deps/abseil_cpp-build/absl/base/libabsl_log_severity.a C:\code\onnxruntime\cmake\external\emsdk\upstream\emscripten\cache\sysroot\lib\wasm32-emscripten\lto\crtbegin.o -lGL-mt-getprocaddr -lal -lhtml5 -lbulkmemory -lstubs -lnoexit -lc-mt -ldlmalloc-mt -lcompiler_rt-mt -lc++-mt-noexcept -lc++abi-mt-noexcept -lsockets-mt -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr C:\Users\yulongw\AppData\Local\Temp\tmpul8tk14rlibemscripten_js_symbols.so --import-memory --shared-memory --strip-debug -u__cxa_atexit --export=malloc --export=free --export=_emscripten_stack_alloc --export=_emscripten_thread_free_data --export=_emscripten_thread_crashed --export=__wasm_call_ctors --export=_emscripten_tls_init --export=_emscripten_thread_init --export=emscripten_stack_get_current --export=_emscripten_stack_restore --export=emscripten_stack_set_limits --export=_emscripten_thread_exit --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-table -z stack-size=5242880 --max-memory=4294967296 --initial-memory=16777216 --no-entry --table-base=1 --global-base=1024' failed (returned 3221225477)
ninja: build stopped: subcommand failed.

wasm-ld version

C:\code\onnxruntime\cmake\external\emsdk\upstream\bin>wasm-ld --version
LLD 20.0.0 (Cswircachegitchromium.googlesource.com-external-github.com-llvm-llvm--project 322eb1a92e6d4266184060346616fa0dbe39e731)

Reproduce steps:

1. clone https://github.com/microsoft/onnxruntime
   • install cmake, python, nodejs
   • pip install ninja
2. use branch fs-eire/emsdk_upgrade_test
3. build with build --config Release --build_wasm --enable_wasm_simd --enable_wasm_threads --use_extensions --cmake_extra_defines onnxruntime_WEBASSEMBLY_DEFAULT_EXTENSION_FLAGS=ON --target onnxruntime_webassembly --skip_tests --enable_wasm_api_exception_catching --disable_rtti

Please let me know if more information is needed for investigation.

[llvmbot]

@llvm/issue-subscribers-lld-wasm

Author: Yulong Wang (fs-eire)

I try to upgrade emsdk to 3.1.74 for onnxruntime and got this crash with build. The build was fine with emsdk 3.1.59.

Error logs:

2025-01-08 12:09:00,886 build [INFO] - C:\Program Files\CMake\bin\cmake.EXE --build C:\code\onnxruntime\\build\Windows\Release --config Release --target onnxruntime_webassembly
[0/2] Re-checking globbed directories...
[1/1] Linking CXX executable ort-wasm-simd-threaded.mjs
FAILED: ort-wasm-simd-threaded.mjs
C:\Windows\system32\cmd.exe /C "cd . &amp;&amp; C:\code\onnxruntime\cmake\external\emsdk\upstream\emscripten\em++.bat -ffunction-sections -fdata-sections -flto -msimd128 -pthread -Wno-pthreads-mem-growth -O3 -DNDEBUG -s ALLOW_UNIMPLEMENTED_SYSCALLS=1 -s DEFAULT_TO_CXX=1    --post-js C:/code/onnxruntime/onnxruntime/wasm/js_post_js.js -s EXPORTED_RUNTIME_METHODS=[stackAlloc,stackRestore,stackSave,UTF8ToString,stringToUTF8,lengthBytesUTF8,getValue,setValue] -s EXPORTED_FUNCTIONS=_malloc,_free -s MAXIMUM_MEMORY=4294967296 -s EXIT_RUNTIME=0 -s ALLOW_MEMORY_GROWTH=1 -s MODULARIZE=1 -s EXPORT_ALL=0 -s VERBOSE=0 -s FILESYSTEM=0 -s INCOMING_MODULE_JS_API=[locateFile,instantiateWasm,wasmBinary] -s WASM_BIGINT=1 "-s DISABLE_EXCEPTION_CATCHING=0" --no-entry --pre-js C:/code/onnxruntime/onnxruntime/wasm/pre.js -s MALLOC=dlmalloc -s STACK_SIZE=5242880 -s ASSERTIONS=0 -s SAFE_HEAP=0 -s STACK_OVERFLOW_CHECK=0 --closure 1 -s DISABLE_EXCEPTION_THROWING=0 -s EXPORT_NAME=ortWasmThreaded -s DEFAULT_PTHREAD_STACK_SIZE=131072 -s PTHREAD_POOL_SIZE=Module[\"numThreads\"]-1 @<!-- -->CMakeFiles\onnxruntime_webassembly.rsp -o ort-wasm-simd-threaded.mjs &amp;&amp; cd ."
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: C:/code/onnxruntime/cmake/external/emsdk/upstream/bin\\wasm-ld.exe -o ort-wasm-simd-threaded.wasm -LC:\\code\\onnxruntime\\cmake\\external\\emsdk\\upstream\\emscripten\\cache\\sysroot\\lib\\wasm32-emscripten\\lto CMakeFiles/onnxruntime_webassembly.dir/C_/code/onnxruntime/onnxruntime/wasm/api.cc.o _deps/protobuf-build/libprotobuf-lite.a _deps/onnx-build/libonnx.a _deps/onnx-build/libonnx_proto.a libonnxruntime_common.a libonnxruntime_lora.a libonnxruntime_flatbuffers.a libonnxruntime_framework.a libonnxruntime_graph.a libonnxruntime_mlas.a libonnxruntime_optimizer.a libonnxruntime_providers.a libonnxruntime_session.a libonnxruntime_util.a _deps/re2-build/libre2.a _deps/protobuf-build/libprotobuf-lite.a libonnxruntime_lora.a libonnxruntime_framework.a lib/libortcustomops.a lib/libocos_operators.a lib/libnoexcep_operators.a _deps/re2-build/libre2.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_reflection.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_private_handle_accessor.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_internal.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_commandlineflag.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_commandlineflag_internal.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_marshalling.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_config.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_program_name.a _deps/abseil_cpp-build/absl/strings/libabsl_cord.a _deps/abseil_cpp-build/absl/strings/libabsl_cordz_info.a _deps/abseil_cpp-build/absl/strings/libabsl_cord_internal.a _deps/abseil_cpp-build/absl/strings/libabsl_cordz_functions.a _deps/abseil_cpp-build/absl/strings/libabsl_cordz_handle.a _deps/abseil_cpp-build/absl/crc/libabsl_crc_cord_state.a _deps/abseil_cpp-build/absl/crc/libabsl_crc32c.a _deps/abseil_cpp-build/absl/crc/libabsl_crc_internal.a _deps/abseil_cpp-build/absl/crc/libabsl_crc_cpu_detect.a _deps/abseil_cpp-build/absl/container/libabsl_raw_hash_set.a _deps/abseil_cpp-build/absl/container/libabsl_hashtablez_sampler.a _deps/abseil_cpp-build/absl/profiling/libabsl_exponential_biased.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_conditions.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_check_op.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_message.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_format.a _deps/abseil_cpp-build/absl/strings/libabsl_str_format_internal.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_log_sink_set.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_globals.a _deps/abseil_cpp-build/absl/log/libabsl_log_sink.a _deps/abseil_cpp-build/absl/log/libabsl_log_entry.a _deps/abseil_cpp-build/absl/log/libabsl_log_globals.a _deps/abseil_cpp-build/absl/hash/libabsl_hash.a _deps/abseil_cpp-build/absl/hash/libabsl_city.a _deps/abseil_cpp-build/absl/hash/libabsl_low_level_hash.a _deps/abseil_cpp-build/absl/types/libabsl_bad_variant_access.a _deps/abseil_cpp-build/absl/log/libabsl_vlog_config_internal.a _deps/abseil_cpp-build/absl/synchronization/libabsl_synchronization.a _deps/abseil_cpp-build/absl/synchronization/libabsl_kernel_timeout_internal.a _deps/abseil_cpp-build/absl/time/libabsl_time.a _deps/abseil_cpp-build/absl/time/libabsl_time_zone.a _deps/abseil_cpp-build/absl/time/libabsl_civil_time.a _deps/abseil_cpp-build/absl/synchronization/libabsl_graphcycles_internal.a _deps/abseil_cpp-build/absl/types/libabsl_bad_optional_access.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_fnmatch.a _deps/abseil_cpp-build/absl/debugging/libabsl_examine_stack.a _deps/abseil_cpp-build/absl/debugging/libabsl_symbolize.a _deps/abseil_cpp-build/absl/base/libabsl_malloc_internal.a _deps/abseil_cpp-build/absl/debugging/libabsl_demangle_internal.a _deps/abseil_cpp-build/absl/debugging/libabsl_demangle_rust.a _deps/abseil_cpp-build/absl/debugging/libabsl_decode_rust_punycode.a _deps/abseil_cpp-build/absl/debugging/libabsl_utf8_for_code_point.a _deps/abseil_cpp-build/absl/debugging/libabsl_stacktrace.a _deps/abseil_cpp-build/absl/debugging/libabsl_debugging_internal.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_proto.a _deps/abseil_cpp-build/absl/base/libabsl_strerror.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_nullguard.a _deps/abseil_cpp-build/absl/strings/libabsl_strings.a _deps/abseil_cpp-build/absl/strings/libabsl_strings_internal.a _deps/abseil_cpp-build/absl/numeric/libabsl_int128.a _deps/abseil_cpp-build/absl/strings/libabsl_string_view.a _deps/abseil_cpp-build/absl/base/libabsl_base.a _deps/abseil_cpp-build/absl/base/libabsl_spinlock_wait.a _deps/abseil_cpp-build/absl/base/libabsl_throw_delegate.a _deps/abseil_cpp-build/absl/base/libabsl_raw_logging_internal.a _deps/abseil_cpp-build/absl/base/libabsl_log_severity.a C:\\code\\onnxruntime\\cmake\\external\\emsdk\\upstream\\emscripten\\cache\\sysroot\\lib\\wasm32-emscripten\\lto\\crtbegin.o -lGL-mt-getprocaddr -lal -lhtml5 -lbulkmemory -lstubs -lnoexit -lc-mt -ldlmalloc-mt -lcompiler_rt-mt -lc++-mt-noexcept -lc++abi-mt-noexcept -lsockets-mt -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr C:\\Users\\yulongw\\AppData\\Local\\Temp\\tmpul8tk14rlibemscripten_js_symbols.so --import-memory --shared-memory --strip-debug -u__cxa_atexit --export=malloc --export=free --export=_emscripten_stack_alloc --export=_emscripten_thread_free_data --export=_emscripten_thread_crashed --export=__wasm_call_ctors --export=_emscripten_tls_init --export=_emscripten_thread_init --export=emscripten_stack_get_current --export=_emscripten_stack_restore --export=emscripten_stack_set_limits --export=_emscripten_thread_exit --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-table -z stack-size=5242880 --max-memory=4294967296 --initial-memory=16777216 --no-entry --table-base=1 --global-base=1024
1.      Running pass "cgscc(function-attrs)" on module "ld-temp.o"
Exception Code: 0xC0000005
 #<!-- -->0 0x00007ff63b177ea9 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x707ea9)
 #<!-- -->1 0x00007ff63b17952c (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x70952c)
 #<!-- -->2 0x00007ff63b178aab (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x708aab)
 #<!-- -->3 0x00007ff63b688f8b (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0xc18f8b)
 #<!-- -->4 0x00007ff63b9091de (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0xe991de)
 #<!-- -->5 0x00007ff63b68ab31 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0xc1ab31)
 #<!-- -->6 0x00007ff63b474837 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0xa04837)
 #<!-- -->7 0x00007ff63b0f2fd0 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x682fd0)
 #<!-- -->8 0x00007ff63b0f5116 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x685116)
 #<!-- -->9 0x00007ff63ac1d8d6 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x1ad8d6)
#<!-- -->10 0x00007ff63af24b27 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x4b4b27)
#<!-- -->11 0x00007ff63aef40f4 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x4840f4)
#<!-- -->12 0x00007ff63ab3b396 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0xcb396)
#<!-- -->13 0x00007ff63ab355d5 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0xc55d5)
#<!-- -->14 0x00007ff63aa9ca1e (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x2ca1e)
#<!-- -->15 0x00007ff63aa71225 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x1225)
#<!-- -->16 0x00007ff63aa71dd4 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x1dd4)
#<!-- -->17 0x00007ff63d2f19f8 (C:\code\onnxruntime\cmake\external\emsdk\upstream\bin\wasm-ld.exe+0x28819f8)
#<!-- -->18 0x00007ffdbc34e8d7 (C:\Windows\System32\KERNEL32.DLL+0x2e8d7)
#<!-- -->19 0x00007ffdbe3dfbcc (C:\Windows\SYSTEM32\ntdll.dll+0xdfbcc)
em++: error: 'C:/code/onnxruntime/cmake/external/emsdk/upstream/bin\wasm-ld.exe -o ort-wasm-simd-threaded.wasm -LC:\code\onnxruntime\cmake\external\emsdk\upstream\emscripten\cache\sysroot\lib\wasm32-emscripten\lto CMakeFiles/onnxruntime_webassembly.dir/C_/code/onnxruntime/onnxruntime/wasm/api.cc.o _deps/protobuf-build/libprotobuf-lite.a _deps/onnx-build/libonnx.a _deps/onnx-build/libonnx_proto.a libonnxruntime_common.a libonnxruntime_lora.a libonnxruntime_flatbuffers.a libonnxruntime_framework.a libonnxruntime_graph.a libonnxruntime_mlas.a libonnxruntime_optimizer.a libonnxruntime_providers.a libonnxruntime_session.a libonnxruntime_util.a _deps/re2-build/libre2.a _deps/protobuf-build/libprotobuf-lite.a libonnxruntime_lora.a libonnxruntime_framework.a lib/libortcustomops.a lib/libocos_operators.a lib/libnoexcep_operators.a _deps/re2-build/libre2.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_reflection.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_private_handle_accessor.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_internal.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_commandlineflag.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_commandlineflag_internal.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_marshalling.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_config.a _deps/abseil_cpp-build/absl/flags/libabsl_flags_program_name.a _deps/abseil_cpp-build/absl/strings/libabsl_cord.a _deps/abseil_cpp-build/absl/strings/libabsl_cordz_info.a _deps/abseil_cpp-build/absl/strings/libabsl_cord_internal.a _deps/abseil_cpp-build/absl/strings/libabsl_cordz_functions.a _deps/abseil_cpp-build/absl/strings/libabsl_cordz_handle.a _deps/abseil_cpp-build/absl/crc/libabsl_crc_cord_state.a _deps/abseil_cpp-build/absl/crc/libabsl_crc32c.a _deps/abseil_cpp-build/absl/crc/libabsl_crc_internal.a _deps/abseil_cpp-build/absl/crc/libabsl_crc_cpu_detect.a _deps/abseil_cpp-build/absl/container/libabsl_raw_hash_set.a _deps/abseil_cpp-build/absl/container/libabsl_hashtablez_sampler.a _deps/abseil_cpp-build/absl/profiling/libabsl_exponential_biased.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_conditions.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_check_op.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_message.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_format.a _deps/abseil_cpp-build/absl/strings/libabsl_str_format_internal.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_log_sink_set.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_globals.a _deps/abseil_cpp-build/absl/log/libabsl_log_sink.a _deps/abseil_cpp-build/absl/log/libabsl_log_entry.a _deps/abseil_cpp-build/absl/log/libabsl_log_globals.a _deps/abseil_cpp-build/absl/hash/libabsl_hash.a _deps/abseil_cpp-build/absl/hash/libabsl_city.a _deps/abseil_cpp-build/absl/hash/libabsl_low_level_hash.a _deps/abseil_cpp-build/absl/types/libabsl_bad_variant_access.a _deps/abseil_cpp-build/absl/log/libabsl_vlog_config_internal.a _deps/abseil_cpp-build/absl/synchronization/libabsl_synchronization.a _deps/abseil_cpp-build/absl/synchronization/libabsl_kernel_timeout_internal.a _deps/abseil_cpp-build/absl/time/libabsl_time.a _deps/abseil_cpp-build/absl/time/libabsl_time_zone.a _deps/abseil_cpp-build/absl/time/libabsl_civil_time.a _deps/abseil_cpp-build/absl/synchronization/libabsl_graphcycles_internal.a _deps/abseil_cpp-build/absl/types/libabsl_bad_optional_access.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_fnmatch.a _deps/abseil_cpp-build/absl/debugging/libabsl_examine_stack.a _deps/abseil_cpp-build/absl/debugging/libabsl_symbolize.a _deps/abseil_cpp-build/absl/base/libabsl_malloc_internal.a _deps/abseil_cpp-build/absl/debugging/libabsl_demangle_internal.a _deps/abseil_cpp-build/absl/debugging/libabsl_demangle_rust.a _deps/abseil_cpp-build/absl/debugging/libabsl_decode_rust_punycode.a _deps/abseil_cpp-build/absl/debugging/libabsl_utf8_for_code_point.a _deps/abseil_cpp-build/absl/debugging/libabsl_stacktrace.a _deps/abseil_cpp-build/absl/debugging/libabsl_debugging_internal.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_proto.a _deps/abseil_cpp-build/absl/base/libabsl_strerror.a _deps/abseil_cpp-build/absl/log/libabsl_log_internal_nullguard.a _deps/abseil_cpp-build/absl/strings/libabsl_strings.a _deps/abseil_cpp-build/absl/strings/libabsl_strings_internal.a _deps/abseil_cpp-build/absl/numeric/libabsl_int128.a _deps/abseil_cpp-build/absl/strings/libabsl_string_view.a _deps/abseil_cpp-build/absl/base/libabsl_base.a _deps/abseil_cpp-build/absl/base/libabsl_spinlock_wait.a _deps/abseil_cpp-build/absl/base/libabsl_throw_delegate.a _deps/abseil_cpp-build/absl/base/libabsl_raw_logging_internal.a _deps/abseil_cpp-build/absl/base/libabsl_log_severity.a C:\code\onnxruntime\cmake\external\emsdk\upstream\emscripten\cache\sysroot\lib\wasm32-emscripten\lto\crtbegin.o -lGL-mt-getprocaddr -lal -lhtml5 -lbulkmemory -lstubs -lnoexit -lc-mt -ldlmalloc-mt -lcompiler_rt-mt -lc++-mt-noexcept -lc++abi-mt-noexcept -lsockets-mt -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr C:\Users\yulongw\AppData\Local\Temp\tmpul8tk14rlibemscripten_js_symbols.so --import-memory --shared-memory --strip-debug -u__cxa_atexit --export=malloc --export=free --export=_emscripten_stack_alloc --export=_emscripten_thread_free_data --export=_emscripten_thread_crashed --export=__wasm_call_ctors --export=_emscripten_tls_init --export=_emscripten_thread_init --export=emscripten_stack_get_current --export=_emscripten_stack_restore --export=emscripten_stack_set_limits --export=_emscripten_thread_exit --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-table -z stack-size=5242880 --max-memory=4294967296 --initial-memory=16777216 --no-entry --table-base=1 --global-base=1024' failed (returned 3221225477)
ninja: build stopped: subcommand failed.

wasm-ld version

C:\code\onnxruntime\cmake\external\emsdk\upstream\bin>wasm-ld --version
LLD 20.0.0 (Cswircachegitchromium.googlesource.com-external-github.com-llvm-llvm--project 322eb1a92e6d4266184060346616fa0dbe39e731)

Reproduce steps:

1. clone https://github.com/microsoft/onnxruntime
   • install cmake, python, nodejs
   • pip install ninja
2. use branch fs-eire/emsdk_upgrade_test
3. build with build --config Release --build_wasm --enable_wasm_simd --enable_wasm_threads --use_extensions --cmake_extra_defines onnxruntime_WEBASSEMBLY_DEFAULT_EXTENSION_FLAGS=ON --target onnxruntime_webassembly --skip_tests --enable_wasm_api_exception_catching --disable_rtti

Please let me know if more information is needed for investigation.

[sbc100]

This looks like an LTO crash, which means this is really more of a compiler crash than a linker crash.

Are you able to build successfully without LTO? It would much easier to debug with a smaller repro such as a single object file.

Failing that adding -Wl,--reproduce=repro.tar and then attaching repro.tar to this ticket would be very usefull.

[fs-eire]

This looks like an LTO crash, which means this is really more of a compiler crash than a linker crash.

Are you able to build successfully without LTO? It would much easier to debug with a smaller repro such as a single object file.

Failing that adding -Wl,--reproduce=repro.tar and then attaching repro.tar to this ticket would be very usefull.

I tried to build without "-flto". The build passes without LTO.

[sbc100]

I tried to build without "-flto". The build passes without LTO.

Thats unfortunate since it means that the repo case is the entire program :) Hopefully we can use llvm-reduce on it.

[fs-eire]

What do you suggest me to do? I am currently appending this to my CMakeFile:

    string(APPEND CMAKE_C_FLAGS " -Wl,--reproduce=repro.tar")
    string(APPEND CMAKE_CXX_FLAGS " -Wl,--reproduce=repro.tar")

Not sure if this is doing correctly.

[sbc100]

You would need to add the flag at link time, not compile time.

[fs-eire]

I have generated the repro.tar (121MB). Unfortunately, the llvm-reduce does not come with EMSDK. Is it required to use llvm-reduce of the same version to wasm-ld, or any llvm-reduce is fine (eg. https://github.com/llvm/llvm-project/releases/) to process this tar?

I also tried "-flto=thin", it seems working with the following warnings:

[1200/1203] Linking CXX executable ort-wasm-simd-threaded.mjs
cache:INFO: generating system asset: symbol_lists/30743cd420782ee9e353319a0933a76dd1e62953.json... (this will be cached in "C:\code\onnxruntime\cmake\external\emsdk\upstream\emscripten\cache\symbol_lists\30743cd420782ee9e353319a0933a76dd1e62953.json" for subsequent builds)
cache:INFO:  - ok
wasm-ld: warning: function signature mismatch: _ZNSt3__215basic_streambufIcNS_11char_traitsIcEEE7seekposENS_4fposI11__mbstate_tEEj
>>> defined as () -> void in C:\code\onnxruntime\cmake\external\emsdk\upstream\emscripten\cache\sysroot\lib\wasm32-emscripten\thinlto\libc++-mt.a(ios.instantiations.o)
>>> defined as (i32, i32, i32, i32) -> void in lto.tmp

wasm-ld: warning: function signature mismatch: _ZNK11onnxruntime3QDQ12BaseSelector6SelectERKNS_11GraphViewerERKNS_4NodeE
>>> defined as () -> void in libonnxruntime_optimizer.a(qdq_selectors.cc.o)
>>> defined as (i32, i32, i32, i32) -> void in lto.tmp

wasm-ld: warning: function signature mismatch: _ZNSt3__215basic_streambufIcNS_11char_traitsIcEEE7seekoffExNS_8ios_base7seekdirEj
>>> defined as () -> void in C:\code\onnxruntime\cmake\external\emsdk\upstream\emscripten\cache\sysroot\lib\wasm32-emscripten\thinlto\libc++-mt.a(ios.instantiations.o)
>>> defined as (i32, i32, i64, i32, i32) -> void in lto.tmp

wasm-ld: warning: function signature mismatch: _ZNK11onnxruntime18IExecutionProvider13GetCapabilityERKNS_11GraphViewerERKNS0_13IKernelLookupE
>>> defined as () -> void in libonnxruntime_framework.a(execution_provider.cc.o)
>>> defined as (i32, i32, i32, i32) -> void in lto.tmp

EDIT:

While the thinLTO build is completed successfully, it fails at runtime:

worker sent an error! undefined:undefined: getWasmTableEntry(...) is not a function

node:internal/event_target:1090
  process.nextTick(() => { throw err; });
                           ^
TypeError [Error]: getWasmTableEntry(...) is not a function
    at invoke_viiiiiii (C:\code\onnxruntime\build_thin_lto\Release\onnxruntime_test_all.js:9302:31)
    at wasm://wasm/08d95992:wasm-function[28855]:0x16dffd0
    at invoke_vii (C:\code\onnxruntime\build_thin_lto\Release\onnxruntime_test_all.js:9162:31)
    at wasm://wasm/08d95992:wasm-function[7984]:0x84b60d
    at invoke_viiiii (C:\code\onnxruntime\build_thin_lto\Release\onnxruntime_test_all.js:9252:31)
    at wasm://wasm/08d95992:wasm-function[8087]:0x85cb60
    at invoke_vi (C:\code\onnxruntime\build_thin_lto\Release\onnxruntime_test_all.js:9202:31)
    at wasm://wasm/08d95992:wasm-function[1646]:0xf947f
    at wasm://wasm/08d95992:wasm-function[16518]:0xd33c16
    at invoke_ii (C:\code\onnxruntime\build_thin_lto\Release\onnxruntime_test_all.js:9142:38)

Node.js v22.11.0

While No LTO build and test both completed successfully.

[sbc100]

You can probably use llvm-reduce from the llvm 19 release.

The steps to use llvm-reduce start with getting the combined bitcode file that is causing the crash. To do that you would link with -Wl,--save-termp which cases wasm-ld to write the bitcode files to disc. Once you have have the combined bitcode file you should be able run it through llc to reproduce the crash. Once you have a failing llc command you can use that in combination with llvm-reduce to find the a much smaller bitcode file with the same crash. As you can see its fairly involved ..