WAMR outputs `LLVM ERROR`

[teyahb8]

Subject of the issue

I generated a test case which I executed with different runtimes. Other runtimes like wasmtime, wasmer, wasmedge, and wamr's AOT mode outputs out of bounds exception. But, wamr's JIT mode outputs LLVM ERROR followed by a crash.

Test case

(module
  (type (;0;) (func (result v128 v128 v128 v128 i32 v128 v128 v128 v128 v128)))
  (memory (;0;) 0 1)
  (export "main" (func 0))
  (func (;0;) (type 0) (result v128 v128 v128 v128 i32 v128 v128 v128 v128 v128)
    (local v128)
    v128.const i32x4 0x9c184c7b 0x0b4b01ec 0x0f34f4cb 0x66297649
    local.set 0
    local.get 0
    i32x4.extract_lane 0
    v128.load8x8_s offset=1123 align=1
    local.tee 0
    local.get 0
    v128.const i32x4 0x84969263 0x1cd3f409 0xb2cf77d8 0x9154ed85
    i16x8.ne
    i32x4.max_s
    local.tee 0
    v128.const i32x4 0x00000000 0x00000000 0x00000000 0x00000000
    f32x4.add
    local.get 0
    i32.const 6224
    i8x16.shr_u
    local.get 0
    i64x2.lt_s
    local.tee 0
    v128.const i32x4 0x00000000 0x00000000 0x00000000 0x00000000
    i32x4.ge_s
    f32x4.sub
    local.get 0
    i32x4.trunc_sat_f32x4_s
    local.tee 0
    i32.const 170
    i64x2.shr_s
    local.tee 0
    v128.const i32x4 0x7598926e 0x5e66b8ab 0xc2efa8cd 0xdfc8f454
    v128.andnot
    v128.xor
    local.get 0
    v128.const i32x4 0xffffffff 0xffffffff 0xffffffff 0xffffffff
    i16x8.max_s
    local.get 0
    i32x4.lt_u
    local.tee 0
    v128.const i32x4 0xffffffff 0xffffffff 0xffffffff 0xffffffff
    i16x8.sub_sat_s
    i16x8.min_u
    local.get 0
    local.get 0
    v128.const i32x4 0xcbdb6b3b 0xc5c82046 0xb9bb9fb5 0x7276884c
    i8x16.narrow_i16x8_s
    local.tee 0
    i32x4.extmul_low_i16x8_u
    local.tee 0
    local.get 0
    f64x2.add
    f32x4.sub
    local.get 0
    v128.const i32x4 0xffffffff 0xffffffff 0xffffffff 0xffffffff
    i16x8.q15mulr_sat_s
    local.tee 0
    local.get 0
    i16x8.mul
    local.tee 0
    local.get 0
    i16x8.ge_s
    i32x4.ge_s
    local.get 0
    local.get 0
    i32x4.ge_s
    local.tee 0
    local.get 0
    f64x2.lt
    i32x4.lt_u
    local.tee 0
    v128.const i32x4 0x00000000 0x00000000 0x00000000 0x00000000
    i8x16.shuffle 13 2 5 13 3 13 11 0 6 14 15 5 2 5 10 7
    local.tee 0
    local.get 0
    i32x4.add
    local.get 0
    i32.const 806
    i8x16.shr_u
    local.tee 0
    local.get 0
    i8x16.swizzle
    i16x8.max_s
    i32.const 739
    i32x4.shl
    v128.const i32x4 0x92101497 0x8db5c122 0xb4b99220 0xa88c5e77
    i64x2.extmul_high_i32x4_s
    local.get 0
    i32.const 7122
    i32x4.shr_s
    v128.const i32x4 0xc31e975c 0xdcb2f154 0xbdd79376 0xba1b2ba7
    i8x16.add_sat_u
    f64x2.pmax
    local.get 0
    i32.const 2605
    i64x2.shl
    v128.const i32x4 0x00000000 0x00000000 0x00000000 0x00000000
    v128.const i32x4 0x63d7378b 0xe7fbd528 0x7218aa4a 0x4a1c03cc
    v128.const i32x4 0x00000000 0x00000000 0x00000000 0x00000000
    v128.const i32x4 0xc174fc1c 0x7cbf7d5d 0xac7bd66d 0xf3ce226f
    local.get 0
    v128.const i32x4 0x4e50113c 0xc8feb817 0x85db4d89 0x7871ebbe
    drop
    drop
    i32.const 0
    select
    v128.const i32x4 0xffffffff 0xffffffff 0xffffffff 0xffffffff
    i32.const 5865
    i8x16.shr_s
    v128.const i32x4 0x00000000 0x00000000 0x00000000 0x00000000
    local.get 0
    v128.const i32x4 0x00000000 0x00000000 0x00000000 0x00000000
    i32x4.dot_i16x8_s
    i32.const 9329
    i32x4.shr_u
    v128.const i32x4 0x53432b45 0xf4af543b 0x8b6c1906 0x9dbea614
    v128.const i32x4 0x00000000 0x00000000 0x00000000 0x00000000
    local.get 0
    local.get 0
    local.get 0
    local.get 0
    i32.const 1973104239
    local.get 0
    local.get 0
    local.get 0
    local.get 0
    local.get 0
    return
  )
)

Your environment

• Host OS (Ubuntu 22.04)
• WAMR version (2.3.1), platform (Linux), cpu architecture (x86_64), running mode (JIT), etc.

Steps to reproduce

1. Convert wat to wasm.
2. Run: iwasm --heap-size=0 --llvm-jit -f main shrunken_test2061838.wasm

Expected behavior

Exception: out of bounds memory access

Actual behavior

LLVM ERROR: Cannot select: 0x7ffddc0aee70: v4i32 = BUILD_VECTOR Constant:i32<65537>, Constant:i32<65537>, Constant:i32<65537>, Constant:i32<65537>
  0x7ffddc0ac6a0: i32 = Constant<65537>
  0x7ffddc0ac6a0: i32 = Constant<65537>
  0x7ffddc0ac6a0: i32 = Constant<65537>
  0x7ffddc0ac6a0: i32 = Constant<65537>
In function: aot_func#0
Aborted

Extra Info

Can you please confirm? Thanks in advance.

[TianlongLiang]

It seems that in the latest commit, this issue is resolved, probably has something to do with this fix: #4236. Can you try it in the latest main branch?

[teyahb8]

I checked with the latest commit (a0de8c7). I am still getting the LLVM ERROR leading to the crash, @TianlongLiang.

[TianlongLiang]

I can't reproduce this issue, even with the release binary 2.3.1. Do you still have this issue with 2.4.1 binary by the way?

[teyahb8]

Yes, I still got this issue with 2.4.1 binary. I can provide some more context. I have attached a screenshot with version details and the manifestation of the bug.

Here is the stack trace from the crash:

* thread #3, name = 'llvm-worker-1', stop reason = signal SIGABRT
  * frame #0: 0x00007ffff4a969fc libc.so.6`__GI___pthread_kill at pthread_kill.c:44:76
    frame #1: 0x00007ffff4a969b0 libc.so.6`__GI___pthread_kill [inlined] __pthread_kill_internal(signo=6, threadid=140737284924992) at pthread_kill.c:78:10
    frame #2: 0x00007ffff4a969b0 libc.so.6`__GI___pthread_kill(threadid=140737284924992, signo=6) at pthread_kill.c:89:10
    frame #3: 0x00007ffff4a42476 libc.so.6`__GI_raise(sig=6) at raise.c:26:13
    frame #4: 0x00007ffff4a287f3 libc.so.6`__GI_abort at abort.c:79:7
    frame #5: 0x00007ffff56d0d05 libiwasm.so.2`llvm::report_fatal_error(llvm::Twine const&, bool) + 437
    frame #6: 0x00007ffff6a09aa0 libiwasm.so.2`llvm::SelectionDAGISel::CannotYetSelect(llvm::SDNode*) + 864
    frame #7: 0x00007ffff6a08fa8 libiwasm.so.2`llvm::SelectionDAGISel::SelectCodeCommon(llvm::SDNode*, unsigned char const*, unsigned int) + 16168
    frame #8: 0x00007ffff6407083 libiwasm.so.2`(anonymous namespace)::X86DAGToDAGISel::Select(llvm::SDNode*) + 26323
    frame #9: 0x00007ffff6a008bf libiwasm.so.2`llvm::SelectionDAGISel::DoInstructionSelection() + 431
    frame #10: 0x00007ffff69fffa1 libiwasm.so.2`llvm::SelectionDAGISel::CodeGenAndEmitDAG() + 1441
    frame #11: 0x00007ffff69ff0c2 libiwasm.so.2`llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&) + 4354
    frame #12: 0x00007ffff69fd27e libiwasm.so.2`llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&) + 1726
    frame #13: 0x00007ffff63fce8f libiwasm.so.2`(anonymous namespace)::X86DAGToDAGISel::runOnMachineFunction(llvm::MachineFunction&) + 79
    frame #14: 0x00007ffff58fc234 libiwasm.so.2`llvm::MachineFunctionPass::runOnFunction(llvm::Function&) + 852
    frame #15: 0x00007ffff57ccf2f libiwasm.so.2`llvm::FPPassManager::runOnFunction(llvm::Function&) + 687
    frame #16: 0x00007ffff57d2823 libiwasm.so.2`llvm::FPPassManager::runOnModule(llvm::Module&) + 51
    frame #17: 0x00007ffff57cd624 libiwasm.so.2`llvm::legacy::PassManagerImpl::run(llvm::Module&) + 1012
    frame #18: 0x00007ffff6285bdc libiwasm.so.2`llvm::orc::SimpleCompiler::operator()(llvm::Module&) + 380
    frame #19: 0x00007ffff6286091 libiwasm.so.2`llvm::orc::ConcurrentIRCompiler::operator()(llvm::Module&) + 129
    frame #20: 0x00007ffff62c9a3b libiwasm.so.2`decltype(auto) llvm::orc::ThreadSafeModule::withModuleDo<llvm::orc::IRCompileLayer::IRCompiler&>(llvm::orc::IRCompileLayer::IRCompiler&) + 91
    frame #21: 0x00007ffff62c9825 libiwasm.so.2`llvm::orc::IRCompileLayer::emit(std::unique_ptr<llvm::orc::MaterializationResponsibility, std::default_delete<llvm::orc::MaterializationResponsibility> >, llvm::orc::ThreadSafeModule) + 53
    frame #22: 0x00007ffff632c51b libiwasm.so.2`llvm::orc::IRTransformLayer::emit(std::unique_ptr<llvm::orc::MaterializationResponsibility, std::default_delete<llvm::orc::MaterializationResponsibility> >, llvm::orc::ThreadSafeModule) + 203
    frame #23: 0x00007ffff632c51b libiwasm.so.2`llvm::orc::IRTransformLayer::emit(std::unique_ptr<llvm::orc::MaterializationResponsibility, std::default_delete<llvm::orc::MaterializationResponsibility> >, llvm::orc::ThreadSafeModule) + 203
    frame #24: 0x00007ffff6283587 libiwasm.so.2`llvm::orc::CompileOnDemandLayer::emitPartition(std::unique_ptr<llvm::orc::MaterializationResponsibility, std::default_delete<llvm::orc::MaterializationResponsibility> >, llvm::orc::ThreadSafeModule, std::map<llvm::orc::SymbolStringPtr, llvm::GlobalValue*, std::less<llvm::orc::SymbolStringPtr>, std::allocator<std::pair<llvm::orc::SymbolStringPtr const, llvm::GlobalValue*> > >) + 6311
    frame #25: 0x00007ffff6284f80 libiwasm.so.2`llvm::orc::PartitioningIRMaterializationUnit::materialize(std::unique_ptr<llvm::orc::MaterializationResponsibility, std::default_delete<llvm::orc::MaterializationResponsibility> >) + 224
    frame #26: 0x00007ffff6297455 libiwasm.so.2`llvm::orc::MaterializationTask::run() + 53
    frame #27: 0x00007ffff62da99d libiwasm.so.2`std::_Function_handler<void (), llvm::orc::LLJIT::LLJIT(llvm::orc::LLJITBuilderState&, llvm::Error&)::$_0::operator()(std::unique_ptr<llvm::orc::Task, std::default_delete<llvm::orc::Task> >) const::'lambda'()>::_M_invoke(std::_Any_data const&) + 13
    frame #28: 0x00007ffff62da4c4 libiwasm.so.2`std::_Function_handler<void (), llvm::ThreadPool::createTaskAndFuture(std::function<void ()>)::'lambda'()>::_M_invoke(std::_Any_data const&) + 36
    frame #29: 0x00007ffff73ee9fe libiwasm.so.2`llvm::ThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) + 606
    frame #30: 0x00007ffff73ef507 libiwasm.so.2`void* llvm::thread::ThreadProxy<std::tuple<llvm::ThreadPool::grow(int)::$_0> >(void*) + 151
    frame #31: 0x00007ffff4a94ac3 libc.so.6`start_thread(arg=<unavailable>) at pthread_create.c:442:8
    frame #32: 0x00007ffff4b26850 libc.so.6`__clone3 at clone3.S:81

I built wamr runtime from source. When building iwasm, I used the following compilation options:

cmake -DCMAKE_BUILD_TYPE=Debug -DWAMR_BUILD_JIT=1 -DWAMR_BUILD_LIBC_WASI=0 -DWAMR_BUILD_LIB_PTHREAD=1 -DBUILD_SHARED_LIBS=1 -DLLVM_DIR=/usr/lib/llvm-18/lib/cmake/llvm ..

I have tested on Ubuntu 22.04 OS and x86_64 CPU. Thanks.

[TianlongLiang]

I use Ubuntu 20.04, I tried Docker Ubuntu 22.04, though, still no luck. Another difference between us could be the CPU features, I printed it with -v=5:

[02:07:56:042 - 7F02E3F417C0]: LLVM ORCJIT detected CPU "skylake", with features "+prfchw,-cldemote,+avx,+aes,+sahf,+pclmul,-xop,+crc32,+xsaves,-avx512fp16,-usermsr,-sm4,+sse4.1,-avx512ifma,+xsave,-avx512pf,+sse4.2,-tsxldtrk,-ptwrite,-widekl,-sm3,+invpcid,+64bit,+xsavec,-avx10.1-512,-avx512vpopcntdq,+cmov,-avx512vp2intersect,-avx512cd,+movbe,-avxvnniint8,-avx512er,-amx-int8,-kl,-avx10.1-256,-sha512,-avxvnni,-rtm,+adx,+avx2,-hreset,-movdiri,-serialize,-vpclmulqdq,-avx512vl,-uintr,+clflushopt,-raoint,-cmpccxadd,+bmi,-amx-tile,+sse,-gfni,-avxvnniint16,-amx-fp16,+xsaveopt,+rdrnd,-avx512f,-amx-bf16,-avx512bf16,-avx512vnni,+cx8,-avx512bw,+sse3,-pku,+fsgsbase,-clzero,-mwaitx,-lwp,+lzcnt,-sha,-movdir64b,-wbnoinvd,-enqcmd,-prefetchwt1,-avxneconvert,-tbm,-pconfig,-amx-complex,+ssse3,+cx16,+bmi2,+fma,+popcnt,-avxifma,+f16c,-avx512bitalg,-rdpru,-clwb,+mmx,+sse2,+rdseed,-avx512vbmi2,-prefetchi,-rdpid,-fma4,-avx512vbmi,-shstk,-vaes,-waitpkg,+sgx,+fxsr,-avx512dq,-sse4a"

I suspect this issue can result from some SIMD instruction support or something

[teyahb8]

This is my output with -v=5:

[16:47:36:418 - 7FFFF7F347C0]: Load type section success.

[16:47:36:418 - 7FFFF7F347C0]: Load function section success.

[16:47:36:418 - 7FFFF7F347C0]: Load memory section success.

[16:47:36:418 - 7FFFF7F347C0]: Load export section success.

[16:47:36:418 - 7FFFF7F347C0]: Load code segment section success.

[16:47:36:419 - 7FFFF7F347C0]: LLVM ORCJIT detected CPU "znver4", with features "+prfchw,-cldemote,+avx,+aes,+sahf,+pclmul,-xop,+crc32,+xsaves,-avx512fp16,-usermsr,-sm4,+sse4.1,+avx512ifma,+xsave,+sse4.2,-avx512pf,-tsxldtrk,-ptwrite,-widekl,-sm3,+invpcid,+64bit,+xsavec,-avx10.1-512,+avx512vpopcntdq,+cmov,-avx512vp2intersect,+avx512cd,+movbe,-avxvnniint8,-avx512er,-amx-int8,-kl,-avx10.1-256,+evex512,-avxvnni,-rtm,+adx,+avx2,-hreset,-movdiri,-serialize,-sha512,+vpclmulqdq,+avx512vl,-uintr,+clflushopt,-raoint,-cmpccxadd,+bmi,-amx-tile,+sse,+gfni,-avxvnniint16,-amx-fp16,+xsaveopt,+rdrnd,+avx512f,-amx-bf16,+avx512bf16,+avx512vnni,+cx8,+avx512bw,+sse3,+pku,+fsgsbase,+clzero,+mwaitx,-lwp,+lzcnt,+sha,-movdir64b,+wbnoinvd,-enqcmd,-prefetchwt1,-avxneconvert,-tbm,-pconfig,-amx-complex,+ssse3,+cx16,+bmi2,+fma,+popcnt,-avxifma,+f16c,+avx512bitalg,+rdpru,+clwb,+mmx,+sse2,+rdseed,+avx512vbmi2,-prefetchi,+rdpid,-fma4,+avx512vbmi,+shstk,+vaes,-waitpkg,-sgx,+fxsr,+avx512dq,+sse4a"

Begin to compile WASM bytecode to LLVM IR        time of last stage: 0 ms, total time: 0 ms
Begin to run llvm optimization passes            time of last stage: 0 ms, total time: 0 ms
Finish llvm optimization passes                  time of last stage: 2 ms, total time: 2 ms
Begin to lookup llvm jit functions               time of last stage: 0 ms, total time: 2 ms
End lookup llvm jit functions                    time of last stage: 2 ms, total time: 4 ms
Begin to compile jit functions                   time of last stage: 0 ms, total time: 4 ms
End compile jit functions                        time of last stage: 0 ms, total time: 4 ms
[16:47:36:424 - 7FFFF7F347C0]: Load module success.

[16:47:36:424 - 7FFFF7F347C0]: Memory instantiate:
[16:47:36:424 - 7FFFF7F347C0]:   page bytes: 0, init pages: 0, max pages: 0
[16:47:36:424 - 7FFFF7F347C0]:   heap offset: 0, heap size: 0

[16:47:36:424 - 7FFFF7F347C0]: Memory instantiate success.
[16:47:36:424 - 7FFFF7F347C0]: wasm/runtimes/wasm-micro-runtime/core/iwasm/common/wasm_application.c, line 376, call a function "main" with 0 arguments
[16:47:36:424 - 7FFFF7F347C0]: No aux stack info for this module, can't create thread
[16:47:36:424 - 7FFFF4C49640]: wasm/runtimes/wasm-micro-runtime/core/iwasm/interpreter/wasm_loader.c, line 5961, compile llvm jit func aot_func#0_wrapper
[16:47:36:424 - 7FFFF45FF640]: wasm/runtimes/wasm-micro-runtime/core/iwasm/compilation/aot_orc_extra.cpp, line 194, requested func aot_func#0
[16:47:36:424 - 7FFFF45FF640]: wasm/runtimes/wasm-micro-runtime/core/iwasm/compilation/aot_orc_extra.cpp, line 232, compile func aot_func#0
[16:47:36:424 - 7FFFF3DFE640]: wasm/runtimes/wasm-micro-runtime/core/iwasm/compilation/aot_orc_extra.cpp, line 194, requested func aot_func#0_wrapper
[16:47:36:424 - 7FFFF3DFE640]: wasm/runtimes/wasm-micro-runtime/core/iwasm/compilation/aot_orc_extra.cpp, line 232, compile func aot_func#0
LLVM ERROR: Cannot select: 0x7fffec0c19b0: v4i32 = BUILD_VECTOR Constant:i32<65537>, Constant:i32<65537>, Constant:i32<65537>, Constant:i32<65537>
  0x7fffec0bf1e0: i32 = Constant<65537>
  0x7fffec0bf1e0: i32 = Constant<65537>
  0x7fffec0bf1e0: i32 = Constant<65537>
  0x7fffec0bf1e0: i32 = Constant<65537>
In function: aot_func#0
Aborted

[TianlongLiang]

I suspect this has something to do with AVX512 and EVEX512 flags in your CPU feature list; it will generate some instructions (such as VPBROADCASTD or something), and it can only use 8/16-bit-imm, but the immediate value 65 537 (0x0001_0001) is too large. On Skylake, LLVM jit engine falls back to a load from the constant pool(or don't go down this road at all). But somehow, with this feature flag(or along with others maybe), on your CPU, the LLVM jit engine will want to keep everything in registers and hence this error.

[teyahb8]

I see. Thanks for the nice explanation.