Julia 1.12 LLVM error: Instruction Combining did not reach a fixpoint after 1 iterations

[pxl-th]

MWE:

using CUDA

function cos_kernel!(y, x)
    idx = (blockIdx().x - 1) * blockDim().x + threadIdx().x
    if idx <= length(x)
        y[idx] = cos(x[idx])
    end
    return
end

x = CUDA.randn(Float32, 256)
y = similar(x)
@cuda threads=256 blocks=1 cos_kernel!(y, x)

Error:

ERROR: LoadError: LLVM error: Instruction Combining did not reach a fixpoint after 1 iterations
Stacktrace:
  [1] handle_error(reason::Cstring)                                                                                                                                 @ LLVM ~/.julia-1.12.0/packages/LLVM/b3kFs/src/core/context.jl:194
  [2] LLVMRunJuliaPasses
    @ ~/.julia-1.12.0/packages/LLVM/b3kFs/lib/18/libLLVM_extra.jl:291 [inlined]
  [3] macro expansion
    @ ~/.julia-1.12.0/packages/LLVM/b3kFs/src/executionengine/utils.jl:25 [inlined]
  [4] run!(pb::LLVM.NewPMPassBuilder, target::LLVM.Module, tm::LLVM.TargetMachine)
    @ LLVM ~/.julia-1.12.0/packages/LLVM/b3kFs/src/newpm.jl:291
  [5] macro expansion
    @ ~/.julia-1.12.0/packages/GPUCompiler/2MI6e/src/optim.jl:20 [inlined]
  [6] macro expansion
    @ ~/.julia-1.12.0/packages/LLVM/b3kFs/src/base.jl:97 [inlined]
  [7] optimize!(job::GPUCompiler.CompilerJob, mod::LLVM.Module; opt_level::Int64)
    @ GPUCompiler ~/.julia-1.12.0/packages/GPUCompiler/2MI6e/src/optim.jl:9
  [8] optimize!
    @ ~/.julia-1.12.0/packages/GPUCompiler/2MI6e/src/optim.jl:3 [inlined]
  [9] emit_llvm(job::GPUCompiler.CompilerJob; kwargs::@Kwargs{})
    @ GPUCompiler ~/.julia-1.12.0/packages/GPUCompiler/2MI6e/src/driver.jl:284
 [10] emit_llvm
    @ ~/.julia-1.12.0/packages/GPUCompiler/2MI6e/src/driver.jl:154 [inlined]
 [11] #compile_unhooked#112
    @ ~/.julia-1.12.0/packages/GPUCompiler/2MI6e/src/driver.jl:95 [inlined]
 [12] compile_unhooked
    @ ~/.julia-1.12.0/packages/GPUCompiler/2MI6e/src/driver.jl:80 [inlined]
 [13] compile(target::Symbol, job::GPUCompiler.CompilerJob; kwargs::@Kwargs{})
    @ GPUCompiler ~/.julia-1.12.0/packages/GPUCompiler/2MI6e/src/driver.jl:67
 [14] compile
    @ ~/.julia-1.12.0/packages/GPUCompiler/2MI6e/src/driver.jl:55 [inlined]
 [15] #compile##0
    @ ~/.julia-1.12.0/packages/CUDA/RQqFT/src/compiler/compilation.jl:255 [inlined]
 [16] JuliaContext(f::CUDA.var"#compile##0#compile##1"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}}; kwargs::@Kwargs{})
    @ GPUCompiler ~/.julia-1.12.0/packages/GPUCompiler/2MI6e/src/driver.jl:34
 [17] JuliaContext(f::Function)
    @ GPUCompiler ~/.julia-1.12.0/packages/GPUCompiler/2MI6e/src/driver.jl:25
 [18] compile(job::GPUCompiler.CompilerJob)
    @ CUDA ~/.julia-1.12.0/packages/CUDA/RQqFT/src/compiler/compilation.jl:254
 [19] actual_compilation(cache::Dict{Any, CuFunction}, src::Core.MethodInstance, world::UInt64, cfg::GPUCompiler.CompilerConfig{GPUCompiler.PTXCompilerTarget, C
UDA.CUDACompilerParams}, compiler::typeof(CUDA.compile), linker::typeof(CUDA.link))
    @ GPUCompiler ~/.julia-1.12.0/packages/GPUCompiler/2MI6e/src/execution.jl:245
 [20] cached_compilation(cache::Dict{Any, CuFunction}, src::Core.MethodInstance, cfg::GPUCompiler.CompilerConfig{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompile
rParams}, compiler::Function, linker::Function)
    @ GPUCompiler ~/.julia-1.12.0/packages/GPUCompiler/2MI6e/src/execution.jl:159
 [21] macro expansion
    @ ~/.julia-1.12.0/packages/CUDA/RQqFT/src/compiler/execution.jl:373 [inlined]
 [22] macro expansion
    @ ./lock.jl:376 [inlined]
 [23] cufunction(f::typeof(cos_kernel!), tt::Type{Tuple{CuDeviceVector{Float32, 1}, CuDeviceVector{Float32, 1}}}; kwargs::@Kwargs{})
    @ CUDA ~/.julia-1.12.0/packages/CUDA/RQqFT/src/compiler/execution.jl:368
 [24] cufunction(f::typeof(cos_kernel!), tt::Type{Tuple{CuDeviceVector{Float32, 1}, CuDeviceVector{Float32, 1}}})
    @ CUDA ~/.julia-1.12.0/packages/CUDA/RQqFT/src/compiler/execution.jl:365

[vchuravy]

Can you run with JULIA_LLVM_ARGS="-print-after-all -print-module-scope" and then upload the module it's failing on? Godbolt is a good verification tool for that as well.

[pxl-th]

How long should it take? I left it running for 1+ hour and it didn't finish printing out stuff.

[maleadt]

You can restrict it to a single function with some -filter arg (check opt --help-hidden) and hack GPUCompiler.jl to make sure that name is used throughout compilation (by doing an LLVM.name! somewhere at the start of the driver). Ideally @cuda name=... should do that, but IIRC that isn't applied early enough.

[miniskar]

I have logged similar issue in Link here.
Surprisingly, it is working good for AMDGPU, but not for CUDA GPU.

BTW, it is working good with Julia release version 1.11.4 for both CUDA and AMDGPU.

[Zentrik]

I think we should just disable this assertion per llvm/llvm-project#92648.

How long should it take? I left it running for 1+ hour and it didn't finish printing out stuff.

Instead of printing after all, you can print only before instcombine. Also if you pipe the output to a file, I imagine it will be much quicker than printing it.

[maleadt]

I think we should just disable this assertion per llvm/llvm-project#92648.

Only added in LLVM 18: llvm/llvm-project@5fbee1c (1.10 uses LLVM 15, 1.11 uses LLVM 16).

max-iterations was added in LLVM 17, so we can't use that either. There seems to be a command-line option before that, but I'm not sure we can safely bump that limit globally?

[Zentrik]

I imagine the assertion was only added in llvm 18 in llvm/llvm-project@4189584, so should be fine.

[vchuravy]

Can someone give JuliaLLVM/LLVM.jl#504 a go?

[maleadt]

I imagine the assertion was only added in llvm 18 in llvm/llvm-project@4189584, so should be fine.

Oh, of course. I'm exposing the options, see JuliaLLVM/LLVM.jl#507, so we'll be able to fix that afterwards.

[maleadt]

Will be fixed with LLVM.jl 9.3.1