"Unsupported dynamic function invocation" problem when I tried to run examples on GPU

[zwei961120]

When I tried to run code "langmuir_turbulence.jl" in example files provided by https://github.com/CliMA/Oceananigans.jl/tree/main/examples on GPU, I met the problem showing that:

ERROR: LoadError: InvalidIRError: compiling kernel #gpu__fill_bottom_and_top_halo!(KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.StaticSize{(32, 32)}, KernelAbstractions.NDIteration.DynamicCheck, Nothing, Nothing, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.StaticSize{(2, 2)}, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, Nothing, Nothing}}, NTuple{4, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}}, Tuple{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Gradient, Float64}}, Tuple{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Float64}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Float64}}, Tuple{Tuple{Face, Center, Center}, Tuple{Center, Face, Center}, Tuple{Center, Center, Face}, Tuple{Center, Center, Center}}, RectilinearGrid{Float64, Periodic, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Nothing}, NamedTuple{(:time, :iteration, :stage), Tuple{Float64, Int64, Int64}}, NamedTuple{(:u, :v, :w, :b), NTuple{4, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}}}) resulted in invalid LLVM IR Reason: unsupported dynamic function invocation (call to _fill_bottom_halo!(i, j, grid, c, bc::Union{BoundaryCondition{<:Oceananigans.BoundaryConditions.Value}, BoundaryCondition{<:Oceananigans.BoundaryConditions.Gradient}}, loc, args...) in Oceananigans.BoundaryConditions at /.julia/packages/Oceananigans/xjmyS/src/BoundaryConditions/fill_halo_regions_value_gradient.jl:73)

I just modified grid = RectilinearGrid(size=(32, 32, 32), extent=(128, 128, 64)) to grid = RectilinearGrid(size=(32, 32, 32), extent=(128, 128, 64)), so as making the code run on GPU. My Oceananigans version is the latest "v0.77.3" , and my Julia version is "v1.8.0". I have browsed the discussions which show the same problems with me (#2527), but the solution does not work and another problem arises if I downgrade my Oceananigans version.

[tomchor]

What happens when you run on Julia 1.6? Version 1.8 isn't officially supported yet.

[zwei961120]

Yeah, when I tried it on Julia 1.6, it runs smoothly, thanks!

[glwagner]

This might be related to #2709. cc @simone-silvestri

[simone-silvestri]

Yeah, it really seems we have problems with Value and Gradient boundary conditions. I ll try to reproduce the error. It might be because of the tupled halo filling