CUDA GPU InvalidIRError when using horizontal gradient on a LevelGrid #2358
Description
Describe the bug
When applying the horizontal gradient operator to a Field that lives on a LevelGrid (i.e., a horizontal slice of a 3D ExtrudedFiniteDifferenceSpace), the CUDA kernel compilation fails. After some digging, I believe issue occurs because the quadrature_style(levelgrid::LevelGrid) function performs a chained field access (levelgrid.full_grid.horizontal_grid) which results in dynamic dispatch.
To Reproduce
Ideally a minimal reproducible example:
using ClimaCore
using CUDA
using ClimaComms
ENV["CLIMACOMMS_DEVICE"] = "CUDA"
context = ClimaComms.SingletonCommsContext(ClimaComms.CUDADevice())
h_domain = ClimaCore.Domains.RectangleDomain(
ClimaCore.Domains.IntervalDomain(
ClimaCore.Geometry.XPoint(0.0),
ClimaCore.Geometry.XPoint(1.0);
periodic = true,
),
ClimaCore.Domains.IntervalDomain(
ClimaCore.Geometry.YPoint(0.0),
ClimaCore.Geometry.YPoint(1.0);
periodic = true,
),
)
h_mesh = ClimaCore.Meshes.RectilinearMesh(h_domain, 4, 4)
h_topology = ClimaCore.Topologies.Topology2D(context, h_mesh)
quad = ClimaCore.Quadratures.GLL{4}()
h_space = ClimaCore.Spaces.SpectralElementSpace2D(h_topology, quad)
z_domain = ClimaCore.Domains.IntervalDomain(
ClimaCore.Geometry.ZPoint(0.0),
ClimaCore.Geometry.ZPoint(1.0);
boundary_names = (:bottom, :top),
)
z_mesh = ClimaCore.Meshes.IntervalMesh(z_domain; nelems = 10)
v_space = ClimaCore.Spaces.CenterFiniteDifferenceSpace(z_mesh)
space = ClimaCore.Spaces.ExtrudedFiniteDifferenceSpace(h_space, v_space)
coords = ClimaCore.Fields.coordinate_field(space)
χ = @. sin(coords.x) * sin(coords.y)
χ_level1 = ClimaCore.Fields.level(χ, 1)
const gradₕ = ClimaCore.Operators.Gradient()
∇χ = gradₕ.(χ_level1)Or even better, add it as a unit test, and open pull request.
Project
If not using the `examples` project: ``` paste your Project.toml here. ``` ``` paste your Manifest.toml here. ```
System details
Any relevant system information:
- Julia version
- operating system
- modules loaded on cluster (
module list)
Related issues / PRs
Please add any relevant links.