Issue with discrete boundary condition accessing model_fields on distributed GPU setup #5461

phyo-wai-thaw · 2026-03-31T08:03:52Z

phyo-wai-thaw
Mar 31, 2026

Hello!

I’m running into an issue with a discrete boundary condition that accesses model_fields, which works correctly on a single GPU, but fails to compile when using distributed GPUs.

Any suggestion would be greatly appreciated.

Thanks a lot!

Environment (NeSI HPC)

Julia 1.11.3
Oceananigans v0.96.35
CUDA.jl v5.9.7
MPI.jl v0.20.23
2× NVIDIA L4 GPUs 
OpenMPI 4.1.5 with CUDA 12.2.2

SLURM script submitted on NeSI

#!/bin/bash -e
#SBATCH --account=uc04420
#SBATCH --job-name=2gpus_spinup
#SBATCH --time=00:20:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=2
#SBATCH --gpus-per-node=L4:2
#SBATCH --mem=5GB

module purge >/dev/null 2>&1
module load Julia/1.11.3-GCC-12.3.0-VTune
module load OpenMPI/4.1.5-NVHPC-23.9-GCC-12.3.0-CUDA-12.2.2

export JULIA_NUM_THREADS=1
export JULIA_CUDA_MEMORY_POOL=none
export UCX_ERROR_SIGNALS="SIGILL,SIGBUS,SIGFPE"
export UCX_WARN_UNUSED_ENV_VARS=n
export UCX_TLS=dc,self,sm,cuda_copy,cuda_ipc
export OMPI_MCA_pml=ucx

nvidia-smi
echo "CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES}"

srun julia --project melt_test.jl

Minimal Working Example

using MPI
using CUDA
using Oceananigans
using Oceananigans.DistributedComputations

MPI.Init()

CUDA.device!(MPI.Comm_rank(MPI.COMM_WORLD))
arch = Distributed(GPU())

Nranks = MPI.Comm_size(arch.communicator)
rank   = MPI.Comm_rank(arch.communicator)

@info "Running on rank $rank of $Nranks"


grid = RectilinearGrid(arch;
    size = (16, 16, 16),
    x = (0, 1), y = (0, 1), z = (0, 1),
    topology = (Periodic, Periodic, Bounded)
)
@show grid

@inline function top_bc_func(i, j, grid, clock, model_fields, parameters)
    k = grid.Nz
    T1 = model_fields.T[i, j, k]
    return T1 * parameters.scale
end

T_bcs = FieldBoundaryConditions(
    top = ValueBoundaryCondition(top_bc_func;
        discrete_form = true,
        parameters = (scale = 1.0,)
    )
)

model = NonhydrostaticModel(
    grid = grid,
    tracers = (:T, :S),
    boundary_conditions = (T = T_bcs,)
)

simulation = Simulation(model, Î”t=0.1, stop_iteration=1)
run!(simulation)

Error

[ Info: Running on rank 1 of 2
[ Info: Running on rank 0 of 2
grid = 8Ã—16Ã—16 RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded} on Distributed{GPU{CUDABackend}} with 3Ã—3Ã—3 halo
â”œâ”€â”€ FullyConnected x âˆˆ [0.0, 0.5) regularly spaced with Î”x=0.0625
â”œâ”€â”€ Periodic y âˆˆ [0.0, 1.0)       regularly spaced with Î”y=0.0625
â””â”€â”€ Bounded  z âˆˆ [0.0, 1.0]       regularly spaced with Î”z=0.0625
grid = 8Ã—16Ã—16 RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded} on Distributed{GPU{CUDABackend}} with 3Ã—3Ã—3 halo
â”œâ”€â”€ FullyConnected x âˆˆ [0.5, 1.0) regularly spaced with Î”x=0.0625
â”œâ”€â”€ Periodic y âˆˆ [0.0, 1.0)       regularly spaced with Î”y=0.0625
â””â”€â”€ Bounded  z âˆˆ [0.0, 1.0]       regularly spaced with Î”z=0.0625
ERROR: LoadError: InvalidIRError: compiling MethodInstance for Oceananigans.BoundaryConditions.gpu__fill_bottom_and_top_halo!(::KernelAbstractions.CompilerMetadata{Oceananigans.Utils.OffsetStaticSize{(1:8, 1:16)}, KernelAbstractions.NDIteration.DynamicCheck, Nothing, Nothing, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.StaticSize{(1, 1)}, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, Tuple{Int64, Int64}, Oceananigans.Utils.OffsetStaticSize{(0, 0)}}}, ::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, ::BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, ::BoundaryCondition{Oceananigans.BoundaryConditions.Value, Oceananigans.BoundaryConditions.DiscreteBoundaryFunction{@NamedTuple{scale::Float64}, typeof(top_bc_func)}}, ::Tuple{Center, Center, Center}, ::RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Nothing}, ::Tuple{Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuDeviceArray{Float64, 3, 1}, recv::CuDeviceArray{Float64, 3, 1}}, @NamedTuple{send::CuDeviceArray{Float64, 3, 1}, recv::CuDeviceArray{Float64, 3, 1}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, @NamedTuple{time::Float64, last_Î”t::Float64, last_stage_Î”t::Float64, iteration::Int64, stage::Int64}, @NamedTuple{u::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, v::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, w::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, T::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, S::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}}}) resulted in invalid LLVM IR
ERROR: LoadError: InvalidIRError: compiling MethodInstance for Oceananigans.BoundaryConditions.gpu__fill_bottom_and_top_halo!(::KernelAbstractions.CompilerMetadata{Oceananigans.Utils.OffsetStaticSize{(1:8, 1:16)}, KernelAbstractions.NDIteration.DynamicCheck, Nothing, Nothing, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.StaticSize{(1, 1)}, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, Tuple{Int64, Int64}, Oceananigans.Utils.OffsetStaticSize{(0, 0)}}}, ::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, ::BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, ::BoundaryCondition{Oceananigans.BoundaryConditions.Value, Oceananigans.BoundaryConditions.DiscreteBoundaryFunction{@NamedTuple{scale::Float64}, typeof(top_bc_func)}}, ::Tuple{Center, Center, Center}, ::RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Nothing}, ::Tuple{Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuDeviceArray{Float64, 3, 1}, recv::CuDeviceArray{Float64, 3, 1}}, @NamedTuple{send::CuDeviceArray{Float64, 3, 1}, recv::CuDeviceArray{Float64, 3, 1}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, @NamedTuple{time::Float64, last_Î”t::Float64, last_stage_Î”t::Float64, iteration::Int64, stage::Int64}, @NamedTuple{u::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, v::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, w::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, T::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, S::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}}}) resulted in invalid LLVM IR
Reason: unsupported call to an unknown function (call to jl_f_getfield)
Reason: unsupported call to an unknown function (call to jl_f_getfield)
Stacktrace:
 [1] getproperty
Stacktrace:
 [1] getproperty
   @ ./Base.jl:49
   @ ./Base.jl:49
 [2] top_bc_func
 [2] top_bc_func
   @ /nesi/nobackup/uc04420/GPUs/melt_test.jl:26
 [3] getbc
   @ /nesi/nobackup/uc04420/GPUs/melt_test.jl:26
 [3] getbc
   @ ~/.julia/packages/Oceananigans/3ZIHr/src/BoundaryConditions/discrete_boundary_function.jl:39
 [4] getbc
   @ ~/.julia/packages/Oceananigans/3ZIHr/src/BoundaryConditions/discrete_boundary_function.jl:39
 [4] getbc
   @ ~/.julia/packages/Oceananigans/3ZIHr/src/BoundaryConditions/boundary_condition.jl:112
 [5] right_gradient
   @ ~/.julia/packages/Oceananigans/3ZIHr/src/BoundaryConditions/boundary_condition.jl:112
 [5] right_gradient
   @ ~/.julia/packages/Oceananigans/3ZIHr/src/BoundaryConditions/fill_halo_regions_value_gradient.jl:13
 [6] _fill_top_halo!
   @ ~/.julia/packages/Oceananigans/3ZIHr/src/BoundaryConditions/fill_halo_regions_value_gradient.jl:13
 [6] _fill_top_halo!
   @ ~/.julia/packages/Oceananigans/3ZIHr/src/BoundaryConditions/fill_halo_regions_value_gradient.jl:97
 [7] macro expansion
   @ ~/.julia/packages/Oceananigans/3ZIHr/src/BoundaryConditions/fill_halo_regions_value_gradient.jl:97
 [7] macro expansion
   @ ~/.julia/packages/Oceananigans/3ZIHr/src/BoundaryConditions/fill_halo_regions.jl:217
 [8] gpu__fill_bottom_and_top_halo!
   @ ~/.julia/packages/Oceananigans/3ZIHr/src/BoundaryConditions/fill_halo_regions.jl:217
 [8] gpu__fill_bottom_and_top_halo!
   @ ~/.julia/packages/KernelAbstractions/aLmvZ/src/macros.jl:332
 [9] gpu__fill_bottom_and_top_halo!
   @ ~/.julia/packages/KernelAbstractions/aLmvZ/src/macros.jl:332
   @ ./none:0
 [9] gpu__fill_bottom_and_top_halo!
   @ ./none:0
Hint: catch this exception as `err` and call `code_typed(err; interactive = true)` to introspect the erroneous code with Cthulhu.jl
Stacktrace:
Hint: catch this exception as `err` and call `code_typed(err; interactive = true)` to introspect the erroneous code with Cthulhu.jl
Stacktrace:
  [1] check_ir(job::GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}, args::LLVM.Module)
  [1] check_ir(job::GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}, args::LLVM.Module)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/OCZFZ/src/validation.jl:167
  [2] macro expansion
    @ GPUCompiler ~/.julia/packages/GPUCompiler/OCZFZ/src/validation.jl:167
  [2] macro expansion
    @ ~/.julia/packages/GPUCompiler/OCZFZ/src/driver.jl:420 [inlined]
    @ ~/.julia/packages/GPUCompiler/OCZFZ/src/driver.jl:420 [inlined]
  [3] emit_llvm(job::GPUCompiler.CompilerJob; kwargs::@Kwargs{})
  [3] emit_llvm(job::GPUCompiler.CompilerJob; kwargs::@Kwargs{})
    @ GPUCompiler ~/.julia/packages/GPUCompiler/OCZFZ/src/utils.jl:116
    @ GPUCompiler ~/.julia/packages/GPUCompiler/OCZFZ/src/utils.jl:116
  [4] emit_llvm(job::GPUCompiler.CompilerJob)
  [4] emit_llvm(job::GPUCompiler.CompilerJob)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/OCZFZ/src/utils.jl:114
    @ GPUCompiler ~/.julia/packages/GPUCompiler/OCZFZ/src/utils.jl:114
  [5] compile_unhooked(output::Symbol, job::GPUCompiler.CompilerJob; kwargs::@Kwargs{})
  [5] compile_unhooked(output::Symbol, job::GPUCompiler.CompilerJob; kwargs::@Kwargs{})
    @ GPUCompiler ~/.julia/packages/GPUCompiler/OCZFZ/src/driver.jl:95
    @ GPUCompiler ~/.julia/packages/GPUCompiler/OCZFZ/src/driver.jl:95
  [6] compile_unhooked
  [6] compile_unhooked
    @ ~/.julia/packages/GPUCompiler/OCZFZ/src/driver.jl:80 [inlined]
    @ ~/.julia/packages/GPUCompiler/OCZFZ/src/driver.jl:80 [inlined]
  [7] compile(target::Symbol, job::GPUCompiler.CompilerJob; kwargs::@Kwargs{})
  [7] compile(target::Symbol, job::GPUCompiler.CompilerJob; kwargs::@Kwargs{})
    @ GPUCompiler ~/.julia/packages/GPUCompiler/OCZFZ/src/driver.jl:67
  [8] compile
    @ GPUCompiler ~/.julia/packages/GPUCompiler/OCZFZ/src/driver.jl:67
  [8] compile
    @ ~/.julia/packages/GPUCompiler/OCZFZ/src/driver.jl:55 [inlined]
  [9] #1188
    @ ~/.julia/packages/GPUCompiler/OCZFZ/src/driver.jl:55 [inlined]
  [9] #1188
    @ ~/.julia/packages/CUDA/724Sm/src/compiler/compilation.jl:250 [inlined]
    @ ~/.julia/packages/CUDA/724Sm/src/compiler/compilation.jl:250 [inlined]
 [10] JuliaContext(f::CUDA.var"#1188#1191"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}}; kwargs::@Kwargs{})
 [10] JuliaContext(f::CUDA.var"#1188#1191"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}}; kwargs::@Kwargs{})
    @ GPUCompiler ~/.julia/packages/GPUCompiler/OCZFZ/src/driver.jl:34
    @ GPUCompiler ~/.julia/packages/GPUCompiler/OCZFZ/src/driver.jl:34
 [11] JuliaContext(f::Function)
 [11] JuliaContext(f::Function)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/OCZFZ/src/driver.jl:25
    @ GPUCompiler ~/.julia/packages/GPUCompiler/OCZFZ/src/driver.jl:25
 [12] compile(job::GPUCompiler.CompilerJob)
 [12] compile(job::GPUCompiler.CompilerJob)
    @ CUDA ~/.julia/packages/CUDA/724Sm/src/compiler/compilation.jl:249
    @ CUDA ~/.julia/packages/CUDA/724Sm/src/compiler/compilation.jl:249
 [13] actual_compilation(cache::Dict{Any, CuFunction}, src::Core.MethodInstance, world::UInt64, cfg::GPUCompiler.CompilerConfig{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}, compiler::typeof(CUDA.compile), linker::typeof(CUDA.link))
 [13] actual_compilation(cache::Dict{Any, CuFunction}, src::Core.MethodInstance, world::UInt64, cfg::GPUCompiler.CompilerConfig{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}, compiler::typeof(CUDA.compile), linker::typeof(CUDA.link))
    @ GPUCompiler ~/.julia/packages/GPUCompiler/OCZFZ/src/execution.jl:245
    @ GPUCompiler ~/.julia/packages/GPUCompiler/OCZFZ/src/execution.jl:245
 [14] cached_compilation(cache::Dict{Any, CuFunction}, src::Core.MethodInstance, cfg::GPUCompiler.CompilerConfig{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}, compiler::Function, linker::Function)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/OCZFZ/src/execution.jl:159
 [14] cached_compilation(cache::Dict{Any, CuFunction}, src::Core.MethodInstance, cfg::GPUCompiler.CompilerConfig{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}, compiler::Function, linker::Function)
 [15] macro expansion
    @ ~/.julia/packages/CUDA/724Sm/src/compiler/execution.jl:373 [inlined]
    @ GPUCompiler ~/.julia/packages/GPUCompiler/OCZFZ/src/execution.jl:159
 [16] macro expansion
 [15] macro expansion
    @ ./lock.jl:273 [inlined]
    @ ~/.julia/packages/CUDA/724Sm/src/compiler/execution.jl:373 [inlined]
 [16] macro expansion
    @ ./lock.jl:273 [inlined]
 [17] cufunction(f::typeof(Oceananigans.BoundaryConditions.gpu__fill_bottom_and_top_halo!), tt::Type{Tuple{KernelAbstractions.CompilerMetadata{Oceananigans.Utils.OffsetStaticSize{(1:8, 1:16)}, KernelAbstractions.NDIteration.DynamicCheck, Nothing, Nothing, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.StaticSize{(1, 1)}, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, Tuple{Int64, Int64}, Oceananigans.Utils.OffsetStaticSize{(0, 0)}}}, OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Value, Oceananigans.BoundaryConditions.DiscreteBoundaryFunction{@NamedTuple{scale::Float64}, typeof(top_bc_func)}}, Tuple{Center, Center, Center}, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Nothing}, Tuple{Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuDeviceArray{Float64, 3, 1}, recv::CuDeviceArray{Float64, 3, 1}}, @NamedTuple{send::CuDeviceArray{Float64, 3, 1}, recv::CuDeviceArray{Float64, 3, 1}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, @NamedTuple{time::Float64, last_Î”t::Float64, last_stage_Î”t::Float64, iteration::Int64, stage::Int64}, @NamedTuple{u::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, v::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, w::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, T::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, S::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}}}}}; kwargs::@Kwargs{always_inline::Bool, maxthreads::Int64})
 [17] cufunction(f::typeof(Oceananigans.BoundaryConditions.gpu__fill_bottom_and_top_halo!), tt::Type{Tuple{KernelAbstractions.CompilerMetadata{Oceananigans.Utils.OffsetStaticSize{(1:8, 1:16)}, KernelAbstractions.NDIteration.DynamicCheck, Nothing, Nothing, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.StaticSize{(1, 1)}, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, Tuple{Int64, Int64}, Oceananigans.Utils.OffsetStaticSize{(0, 0)}}}, OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Value, Oceananigans.BoundaryConditions.DiscreteBoundaryFunction{@NamedTuple{scale::Float64}, typeof(top_bc_func)}}, Tuple{Center, Center, Center}, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Nothing}, Tuple{Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuDeviceArray{Float64, 3, 1}, recv::CuDeviceArray{Float64, 3, 1}}, @NamedTuple{send::CuDeviceArray{Float64, 3, 1}, recv::CuDeviceArray{Float64, 3, 1}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, @NamedTuple{time::Float64, last_Î”t::Float64, last_stage_Î”t::Float64, iteration::Int64, stage::Int64}, @NamedTuple{u::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, v::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, w::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, T::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, S::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}}}}}; kwargs::@Kwargs{always_inline::Bool, maxthreads::Int64})
    @ CUDA ~/.julia/packages/CUDA/724Sm/src/compiler/execution.jl:368
    @ CUDA ~/.julia/packages/CUDA/724Sm/src/compiler/execution.jl:368
 [18] macro expansion
 [18] macro expansion
    @ ~/.julia/packages/CUDA/724Sm/src/compiler/execution.jl:112 [inlined]
    @ ~/.julia/packages/CUDA/724Sm/src/compiler/execution.jl:112 [inlined]
 [19] (::KernelAbstractions.Kernel{CUDABackend, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, Oceananigans.Utils.OffsetStaticSize{(1:8, 1:16)}, typeof(Oceananigans.BoundaryConditions.gpu__fill_bottom_and_top_halo!)})(::OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, ::Vararg{Any}; ndrange::Nothing, workgroupsize::Nothing)
 [19] (::KernelAbstractions.Kernel{CUDABackend, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, Oceananigans.Utils.OffsetStaticSize{(1:8, 1:16)}, typeof(Oceananigans.BoundaryConditions.gpu__fill_bottom_and_top_halo!)})(::OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, ::Vararg{Any}; ndrange::Nothing, workgroupsize::Nothing)
    @ CUDA.CUDAKernels ~/.julia/packages/CUDA/724Sm/src/CUDAKernels.jl:129
    @ CUDA.CUDAKernels ~/.julia/packages/CUDA/724Sm/src/CUDAKernels.jl:129
 [20] (::KernelAbstractions.Kernel{CUDABackend, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, Oceananigans.Utils.OffsetStaticSize{(1:8, 1:16)}, typeof(Oceananigans.BoundaryConditions.gpu__fill_bottom_and_top_halo!)})(::OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, ::Vararg{Any})
 [20] (::KernelAbstractions.Kernel{CUDABackend, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, Oceananigans.Utils.OffsetStaticSize{(1:8, 1:16)}, typeof(Oceananigans.BoundaryConditions.gpu__fill_bottom_and_top_halo!)})(::OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, ::Vararg{Any})
    @ CUDA.CUDAKernels ~/.julia/packages/CUDA/724Sm/src/CUDAKernels.jl:115
    @ CUDA.CUDAKernels ~/.julia/packages/CUDA/724Sm/src/CUDAKernels.jl:115
 [21] _launch!(::GPU{CUDABackend}, ::RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, ::Oceananigans.Utils.KernelParameters{:xy, (0, 0)}, ::Function, ::OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, ::BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, ::Vararg{Any}; exclude_periphery::Bool, reduced_dimensions::Tuple{}, active_cells_map::Nothing)
 [21] _launch!(::GPU{CUDABackend}, ::RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, ::Oceananigans.Utils.KernelParameters{:xy, (0, 0)}, ::Function, ::OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, ::BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, ::Vararg{Any}; exclude_periphery::Bool, reduced_dimensions::Tuple{}, active_cells_map::Nothing)
    @ Oceananigans.Utils ~/.julia/packages/Oceananigans/3ZIHr/src/Utils/kernel_launching.jl:326
    @ Oceananigans.Utils ~/.julia/packages/Oceananigans/3ZIHr/src/Utils/kernel_launching.jl:326
 [22] _launch!(::Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}, ::RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, ::Vararg{Any}; kwargs::@Kwargs{reduced_dimensions::Tuple{}})
 [22] _launch!(::Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}, ::RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, ::Vararg{Any}; kwargs::@Kwargs{reduced_dimensions::Tuple{}})
    @ Oceananigans.DistributedComputations ~/.julia/packages/Oceananigans/3ZIHr/src/DistributedComputations/distributed_kernel_launching.jl:5
 [23] _launch!
    @ Oceananigans.DistributedComputations ~/.julia/packages/Oceananigans/3ZIHr/src/DistributedComputations/distributed_kernel_launching.jl:5
    @ ~/.julia/packages/Oceananigans/3ZIHr/src/DistributedComputations/distributed_kernel_launching.jl:3 [inlined]
 [23] _launch!
 [24] launch!
    @ ~/.julia/packages/Oceananigans/3ZIHr/src/Utils/kernel_launching.jl:280 [inlined]
    @ ~/.julia/packages/Oceananigans/3ZIHr/src/DistributedComputations/distributed_kernel_launching.jl:3 [inlined]
 [24] launch!
    @ ~/.julia/packages/Oceananigans/3ZIHr/src/Utils/kernel_launching.jl:280 [inlined]
 [25] fill_bottom_and_top_halo!(::OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, ::BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, ::BoundaryCondition{Oceananigans.BoundaryConditions.Value, Oceananigans.BoundaryConditions.DiscreteBoundaryFunction{@NamedTuple{scale::Float64}, typeof(top_bc_func)}}, ::Symbol, ::Tuple{Int64, Int64}, ::Tuple{Center, Center, Center}, ::Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}, ::RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, ::Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, ::Vararg{Any}; only_local_halos::Bool, kwargs::@Kwargs{reduced_dimensions::Tuple{}})
    @ Oceananigans.BoundaryConditions ~/.julia/packages/Oceananigans/3ZIHr/src/BoundaryConditions/fill_halo_regions.jl:341
 [25] fill_bottom_and_top_halo!(::OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, ::BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, ::BoundaryCondition{Oceananigans.BoundaryConditions.Value, Oceananigans.BoundaryConditions.DiscreteBoundaryFunction{@NamedTuple{scale::Float64}, typeof(top_bc_func)}}, ::Symbol, ::Tuple{Int64, Int64}, ::Tuple{Center, Center, Center}, ::Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}, ::RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, ::Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, ::Vararg{Any}; only_local_halos::Bool, kwargs::@Kwargs{reduced_dimensions::Tuple{}})
    @ Oceananigans.BoundaryConditions ~/.julia/packages/Oceananigans/3ZIHr/src/BoundaryConditions/fill_halo_regions.jl:341
 [26] fill_halo_event!(::OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, ::Function, ::Tuple{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Value, Oceananigans.BoundaryConditions.DiscreteBoundaryFunction{@NamedTuple{scale::Float64}, typeof(top_bc_func)}}}, ::Tuple{Colon, Colon, Colon}, ::Tuple{Center, Center, Center}, ::Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}, ::RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, ::Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, ::Clock{Float64, Float64, Int64, Int64}, ::Vararg{Any}; async::Bool, only_local_halos::Bool, kwargs::@Kwargs{reduced_dimensions::Tuple{}})
    @ Oceananigans.DistributedComputations ~/.julia/packages/Oceananigans/3ZIHr/src/DistributedComputations/halo_communication.jl:223
 [26] fill_halo_event!(::OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, ::Function, ::Tuple{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Value, Oceananigans.BoundaryConditions.DiscreteBoundaryFunction{@NamedTuple{scale::Float64}, typeof(top_bc_func)}}}, ::Tuple{Colon, Colon, Colon}, ::Tuple{Center, Center, Center}, ::Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}, ::RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, ::Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, ::Clock{Float64, Float64, Int64, Int64}, ::Vararg{Any}; async::Bool, only_local_halos::Bool, kwargs::@Kwargs{reduced_dimensions::Tuple{}})
    @ Oceananigans.DistributedComputations ~/.julia/packages/Oceananigans/3ZIHr/src/DistributedComputations/halo_communication.jl:223
 [27] fill_halo_regions!(::OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, ::FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Value, Oceananigans.BoundaryConditions.DiscreteBoundaryFunction{@NamedTuple{scale::Float64}, typeof(top_bc_func)}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, ::Tuple{Colon, Colon, Colon}, ::Tuple{Center, Center, Center}, ::RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, ::Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, ::Clock{Float64, Float64, Int64, Int64}, ::Vararg{Any}; fill_boundary_normal_velocities::Bool, kwargs::@Kwargs{reduced_dimensions::Tuple{}, async::Bool})
    @ Oceananigans.DistributedComputations ~/.julia/packages/Oceananigans/3ZIHr/src/DistributedComputations/halo_communication.jl:125
 [28] fill_halo_regions!
    @ ~/.julia/packages/Oceananigans/3ZIHr/src/DistributedComputations/halo_communication.jl:112 [inlined]
 [29] #fill_halo_regions!#63
    @ ~/.julia/packages/Oceananigans/3ZIHr/src/DistributedComputations/halo_communication.jl:101 [inlined]
 [27] fill_halo_regions!(::OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, ::FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Value, Oceananigans.BoundaryConditions.DiscreteBoundaryFunction{@NamedTuple{scale::Float64}, typeof(top_bc_func)}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, ::Tuple{Colon, Colon, Colon}, ::Tuple{Center, Center, Center}, ::RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, ::Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, ::Clock{Float64, Float64, Int64, Int64}, ::Vararg{Any}; fill_boundary_normal_velocities::Bool, kwargs::@Kwargs{reduced_dimensions::Tuple{}, async::Bool})
    @ Oceananigans.DistributedComputations ~/.julia/packages/Oceananigans/3ZIHr/src/DistributedComputations/halo_communication.jl:125
 [28] fill_halo_regions!
    @ ~/.julia/packages/Oceananigans/3ZIHr/src/DistributedComputations/halo_communication.jl:112 [inlined]
 [29] #fill_halo_regions!#63
    @ ~/.julia/packages/Oceananigans/3ZIHr/src/DistributedComputations/halo_communication.jl:101 [inlined]
 [30] tupled_fill_halo_regions!(::Tuple{Field{Face, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Face, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Face, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open{Nothing}, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open{Nothing}, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Value, Oceananigans.BoundaryConditions.DiscreteBoundaryFunction{@NamedTuple{scale::Float64}, typeof(top_bc_func)}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, ::RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, ::Clock{Float64, Float64, Int64, Int64}, ::Vararg{Any}; kwargs::@Kwargs{fill_boundary_normal_velocities::Bool, async::Bool})
    @ Oceananigans.DistributedComputations ~/.julia/packages/Oceananigans/3ZIHr/src/DistributedComputations/halo_communication.jl:94
 [30] tupled_fill_halo_regions!(::Tuple{Field{Face, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Face, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Face, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open{Nothing}, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open{Nothing}, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Value, Oceananigans.BoundaryConditions.DiscreteBoundaryFunction{@NamedTuple{scale::Float64}, typeof(top_bc_func)}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, ::RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, ::Clock{Float64, Float64, Int64, Int64}, ::Vararg{Any}; kwargs::@Kwargs{fill_boundary_normal_velocities::Bool, async::Bool})
    @ Oceananigans.DistributedComputations ~/.julia/packages/Oceananigans/3ZIHr/src/DistributedComputations/halo_communication.jl:94
 [31] fill_halo_regions!(::@NamedTuple{u::Field{Face, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, v::Field{Center, Face, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, w::Field{Center, Center, Face, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open{Nothing}, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open{Nothing}, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, T::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Value, Oceananigans.BoundaryConditions.DiscreteBoundaryFunction{@NamedTuple{scale::Float64}, typeof(top_bc_func)}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, S::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, ::Clock{Float64, Float64, Int64, Int64}, ::Vararg{Any}; signed::Bool, kwargs::@Kwargs{fill_boundary_normal_velocities::Bool, async::Bool})
    @ Oceananigans.Fields ~/.julia/packages/Oceananigans/3ZIHr/src/Fields/field_tuples.jl:66
 [31] fill_halo_regions!(::@NamedTuple{u::Field{Face, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, v::Field{Center, Face, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, w::Field{Center, Center, Face, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open{Nothing}, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open{Nothing}, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, T::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Value, Oceananigans.BoundaryConditions.DiscreteBoundaryFunction{@NamedTuple{scale::Float64}, typeof(top_bc_func)}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, S::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, ::Clock{Float64, Float64, Int64, Int64}, ::Vararg{Any}; signed::Bool, kwargs::@Kwargs{fill_boundary_normal_velocities::Bool, async::Bool})
    @ Oceananigans.Fields ~/.julia/packages/Oceananigans/3ZIHr/src/Fields/field_tuples.jl:66
 [32] update_state!(model::NonhydrostaticModel{Oceananigans.TimeSteppers.RungeKutta3TimeStepper{Float64, @NamedTuple{u::Field{Face, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, v::Field{Center, Face, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, w::Field{Center, Center, Face, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, T::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, S::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, Nothing}, Nothing, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Float64, Nothing, Nothing, Nothing, @NamedTuple{u::Field{Face, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, v::Field{Center, Face, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, w::Field{Center, Center, Face, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open{Nothing}, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open{Nothing}, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, @NamedTuple{T::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Value, Oceananigans.BoundaryConditions.DiscreteBoundaryFunction{@NamedTuple{scale::Float64}, typeof(top_bc_func)}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, S::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, @NamedTuple{pNHS::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, pHYâ€²::Nothing}, @NamedTuple{u::typeof(Oceananigans.Forcings.zeroforcing), v::typeof(Oceananigans.Forcings.zeroforcing), w::typeof(Oceananigans.Forcings.zeroforcing), T::typeof(Oceananigans.Forcings.zeroforcing), S::typeof(Oceananigans.Forcings.zeroforcing)}, Centered{1, Float64, Nothing}, DistributedFFTBasedPoissonSolver{@NamedTuple{forward::@NamedTuple{z!::Oceananigans.Solvers.DiscreteTransform{CUDA.CUFFT.CuFFTPlan{ComplexF64, ComplexF64, -1, true, 3, 1, Nothing}, Oceananigans.Solvers.Forward, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Int64, Bounded, Int64, @NamedTuple{forward::CuArray{ComplexF64, 3, CUDA.DeviceMemory}, backward::CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, Nothing}, y!::Oceananigans.Solvers.DiscreteTransform{CUDA.CUFFT.CuFFTPlan{ComplexF64, ComplexF64, -1, true, 3, 1, Nothing}, Oceananigans.Solvers.Forward, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Int64, Periodic, Int64, Nothing, Tuple{Int64, Int64, Int64}}, x!::Oceananigans.Solvers.DiscreteTransform{CUDA.CUFFT.CuFFTPlan{ComplexF64, ComplexF64, -1, true, 3, 1, Nothing}, Oceananigans.Solvers.Forward, RectilinearGrid{Float64, Periodic, Oceananigans.Grids.FullyConnected, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Nothing, Nothing, Int64, Int64, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Int64, Periodic, Int64, Nothing, Nothing}}, backward::@NamedTuple{x!::Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, CUDA.CUFFT.CuFFTPlan{ComplexF64, ComplexF64, 1, true, 3, 1, Nothing}, Float64}, Oceananigans.Solvers.Backward, RectilinearGrid{Float64, Periodic, Oceananigans.Grids.FullyConnected, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Nothing, Nothing, Int64, Int64, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Int64, Periodic, Int64, Nothing, Nothing}, y!::Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, CUDA.CUFFT.CuFFTPlan{ComplexF64, ComplexF64, 1, true, 3, 1, Nothing}, Float64}, Oceananigans.Solvers.Backward, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Int64, Periodic, Int64, Nothing, Tuple{Int64, Int64, Int64}}, z!::Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, CUDA.CUFFT.CuFFTPlan{ComplexF64, ComplexF64, 1, true, 3, 1, Nothing}, Float64}, Oceananigans.Solvers.Backward, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Int64, Bounded, Int64, @NamedTuple{forward::CuArray{ComplexF64, 3, CUDA.DeviceMemory}, backward::CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, Nothing}}}, RectilinearGrid{Float64, Periodic, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, GPU{CUDABackend}}, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{CuArray{Float64, 1, CUDA.DeviceMemory}, CuArray{Float64, 1, CUDA.DeviceMemory}, CuArray{Float64, 1, CUDA.DeviceMemory}}, @NamedTuple{x::Nothing, y::CuArray{ComplexF64, 3, CUDA.DeviceMemory}, z::CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, Oceananigans.DistributedComputations.TransposableField{Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Periodic, Oceananigans.Grids.FullyConnected, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Nothing, Nothing, Int64, Int64, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{UnitRange{Int64}, UnitRange{Int64}, UnitRange{Int64}}, OffsetArrays.OffsetArray{ComplexF64, 3, CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, ComplexF64, FieldBoundaryConditions{Nothing, Nothing, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{Nothing, Nothing, @NamedTuple{send::CuArray{ComplexF64, 3, CUDA.DeviceMemory}, recv::CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{ComplexF64, 3, CUDA.DeviceMemory}, recv::CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{UnitRange{Int64}, UnitRange{Int64}, UnitRange{Int64}}, OffsetArrays.OffsetArray{ComplexF64, 3, CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, ComplexF64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Nothing, Nothing, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{ComplexF64, 3, CUDA.DeviceMemory}, recv::CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{ComplexF64, 3, CUDA.DeviceMemory}, recv::CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{UnitRange{Int64}, UnitRange{Int64}, UnitRange{Int64}}, OffsetArrays.OffsetArray{ComplexF64, 3, CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, ComplexF64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Nothing, Nothing, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{ComplexF64, 3, CUDA.DeviceMemory}, recv::CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{ComplexF64, 3, CUDA.DeviceMemory}, recv::CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Nothing, @NamedTuple{send::CuArray{ComplexF64, 1, CUDA.DeviceMemory}, recv::CuArray{ComplexF64, 1, CUDA.DeviceMemory}}, @NamedTuple{yz::Vector{Int64}, xy::Vector{Int64}}, @NamedTuple{yz::MPI.Comm, xy::MPI.Comm}}}, Nothing, Oceananigans.Models.NonhydrostaticModels.BackgroundFields{false, @NamedTuple{u::Oceananigans.Fields.ZeroField{Int64, 3}, v::Oceananigans.Fields.ZeroField{Int64, 3}, w::Oceananigans.Fields.ZeroField{Int64, 3}}, @NamedTuple{T::Oceananigans.Fields.ZeroField{Int64, 3}, S::Oceananigans.Fields.ZeroField{Int64, 3}}}, Nothing, Nothing, @NamedTuple{}}, callbacks::Vector{Any}; compute_tendencies::Bool)
    @ Oceananigans.Models.NonhydrostaticModels ~/.julia/packages/Oceananigans/3ZIHr/src/Models/NonhydrostaticModels/update_nonhydrostatic_model_state.jl:34
 [33] update_state!
    @ ~/.julia/packages/Oceananigans/3ZIHr/src/Models/NonhydrostaticModels/update_nonhydrostatic_model_state.jl:20 [inlined]
 [32] update_state!(model::NonhydrostaticModel{Oceananigans.TimeSteppers.RungeKutta3TimeStepper{Float64, @NamedTuple{u::Field{Face, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, v::Field{Center, Face, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, w::Field{Center, Center, Face, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, T::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, S::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, Nothing}, Nothing, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Float64, Nothing, Nothing, Nothing, @NamedTuple{u::Field{Face, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, v::Field{Center, Face, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, w::Field{Center, Center, Face, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open{Nothing}, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open{Nothing}, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, @NamedTuple{T::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Value, Oceananigans.BoundaryConditions.DiscreteBoundaryFunction{@NamedTuple{scale::Float64}, typeof(top_bc_func)}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, S::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, @NamedTuple{pNHS::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, pHYâ€²::Nothing}, @NamedTuple{u::typeof(Oceananigans.Forcings.zeroforcing), v::typeof(Oceananigans.Forcings.zeroforcing), w::typeof(Oceananigans.Forcings.zeroforcing), T::typeof(Oceananigans.Forcings.zeroforcing), S::typeof(Oceananigans.Forcings.zeroforcing)}, Centered{1, Float64, Nothing}, DistributedFFTBasedPoissonSolver{@NamedTuple{forward::@NamedTuple{z!::Oceananigans.Solvers.DiscreteTransform{CUDA.CUFFT.CuFFTPlan{ComplexF64, ComplexF64, -1, true, 3, 1, Nothing}, Oceananigans.Solvers.Forward, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Int64, Bounded, Int64, @NamedTuple{forward::CuArray{ComplexF64, 3, CUDA.DeviceMemory}, backward::CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, Nothing}, y!::Oceananigans.Solvers.DiscreteTransform{CUDA.CUFFT.CuFFTPlan{ComplexF64, ComplexF64, -1, true, 3, 1, Nothing}, Oceananigans.Solvers.Forward, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Int64, Periodic, Int64, Nothing, Tuple{Int64, Int64, Int64}}, x!::Oceananigans.Solvers.DiscreteTransform{CUDA.CUFFT.CuFFTPlan{ComplexF64, ComplexF64, -1, true, 3, 1, Nothing}, Oceananigans.Solvers.Forward, RectilinearGrid{Float64, Periodic, Oceananigans.Grids.FullyConnected, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Nothing, Nothing, Int64, Int64, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Int64, Periodic, Int64, Nothing, Nothing}}, backward::@NamedTuple{x!::Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, CUDA.CUFFT.CuFFTPlan{ComplexF64, ComplexF64, 1, true, 3, 1, Nothing}, Float64}, Oceananigans.Solvers.Backward, RectilinearGrid{Float64, Periodic, Oceananigans.Grids.FullyConnected, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Nothing, Nothing, Int64, Int64, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Int64, Periodic, Int64, Nothing, Nothing}, y!::Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, CUDA.CUFFT.CuFFTPlan{ComplexF64, ComplexF64, 1, true, 3, 1, Nothing}, Float64}, Oceananigans.Solvers.Backward, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Int64, Periodic, Int64, Nothing, Tuple{Int64, Int64, Int64}}, z!::Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, CUDA.CUFFT.CuFFTPlan{ComplexF64, ComplexF64, 1, true, 3, 1, Nothing}, Float64}, Oceananigans.Solvers.Backward, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Int64, Bounded, Int64, @NamedTuple{forward::CuArray{ComplexF64, 3, CUDA.DeviceMemory}, backward::CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, Nothing}}}, RectilinearGrid{Float64, Periodic, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, GPU{CUDABackend}}, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{CuArray{Float64, 1, CUDA.DeviceMemory}, CuArray{Float64, 1, CUDA.DeviceMemory}, CuArray{Float64, 1, CUDA.DeviceMemory}}, @NamedTuple{x::Nothing, y::CuArray{ComplexF64, 3, CUDA.DeviceMemory}, z::CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, Oceananigans.DistributedComputations.TransposableField{Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Periodic, Oceananigans.Grids.FullyConnected, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Nothing, Nothing, Int64, Int64, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{UnitRange{Int64}, UnitRange{Int64}, UnitRange{Int64}}, OffsetArrays.OffsetArray{ComplexF64, 3, CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, ComplexF64, FieldBoundaryConditions{Nothing, Nothing, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{Nothing, Nothing, @NamedTuple{send::CuArray{ComplexF64, 3, CUDA.DeviceMemory}, recv::CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{ComplexF64, 3, CUDA.DeviceMemory}, recv::CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{UnitRange{Int64}, UnitRange{Int64}, UnitRange{Int64}}, OffsetArrays.OffsetArray{ComplexF64, 3, CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, ComplexF64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Nothing, Nothing, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{ComplexF64, 3, CUDA.DeviceMemory}, recv::CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{ComplexF64, 3, CUDA.DeviceMemory}, recv::CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{UnitRange{Int64}, UnitRange{Int64}, UnitRange{Int64}}, OffsetArrays.OffsetArray{ComplexF64, 3, CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, ComplexF64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Nothing, Nothing, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{ComplexF64, 3, CUDA.DeviceMemory}, recv::CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{ComplexF64, 3, CUDA.DeviceMemory}, recv::CuArray{ComplexF64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Nothing, @NamedTuple{send::CuArray{ComplexF64, 1, CUDA.DeviceMemory}, recv::CuArray{ComplexF64, 1, CUDA.DeviceMemory}}, @NamedTuple{yz::Vector{Int64}, xy::Vector{Int64}}, @NamedTuple{yz::MPI.Comm, xy::MPI.Comm}}}, Nothing, Oceananigans.Models.NonhydrostaticModels.BackgroundFields{false, @NamedTuple{u::Oceananigans.Fields.ZeroField{Int64, 3}, v::Oceananigans.Fields.ZeroField{Int64, 3}, w::Oceananigans.Fields.ZeroField{Int64, 3}}, @NamedTuple{T::Oceananigans.Fields.ZeroField{Int64, 3}, S::Oceananigans.Fields.ZeroField{Int64, 3}}}, Nothing, Nothing, @NamedTuple{}}, callbacks::Vector{Any}; compute_tendencies::Bool)
    @ Oceananigans.Models.NonhydrostaticModels ~/.julia/packages/Oceananigans/3ZIHr/src/Models/NonhydrostaticModels/update_nonhydrostatic_model_state.jl:34
 [33] update_state!
    @ ~/.julia/packages/Oceananigans/3ZIHr/src/Models/NonhydrostaticModels/update_nonhydrostatic_model_state.jl:20 [inlined]
 [34] NonhydrostaticModel(; grid::RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, clock::Clock{Float64, Float64, Int64, Int64}, advection::Centered{1, Float64, Nothing}, buoyancy::Nothing, coriolis::Nothing, stokes_drift::Nothing, forcing::@NamedTuple{}, closure::Nothing, boundary_conditions::@NamedTuple{T::FieldBoundaryConditions{Oceananigans.BoundaryConditions.DefaultBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Oceananigans.BoundaryConditions.DefaultBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Oceananigans.BoundaryConditions.DefaultBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Oceananigans.BoundaryConditions.DefaultBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Oceananigans.BoundaryConditions.DefaultBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, BoundaryCondition{Oceananigans.BoundaryConditions.Value, Oceananigans.BoundaryConditions.DiscreteBoundaryFunction{@NamedTuple{scale::Float64}, typeof(top_bc_func)}}, Oceananigans.BoundaryConditions.DefaultBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}}, tracers::Tuple{Symbol, Symbol}, timestepper::Symbol, background_fields::@NamedTuple{}, particles::Nothing, biogeochemistry::Nothing, velocities::Nothing, hydrostatic_pressure_anomaly::Oceananigans.Models.NonhydrostaticModels.DefaultHydrostaticPressureAnomaly, nonhydrostatic_pressure::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, diffusivity_fields::Nothing, pressure_solver::Nothing, auxiliary_fields::@NamedTuple{})
    @ Oceananigans.Models.NonhydrostaticModels ~/.julia/packages/Oceananigans/3ZIHr/src/Models/NonhydrostaticModels/nonhydrostatic_model.jl:236
 [35] top-level scope
    @ /nesi/nobackup/uc04420/GPUs/melt_test.jl:37
in expression starting at /nesi/nobackup/uc04420/GPUs/melt_test.jl:37
 [34] NonhydrostaticModel(; grid::RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, clock::Clock{Float64, Float64, Int64, Int64}, advection::Centered{1, Float64, Nothing}, buoyancy::Nothing, coriolis::Nothing, stokes_drift::Nothing, forcing::@NamedTuple{}, closure::Nothing, boundary_conditions::@NamedTuple{T::FieldBoundaryConditions{Oceananigans.BoundaryConditions.DefaultBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Oceananigans.BoundaryConditions.DefaultBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Oceananigans.BoundaryConditions.DefaultBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Oceananigans.BoundaryConditions.DefaultBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Oceananigans.BoundaryConditions.DefaultBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, BoundaryCondition{Oceananigans.BoundaryConditions.Value, Oceananigans.BoundaryConditions.DiscreteBoundaryFunction{@NamedTuple{scale::Float64}, typeof(top_bc_func)}}, Oceananigans.BoundaryConditions.DefaultBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}}, tracers::Tuple{Symbol, Symbol}, timestepper::Symbol, background_fields::@NamedTuple{}, particles::Nothing, biogeochemistry::Nothing, velocities::Nothing, hydrostatic_pressure_anomaly::Oceananigans.Models.NonhydrostaticModels.DefaultHydrostaticPressureAnomaly, nonhydrostatic_pressure::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Oceananigans.Grids.FullyConnected, Periodic, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Distributed{GPU{CUDABackend}, false, Partition{Int64, Nothing, Nothing}, Tuple{Int64, Int64, Int64}, Int64, Tuple{Int64, Int64, Int64}, Oceananigans.DistributedComputations.NeighboringRanks{Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, MPI.Comm, Vector{MPI.Request}, Base.RefValue{Int64}, Nothing}}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.DistributedComputations.CommunicationBuffers{@NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, @NamedTuple{send::CuArray{Float64, 3, CUDA.DeviceMemory}, recv::CuArray{Float64, 3, CUDA.DeviceMemory}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, diffusivity_fields::Nothing, pressure_solver::Nothing, auxiliary_fields::@NamedTuple{})
    @ Oceananigans.Models.NonhydrostaticModels ~/.julia/packages/Oceananigans/3ZIHr/src/Models/NonhydrostaticModels/nonhydrostatic_model.jl:236
 [35] top-level scope
    @ /nesi/nobackup/uc04420/GPUs/melt_test.jl:37
in expression starting at /nesi/nobackup/uc04420/GPUs/melt_test.jl:37
srun: error: g09: tasks 0-1: Exited with exit code 1
srun: Terminating StepId=5264604.0

glwagner · 2026-04-01T23:08:12Z

glwagner
Apr 1, 2026
Maintainer

@phyo-wai-thaw can you try using this boundary condition?

@inline function top_bc_func(i, j, grid, clock, model_fields, parameters)
    k = grid.Nz
    T1 = @inbounds model_fields.T[i, j, k]
    return T1 * parameters.scale
end

you're missing an inbounds --- not sure if this is the exact problem, but it looks like you're facing a compilation failure.

The other suggestion is to try on Distributed(CPU()) --- what is the error then?

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue with discrete boundary condition accessing model_fields on distributed GPU setup #5461

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Issue with discrete boundary condition accessing model_fields on distributed GPU setup #5461

Uh oh!

phyo-wai-thaw Mar 31, 2026

Replies: 1 comment

Uh oh!

Uh oh!

glwagner Apr 1, 2026 Maintainer

phyo-wai-thaw
Mar 31, 2026

glwagner
Apr 1, 2026
Maintainer