New regional Panantarctic example by francispoulin · Pull Request #142 · CliMA/ClimaOcean.jl

francispoulin · 2024-08-15T12:53:38Z

Following up on #106, this is a first attempt to create a regional model with ECCO-derived restoring at the boundaries. We decided to try focusing on the ACC in the southern ocean.

It does not run yet, but after it does, it would be good to know if people agree this is a good example to include. If yes, then we need to turn this into an example.

francispoulin · 2024-08-15T12:54:48Z

The error that I get is copied below.

@simone-silvestri @glwagner

[ Info: In-painting ecco salinity
[ Info: In-painting ecco salinity
ERROR: a bounds error was thrown during kernel execution on thread (65, 1, 1) in block (3, 1, 1).
Stacktrace not available, run Julia on debug level 2 for more details (by passing -g2 to the executable).

ERROR: LoadError: KernelException: exception thrown during kernel execution on device NVIDIA A100-SXM4-40GB
Stacktrace:
  [1] check_exceptions()
    @ CUDA ~/.julia/packages/CUDA/Tl08O/src/compiler/exceptions.jl:39
  [2] device_synchronize(; blocking::Bool, spin::Bool)

simone-silvestri · 2024-08-15T12:55:33Z

what if you use the CPU and start julia with --check-bounds=yes?

francispoulin · 2024-08-15T12:58:18Z

Thanks @simone-silvestri for the suggestion. Will try it now.

francispoulin · 2024-08-15T13:46:22Z

Things go a lot further but there is a problem with the lines that defines coupled_model. It seems that the matrix it has and wants are not the same size.

It seems this is with assemble_atmosphere_ocean_fluxes.

julia> include("acc_regional_simulation.jl")
[ Info: Regridding bathymetry from existing file /u/fpoulin/.julia/scratchspaces/0376089a-ecfe-4b0e-a64f-9c555d74d754/Bathymetry/ETOPO_2022_v1_60s_N90W180_surface.nc.
┌ Warning: The westernmost meridian of `target_grid` 0.0 does not coincide with the closest meridian of the bathymetry grid, -1.4210854715202004e-14.
└ @ ClimaOcean.Bathymetry ~/software/ClimaOcean.jl/src/Bathymetry.jl:147
[ Info: In-painting ecco temperature
[ Info: In-painting ecco temperature
[ Info: In-painting ecco salinity
[ Info: In-painting ecco salinity
┌ Warning: This simulation will run forever as stop iteration = stop time = wall time limit = Inf.
└ @ Oceananigans.Simulations ~/.julia/packages/Oceananigans/dvdXO/src/Simulations/simulation.jl:55
[ Info: In-painting ecco temperature
[ Info: In-painting ecco salinity
ERROR: LoadError: BoundsError: attempt to access 21×46 Matrix{Float64} at index [22, 40]
Stacktrace:
  [1] getindex
    @ ./essentials.jl:14 [inlined]
  [2] net_downwelling_radiation
    @ ~/software/ClimaOcean.jl/src/OceanSeaIceModels/CrossRealmFluxes/tabulated_albedo.jl:156 [inlined]
  [3] macro expansion
    @ ~/software/ClimaOcean.jl/src/OceanSeaIceModels/CrossRealmFluxes/atmosphere_ocean_fluxes.jl:275 [inlined]
  [4] cpu__assemble_atmosphere_ocean_fluxes!
    @ ~/.julia/packages/KernelAbstractions/QE5mt/src/macros.jl:287 [inlined]
  [5] cpu__assemble_atmosphere_ocean_fluxes!(__ctx__::KernelAbstractions.CompilerMetadata{…}, centered_velocity_fluxes::@NamedTuple{…}, net_tracer_fluxes::@NamedTuple{…}, grid::ImmersedBoundaryGrid{…}, clock::Clock{…}, ocean_temperature::SubArray{…}, ocean_salinity::SubArray{…}, ocean_temperature_units::ClimaOcean.OceanSeaIceModels.CrossRealmFluxes.DegreesCelsius, similarity_theory_fields::@NamedTuple{…}, downwelling_radiation::@NamedTuple{…}, prescribed_freshwater_flux::@NamedTuple{…}, atmos_grid::Oceananigans.Grids.ZRegularLLG{…}, atmos_times::StepRangeLen{…}, atmos_backend::JRA55NetCDFBackend, atmos_time_indexing::Oceananigans.OutputReaders.Cyclical{…}, runoff_args::Tuple{…}, radiation_properties::Radiation{…}, ocean_reference_density::Float64, ocean_heat_capacity::Float64, freshwater_density::Float64)
    @ ClimaOcean.OceanSeaIceModels.CrossRealmFluxes ./none:0
  [6] __thread_run(tid::Int64, len::Int64, rem::Int64, obj::KernelAbstractions.Kernel{…}, ndrange::Nothing, iterspace::KernelAbstractions.NDIteration.NDRange{…}, args::Tuple{…}, dynamic::KernelAbstractions.NDIteration.DynamicCheck)
    @ KernelAbstractions ~/.julia/packages/KernelAbstractions/QE5mt/src/cpu.jl:140
  [7] __run(obj::KernelAbstractions.Kernel{…}, ndrange::Nothing, iterspace::KernelAbstractions.NDIteration.NDRange{…}, args::Tuple{…}, dynamic::KernelAbstractions.NDIteration.DynamicCheck, static_threads::Bool)
    @ KernelAbstractions ~/.julia/packages/KernelAbstractions/QE5mt/src/cpu.jl:107
  [8] (::KernelAbstractions.Kernel{…})(::@NamedTuple{…}, ::Vararg{…}; ndrange::Nothing, workgroupsize::Nothing)
    @ KernelAbstractions ~/.julia/packages/KernelAbstractions/QE5mt/src/cpu.jl:46
  [9] (::KernelAbstractions.Kernel{…})(::@NamedTuple{…}, ::Vararg{…})
    @ KernelAbstractions ~/.julia/packages/KernelAbstractions/QE5mt/src/cpu.jl:39
 [10] launch!(::CPU, ::ImmersedBoundaryGrid{…}, ::Oceananigans.Utils.KernelParameters{…}, ::typeof(ClimaOcean.OceanSeaIceModels.CrossRealmFluxes._assemble_atmosphere_ocean_fluxes!), ::@NamedTuple{…}, ::Vararg{…}; include_right_boundaries::Bool, reduced_dimensions::Tuple{}, location::Nothing, active_cells_map::Nothing, kwargs::@Kwargs{})
    @ Oceananigans.Utils ~/.julia/packages/Oceananigans/dvdXO/src/Utils/kernel_launching.jl:168
 [11] launch!(::CPU, ::ImmersedBoundaryGrid{…}, ::Oceananigans.Utils.KernelParameters{…}, ::Function, ::@NamedTuple{…}, ::Vararg{…})
    @ Oceananigans.Utils ~/.julia/packages/Oceananigans/dvdXO/src/Utils/kernel_launching.jl:154
 [12] compute_atmosphere_ocean_fluxes!(coupled_model::OceanSeaIceModel{…})
    @ ClimaOcean.OceanSeaIceModels.CrossRealmFluxes ~/software/ClimaOcean.jl/src/OceanSeaIceModels/CrossRealmFluxes/atmosphere_ocean_fluxes.jl:77
 [13] update_state!(coupled_model::OceanSeaIceModel{…}, callbacks::Vector{…}; compute_tendencies::Bool)
    @ ClimaOcean.OceanSeaIceModels ~/software/ClimaOcean.jl/src/OceanSeaIceModels/ocean_only_model.jl:42
 [14] update_state!
    @ ClimaOcean.OceanSeaIceModels ~/software/ClimaOcean.jl/src/OceanSeaIceModels/ocean_only_model.jl:30 [inlined]
 [15] update_state!(coupled_model::OceanSeaIceModel{…})
    @ ClimaOcean.OceanSeaIceModels ~/software/ClimaOcean.jl/src/OceanSeaIceModels/ocean_only_model.jl:30
 [16] OceanSeaIceModel(ocean::Simulation{…}, sea_ice::ClimaOcean.OceanSeaIceModels.MinimumTemperatureSeaIce{…}; atmosphere::ClimaOcean.OceanSeaIceModels.PrescribedAtmospheres.PrescribedAtmosphere{…}, radiation::Radiation{…}, similarity_theory::Nothing, ocean_reference_density::Float64, ocean_heat_capacity::Float64, clock::Clock{…})
    @ ClimaOcean.OceanSeaIceModels ~/software/ClimaOcean.jl/src/OceanSeaIceModels/ocean_sea_ice_model.jl:82
 [17] top-level scope
    @ REPL[15]:1
Some type information was truncated. Use `show(err)` to see complete types.

francispoulin · 2024-08-20T16:51:46Z

@simone-silvestri , any advice on what is going wrong here?

simone-silvestri · 2024-08-20T18:20:04Z

It looks like there is a bug in the TabulatedAlbedo function. i.e., there is no check to make sure that we stay within bounds when interpolating in the table.
I will open a PR to fix this issue. In the meantime, if you want to proceed with the implementation without incurring in this problem, you can use radiation = Radiation(ocean_albedo = LatitudeDependentAlbedo())

francispoulin · 2024-08-20T18:28:38Z

Thanks @simone-silvestri , I will give that a try!

Make changes so that it runs

francispoulin · 2024-08-20T19:17:21Z

@simone-silvestri : I tried it and it seems like a function is not defined.

I added this at the beginning and now it seems to be running!

using ClimaOcean.OceanSeaIceModels.CrossRealmFluxes: LatitudeDependentAlbedo

simone-silvestri · 2024-08-20T20:00:00Z

Ah nice. I think we can export that type.

codecov · 2024-08-20T20:07:05Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (b3ae3fe) to head (8cc2719).
Report is 5 commits behind head on main.

Additional details and impacted files

@@          Coverage Diff          @@
##            main    #142   +/-   ##
=====================================
  Coverage   0.00%   0.00%           
=====================================
  Files         34      34           
  Lines       1962    1983   +21     
=====================================
- Misses      1962    1983   +21

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

francispoulin · 2024-08-20T20:08:25Z

It's running on a CPU (i.e. slow) and still on the initial time step.

I made all these changes on the branch and can revert back to what we had previously as other fixes come along.

Maybe I'll have something to share tomorrow.

francispoulin · 2024-08-22T16:40:12Z

I started the job yesterday and it hasn't updated the output files in over 24 hours. I think something has gone wrong. Below is the currently display that I have. It hasn't stopped and still running on a CPU. Maybe we need to try it on a GPU or have more output to see what has gone wrong? Any suggestions?

julia> include("acc_regional_simulation.jl")
[ Info: Regridding bathymetry from existing file /u/fpoulin/.julia/scratchspaces/0376089a-ecfe-4b0e-a64f-9c555d74d754/Bathymetry/ETOPO_2022_v1_60s_N90W180_surface.nc.
┌ Warning: The westernmost meridian of `target_grid` 0.0 does not coincide with the closest meridian of the bathymetry grid, -1.4210854715202004e-14.
└ @ ClimaOcean.Bathymetry ~/software/ClimaOcean.jl/src/Bathymetry.jl:147
[ Info: In-painting ecco temperature
[ Info: In-painting ecco temperature
[ Info: In-painting ecco salinity
[ Info: In-painting ecco salinity
┌ Warning: This simulation will run forever as stop iteration = stop time = wall time limit = Inf.
└ @ Oceananigans.Simulations ~/.julia/packages/Oceananigans/dvdXO/src/Simulations/simulation.jl:55
[ Info: In-painting ecco temperature
[ Info: In-painting ecco salinity
[ Info: Initializing simulation...
[ Info: Time: 0 seconds, Iteration 0, Δt 5 minutes, max(vel): (0.00e+00, 0.00e+00, 0.00e+00), max(T): 29.70, min(T): -1.94, wtime: 2.679 minutes 
[ Info:     ... simulation initialization complete (14.029 seconds)
[ Info: Executing initial time step...
┌ Warning: Simulation stopped during initialization.
└ @ Oceananigans.Simulations ~/.julia/packages/Oceananigans/dvdXO/src/Simulations/run.jl:129

francispoulin · 2024-08-27T18:03:19Z

My correction. It is still running on one CPU. It is at 4 days after 7 days of computing. Not a great ratio.

What needs to be done so we need to do to run this on a GPU? @simone-silvestri

simone-silvestri · 2024-08-27T18:17:53Z

Wow, that seems quite slow! What if you move it on the GPU?

francispoulin · 2024-09-23T13:57:42Z

Wow, that seems quite slow! What if you move it on the GPU?

Sorry @simone-silvestri for the late reply.

I am happy to try it again on a GPU but last time there was an error. I can try it again and let you know what the error is.

francispoulin · 2024-09-23T14:18:22Z

@simone-silvestri
I ran it on a GPU and found the following error. It suggests I try passing -g2 when I run Julia. I can try that but I believe I tried this before and didn't see much more.

ERROR: a bounds error was thrown during kernel execution on thread (225, 1, 1) in block (7, 1, 1).
Stacktrace not available, run Julia on debug level 2 for more details (by passing -g2 to the executable).

francispoulin · 2024-09-23T14:45:16Z

I stand corrected, there is more information. To me this actually looks very different even.

julia> include("acc_regional_simulation.jl")
Precompiling Oceananigans
  166 dependencies successfully precompiled in 109 seconds
Precompiling ClimaOcean
        Info Given ClimaOcean was explicitly requested, output will be shown live 
WARNING: using Units.day in module ECCO conflicts with an existing identifier.
  204 dependencies successfully precompiled in 143 seconds. 168 already precompiled.
  2 dependencies had output during precompilation:
┌ ClimaOcean
│  [Output was shown above]
└  
┌ Accessors → AccessorsUnitfulExt
│  [pid 760940] waiting for IO to finish:
│   Handle type        uv_handle_t->data
│   fs_event           0x25d5fe0->0x7fdd5f8fbeb0
│   timer              0x2385440->0x7fdd5f8fbee0
│  This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
└  
[ Info: Regridding bathymetry from existing file /u/fpoulin/.julia/scratchspaces/0376089a-ecfe-4b0e-a64f-9c555d74d754/Bathymetry/ETOPO_2022_v1_60s_N90W180_surface.nc.
┌ Warning: The westernmost meridian of `target_grid` 0.0 does not coincide with the closest meridian of the bathymetry grid, -1.4210854715202004e-14.
└ @ ClimaOcean.Bathymetry ~/software/ClimaOcean.jl/src/Bathymetry.jl:147
[ Info: In-painting ecco temperature
[ Info: In-painting ecco temperature
[ Info: In-painting ecco salinity
[ Info: In-painting ecco salinity
ERROR: a bounds error was thrown during kernel execution on thread (1, 1, 1) in block (3, 1, 1).
Stacktrace:
 [1] indexed_iterate at ./tuple.jl:92
 [2] indexed_iterate at ./tuple.jl:92
 [3] stateindex at /u/fpoulin/software/ClimaOcean.jl/src/ClimaOcean.jl:40
 [4] ECCORestoring at /u/fpoulin/software/ClimaOcean.jl/src/DataWrangling/ecco_restoring.jl:210
 [5] DiscreteForcing at /u/fpoulin/.julia/packages/Oceananigans/dvdXO/src/Forcings/discrete_forcing.jl:51
 [6] hydrostatic_free_surface_tracer_tendency at /u/fpoulin/.julia/packages/Oceananigans/dvdXO/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:133
 [7] macro expansion at /u/fpoulin/.julia/packages/Oceananigans/dvdXO/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:240
 [8] gpu_compute_hydrostatic_free_surface_Gc! at /u/fpoulin/.julia/packages/KernelAbstractions/QE5mt/src/macros.jl:95
 [9] gpu_compute_hydrostatic_free_surface_Gc! at ./none:0

ERROR: a bounds error was thrown during kernel execution on thread (1, 1, 1) in block (67, 1, 1).
Stacktrace:
 [1] indexed_iterate at ./tuple.jl:92
 [2] indexed_iterate at ./tuple.jl:92
 [3] stateindex at /u/fpoulin/software/ClimaOcean.jl/src/ClimaOcean.jl:40
 [4] ECCORestoring at /u/fpoulin/software/ClimaOcean.jl/src/DataWrangling/ecco_restoring.jl:210
 [5] DiscreteForcing at /u/fpoulin/.julia/packages/Oceananigans/dvdXO/src/Forcings/discrete_forcing.jl:51
 [6] hydrostatic_free_surface_tracer_tendency at /u/fpoulin/.julia/packages/Oceananigans/dvdXO/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:133
 [7] macro expansion at /u/fpoulin/.julia/packages/Oceananigans/dvdXO/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:240
Unhandled Task ERROR: KernelException: exception thrown during kernel execution on device NVIDIA A100-SXM4-40GB

glwagner · 2024-09-23T18:29:37Z

Can you make an MWE for this and open an issue?

francispoulin · 2024-09-23T20:54:33Z

Can you make an MWE for this and open an issue?

I will certainly give it a try and see what part of it is causing the issue. This will likely take me a day or two to get to.

francispoulin · 2024-12-10T16:41:59Z

@simone-silvestri , I realize it's been a few months but I am still keen to this this example up and running.

I can try this all again this week but if you had time to meet for an hour, I wonder if that would help?

simone-silvestri · 2024-12-10T17:47:24Z

Sure, I ll text on slack.

francispoulin · 2025-09-15T19:05:02Z

Interestingly, when the new example is competely removed (thank you again @glwagner), the docs still fail. Sadly, I still can't see the errors.

I am trying to build the docs on a server and I will let you know if I come across errors of any kind.

francispoulin · 2025-09-16T00:13:39Z

When I tried running the make.jl script on another server to make the docs, I don't get an error. I haven't looked at any of them but it seems to have created all of them, see below, including the new example without a single error.

Question: why does it fail on these two checks on github?

shell> ls -lrt src/literated/
total 90056
-rw-rw-r-- 1 fpoulin fpoulin   276168 Sep 15 16:06 panantarctic_bathymetry.png
-rw-rw-r-- 1 fpoulin fpoulin  1590517 Sep 15 16:54 acc_snapshot.png
-rw-rw-r-- 1 fpoulin fpoulin 18169005 Sep 15 16:54 panantarctic_regional_surface.mp4
-rw-rw-r-- 1 fpoulin fpoulin    18135 Sep 15 16:54 panantarctic_regional_simulation.md
-rw-rw-r-- 1 fpoulin fpoulin  1747402 Sep 15 17:01 single_column_profiles.mp4
-rw-rw-r-- 1 fpoulin fpoulin   596778 Sep 15 17:01 single_column_os_papa_simulation.md
-rw-rw-r-- 1 fpoulin fpoulin   598273 Sep 15 18:05 global_snapshot.png
-rw-rw-r-- 1 fpoulin fpoulin 50084758 Sep 15 18:06 one_degree_global_ocean_surface.mp4
-rw-rw-r-- 1 fpoulin fpoulin   696041 Sep 15 18:06 one_degree_simulation.md
-rw-rw-r-- 1 fpoulin fpoulin   505603 Sep 15 18:07 bathymetry.png
-rw-rw-r-- 1 fpoulin fpoulin  2891474 Sep 15 19:13 snapshot.png
-rw-rw-r-- 1 fpoulin fpoulin 15000775 Sep 15 19:13 near_global_ocean_surface.mp4
-rw-rw-r-- 1 fpoulin fpoulin    17508 Sep 15 19:13 near_global_ocean_simulation.md

glwagner · 2025-09-16T12:52:26Z

It's an out-of-memory error on GPU

To display the buildkite publically we probably need to fiddle with buildkite settings

Let's try a coarser grid to see if this allows us to avoid the memory errors.

francispoulin · 2025-09-16T13:45:36Z

Ah, good to know!

I'm trying a coarser grid, 720x120x40, to see if this avoids the error.

francispoulin · 2025-09-16T21:47:58Z

Funny that when I reduce the resolution, and don't do anything else differently, now we have more errors. See below.

I'll return the parameters since this clearly did not help.

��� Warning: Opening file with JLD2.MmapIO failed, falling back to IOStream
��� @ JLD2 /central/scratch/esm/slurm-buildkite/climaocean-ci/4801/depot/default/packages/JLD2/WDhXU/src/JLD2.jl:162
Field utilities: Error During Test at /central/scratch/esm/slurm-buildkite/climaocean-ci/4801/climaocean-ci/test/test_ecco2_daily.jl:51
  Got exception outside of a @test
  EOFError: read end of file

Returning to the desired resolution since more errors happened with a coarser grid. Hmm...

francispoulin · 2025-12-08T15:41:39Z

Thanks @navidcy for updating it. I hope we get this pushed sooner than later.

I looked at the errors but when I do I get a login page for buildkite. Is that expected? Should I have a login to see those errors?

navidcy · 2025-12-08T18:17:00Z

I pushed some updates; let's see!

francispoulin · 2025-12-08T18:54:32Z

Looks like we are getting different errors from before, so maybe that's progress. The part that seems to be failing is with ecco4 dataset, in particular I think it's this line in test_ecco4_en4.jl.

        @testset "Timestepping with fields from Dataset" begin
            test_timestepping_with_dataset(arch, dataset, start_date, inpainting, 
                                           varnames=test_names[dataset], 
                                           fldnames=test_fields[dataset])

francispoulin · 2025-12-08T22:10:22Z

The error message that we get in the test is copied below.

What's strange is that when I try running this test on a server all the tests in this script. On this server I get a problem with the package CopernicusMarine not found, but if our tests don't complain about it then maybe it's not something to worry about. At least not now.

┌ Warning: Opening file with JLD2.MmapIO failed, falling back to IOStream
└ @ JLD2 /resnick/scratch/esm/slurm-buildkite/climaocean-ci/depot/default/packages/JLD2/hbsZG/src/JLD2.jl:162
Timestepping with fields from Dataset: Error During Test at /resnick/scratch/esm/slurm-buildkite/climaocean-ci/5288/climaocean-ci/test/runtests_setup.jl:90
  Test threw exception
  Expression: begin
    #= /resnick/scratch/esm/slurm-buildkite/climaocean-ci/5288/climaocean-ci/test/runtests_setup.jl:91 =#
    for name = varnames
        #= /resnick/scratch/esm/slurm-buildkite/climaocean-ci/5288/climaocean-ci/test/runtests_setup.jl:92 =#
        set!(field, Metadatum(name; dataset, date = start_date); inpainting)
        #= /resnick/scratch/esm/slurm-buildkite/climaocean-ci/5288/climaocean-ci/test/runtests_setup.jl:93 =#
    end
    #= /resnick/scratch/esm/slurm-buildkite/climaocean-ci/5288/climaocean-ci/test/runtests_setup.jl:94 =#
    true
end
  EOFError: read end of file
  Stacktrace:
    [1] jldopen(fname::String, wr::Bool, create::Bool, truncate::Bool, iotype::Type{JLD2.MmapIO}; fallback::Type{IOStream}, compress::Bool, mmaparrays::Bool, typemap::Function, parallel_read::Bool, plain::Bool)
      @ JLD2 /resnick/scratch/esm/slurm-buildkite/climaocean-ci/depot/default/packages/JLD2/hbsZG/src/JLD2.jl:242
    [2] jldopen
      @ /resnick/scratch/esm/slurm-buildkite/climaocean-ci/depot/default/packages/JLD2/hbsZG/src/JLD2.jl:174 [inlined]
    [3] #jldopen#23
      @ /resnick/scratch/esm/slurm-buildkite/climaocean-ci/depot/default/packages/JLD2/hbsZG/src/JLD2.jl:297 [inlined]
    [4] jldopen
      @ /resnick/scratch/esm/slurm-buildkite/climaocean-ci/depot/default/packages/JLD2/hbsZG/src/JLD2.jl:290 [inlined]
    [5] Field(metadata::Metadata{ECCO4Monthly, DateTime, Nothing}, arch::CPU; inpainting::NearestNeighborInpainting{Int64}, mask::Nothing, halo::Tuple{Int64, Int64, Int64}, cache_inpainted_data::Bool)
      @ ClimaOcean.DataWrangling /resnick/scratch/esm/slurm-buildkite/climaocean-ci/5288/climaocean-ci/src/DataWrangling/metadata_field.jl:102
    [6] Field
      @ /resnick/scratch/esm/slurm-buildkite/climaocean-ci/5288/climaocean-ci/src/DataWrangling/metadata_field.jl:87 [inlined]

francispoulin · 2025-12-09T01:06:15Z

Can someone explain to me why it is that when I click on the first two failed tests I can see exactly what has been returned, but when I do the last two, I get a login screen for buildkite?

glwagner · 2025-12-09T03:18:21Z

It's the same for me re: login screen. @cmbengue any ideas?

simone-silvestri · 2025-12-09T12:00:44Z

The error message that we get in the test is copied below.

What's strange is that when I try running this test on a server all the tests in this script. On this server I get a problem with the package CopernicusMarine not found, but if our tests don't complain about it then maybe it's not something to worry about. At least not now.

┌ Warning: Opening file with JLD2.MmapIO failed, falling back to IOStream
└ @ JLD2 /resnick/scratch/esm/slurm-buildkite/climaocean-ci/depot/default/packages/JLD2/hbsZG/src/JLD2.jl:162
Timestepping with fields from Dataset: Error During Test at /resnick/scratch/esm/slurm-buildkite/climaocean-ci/5288/climaocean-ci/test/runtests_setup.jl:90
  Test threw exception
  Expression: begin
    #= /resnick/scratch/esm/slurm-buildkite/climaocean-ci/5288/climaocean-ci/test/runtests_setup.jl:91 =#
    for name = varnames
        #= /resnick/scratch/esm/slurm-buildkite/climaocean-ci/5288/climaocean-ci/test/runtests_setup.jl:92 =#
        set!(field, Metadatum(name; dataset, date = start_date); inpainting)
        #= /resnick/scratch/esm/slurm-buildkite/climaocean-ci/5288/climaocean-ci/test/runtests_setup.jl:93 =#
    end
    #= /resnick/scratch/esm/slurm-buildkite/climaocean-ci/5288/climaocean-ci/test/runtests_setup.jl:94 =#
    true
end
  EOFError: read end of file
  Stacktrace:
    [1] jldopen(fname::String, wr::Bool, create::Bool, truncate::Bool, iotype::Type{JLD2.MmapIO}; fallback::Type{IOStream}, compress::Bool, mmaparrays::Bool, typemap::Function, parallel_read::Bool, plain::Bool)
      @ JLD2 /resnick/scratch/esm/slurm-buildkite/climaocean-ci/depot/default/packages/JLD2/hbsZG/src/JLD2.jl:242
    [2] jldopen
      @ /resnick/scratch/esm/slurm-buildkite/climaocean-ci/depot/default/packages/JLD2/hbsZG/src/JLD2.jl:174 [inlined]
    [3] #jldopen#23
      @ /resnick/scratch/esm/slurm-buildkite/climaocean-ci/depot/default/packages/JLD2/hbsZG/src/JLD2.jl:297 [inlined]
    [4] jldopen
      @ /resnick/scratch/esm/slurm-buildkite/climaocean-ci/depot/default/packages/JLD2/hbsZG/src/JLD2.jl:290 [inlined]
    [5] Field(metadata::Metadata{ECCO4Monthly, DateTime, Nothing}, arch::CPU; inpainting::NearestNeighborInpainting{Int64}, mask::Nothing, halo::Tuple{Int64, Int64, Int64}, cache_inpainted_data::Bool)
      @ ClimaOcean.DataWrangling /resnick/scratch/esm/slurm-buildkite/climaocean-ci/5288/climaocean-ci/src/DataWrangling/metadata_field.jl:102
    [6] Field
      @ /resnick/scratch/esm/slurm-buildkite/climaocean-ci/5288/climaocean-ci/src/DataWrangling/metadata_field.jl:87 [inlined]

This error makes me think that there are corrupted jld2 files. It might be that both CPU and GPU are trying to download and inpaint data and they conflict with each other. We should specify different donwload directories for the CPU and GPU tests I think.

francispoulin · 2025-12-09T12:55:38Z

Thanks @simone-silvestri for the suggestion. I'm giving that a try now for this one test. If it passes, it might be useful to do it for all the tests I suppose. We will see.

francispoulin · 2025-12-09T14:04:17Z

I tried making the suggested change. Maybe someone and double check what I changed to see if it makes sense to you?

With these changes the tests still pass on my server, so it might be working correctly.

However, we are still getting an error with the same function but now it says we are using too much memory. Using 4.078 KiB does not seem like a lot to me. Is this a limit we can change to faciliate testing?

  Test threw exception
  Expression: begin
    #= /resnick/scratch/esm/slurm-buildkite/climaocean-ci/5312/climaocean-ci/test/runtests_setup.jl:225 =#
    time_step!(ocean)
    #= /resnick/scratch/esm/slurm-buildkite/climaocean-ci/5312/climaocean-ci/test/runtests_setup.jl:226 =#
    time_step!(ocean)
    #= /resnick/scratch/esm/slurm-buildkite/climaocean-ci/5312/climaocean-ci/test/runtests_setup.jl:227 =#
    true
end
  Kernel invocation uses too much parameter memory.
  4.078 KiB exceeds the 4.000 KiB limit imposed by sm_60 / PTX v8.2.

francispoulin · 2025-12-16T18:01:47Z

Any idea why we get this error when running distributed tests?

🚨 Error: The command exited with status 1

Also, can anyone see the error from the docs? I get a login page but can't login. (@cmbengue ?)

francispoulin · 2025-12-17T12:09:11Z

Thanks @navidcy for the update!

Things are better in that now the failed test passes. Yay!

I still can't see the error with building of the example. Can anyone else see this?

I also see this error with the downloading. I wonder if this is a fluke and might go away on future attemps?

ERROR: LoadError: The following 2 direct dependencies failed to precompile:

ClimaOceanSpeedyWeatherExt --code-coverage=@D:\a\ClimaOcean.jl\ClimaOcean.jl --color=yes --check-bounds=yes --warn-overwrite=yes --depwarn=yes --inline=yes --startup-file=no --track-allocation=none --check-bounds=yes --compiled-modules=yes --depwarn=yes 

Failed to precompile ClimaOceanSpeedyWeatherExt [6367661e-0033-54c4-8969-7c7b61ae9771] to "C:\\Users\\runneradmin\\.julia\\compiled\\v1.12\\ClimaOceanSpeedyWeatherExt\\jl_FFB8.tmp".
ERROR: LoadError: InitError: XESMF.jl requires the ESMF library to be installed.
This is usually handled automatically by CondaPkg, but on some systems
(particularly Windows) it may need to be installed manually.

Try running:
julia -e "using CondaPkg; CondaPkg.add(["esmf", "esmpy"])"

cmbengue · 2025-12-18T20:12:25Z

Hi, @glwagner and @francispoulin, I don't know why you see the login page for some tests on Buildkite, as I neither actively manage nor have admin access to Buildkite.

francispoulin · 2025-12-18T20:34:23Z

Thanks for your reply and sorry to have bothered you with this.

creating new example for the ACC

aeae520

Update acc_regional_simulation.jl

705f078

Update acc_regional_simulation.jl

bc31849

Make changes so that it runs

francispoulin mentioned this pull request Aug 23, 2024

Example that uses restoring to the ECCO solution on a tripolar grid? #159

Open

simone-silvestri and others added 5 commits December 10, 2024 19:01

Merge branch 'main' into fjp/acc_regional_model

67c3eb7

Merge branch 'main' into fjp/acc_regional_model

06ab5d4

updated script

7741ea7

add different masking

a398d0c

updating script to get further

e377edf

Update panantarctic_regional_simulation.jl

0bb4542

Let's try a coarser grid to see if this allows us to avoid the memory errors.

francispoulin and others added 4 commits September 16, 2025 17:48

Update panantarctic_regional_simulation.jl

3aa53ac

Returning to the desired resolution since more errors happened with a coarser grid. Hmm...

Merge branch 'main' into fjp/acc_regional_model

5a69b1c

Merge branch 'main' into fjp/acc_regional_model

8e4c837

Merge branch 'main' into fjp/acc_regional_model

db0fd76

navidcy added 3 commits December 9, 2025 05:12

ExponentialCoordinate -> ExponentialDiscretization

c09ea18

no need to spinup with RK3

994a0b4

Merge branch 'main' into fjp/acc_regional_model

251d01e

saving data to different folder based on arch

02f59c3

Merge branch 'main' into fjp/acc_regional_model

4f58ca3

Merge branch 'main' into fjp/acc_regional_model

b7809c4

Merge branch 'main' into fjp/acc_regional_model

76d9933

Conversation

francispoulin commented Aug 15, 2024 • edited by navidcy Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

francispoulin commented Aug 15, 2024

Uh oh!

simone-silvestri commented Aug 15, 2024

Uh oh!

francispoulin commented Aug 15, 2024

Uh oh!

francispoulin commented Aug 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

francispoulin commented Aug 20, 2024

Uh oh!

simone-silvestri commented Aug 20, 2024

Uh oh!

francispoulin commented Aug 20, 2024

Uh oh!

francispoulin commented Aug 20, 2024

Uh oh!

simone-silvestri commented Aug 20, 2024

Uh oh!

codecov bot commented Aug 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

francispoulin commented Aug 20, 2024

Uh oh!

francispoulin commented Aug 22, 2024

Uh oh!

francispoulin commented Aug 27, 2024

Uh oh!

simone-silvestri commented Aug 27, 2024

Uh oh!

francispoulin commented Sep 23, 2024

Uh oh!

francispoulin commented Sep 23, 2024

Uh oh!

francispoulin commented Sep 23, 2024

Uh oh!

glwagner commented Sep 23, 2024

Uh oh!

francispoulin commented Sep 23, 2024

Uh oh!

francispoulin commented Dec 10, 2024

Uh oh!

simone-silvestri commented Dec 10, 2024

Uh oh!

francispoulin commented Sep 15, 2025

Uh oh!

francispoulin commented Sep 16, 2025

Uh oh!

glwagner commented Sep 16, 2025

Uh oh!

francispoulin commented Sep 16, 2025

Uh oh!

francispoulin commented Sep 16, 2025

Uh oh!

francispoulin commented Dec 8, 2025

Uh oh!

navidcy commented Dec 8, 2025

Uh oh!

francispoulin commented Dec 8, 2025

Uh oh!

francispoulin commented Dec 8, 2025

Uh oh!

francispoulin commented Dec 9, 2025

Uh oh!

glwagner commented Dec 9, 2025

Uh oh!

simone-silvestri commented Dec 9, 2025

Uh oh!

francispoulin commented Dec 9, 2025

Uh oh!

francispoulin commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

francispoulin commented Dec 16, 2025

Uh oh!

francispoulin commented Aug 15, 2024 •

edited by navidcy

Loading

francispoulin commented Aug 15, 2024 •

edited

Loading

codecov bot commented Aug 20, 2024 •

edited

Loading

francispoulin commented Dec 9, 2025 •

edited

Loading