JLD2Writer does not create per-rank files when filename contains .jld2 extension

[ruoyan88]

Behavior

• ✅ Works: filename = "output" → creates output_rank0.jld2, output_rank1.jld2, etc.
• ❌ Fails: filename = "output.jld2" → hangs (all ranks attempt to write to same file)

Reproducer

using MPI, Oceananigans
MPI.Init()

arch = Distributed(GPU(); partition = Partition(2, 2))
grid = RectilinearGrid(arch, size=(64, 64, 16), extent=(1, 1, 1))
model = NonhydrostaticModel(; grid)
simulation = Simulation(model, Δt=10, stop_iteration=100)

# This hangs:
simulation.output_writers[:out] = JLD2Writer(model, model.tracers,
    filename = "output.jld2",
    schedule = IterationInterval(10))

run!(simulation)

Test Results

Tested with filename = "output" (without extension):

• Grid: 1440×720×30 (0.25° Arctic Ocean)
• GPUs: 4× RTX 4090
• Partition: Partition(2, 2)
• Output interval: 6 hours

Results:

• ✅ All 4 ranks completed successfully
• ✅ Created output_rank0.jld2 through output_rank3.jld2
• ✅ Good I/O performance (~10ms write time after initialization)

---

Originally reported in ClimaOcean.jl: CliMA/ClimaOcean.jl#746
Credit to @glwagner for diagnosing the bug.

[navidcy]

Is the problem specific to distributed GPU? What about distributed CPU?

[glwagner]

its probably just in the constructor / extension handling.

[glwagner]

Oceananigans.jl/src/OutputWriters/jld2_writer.jl

Lines 178 to 179 in 0b02506

	filename = auto_extension(filename, ".jld2")
	filename = with_architecture_suffix(architecture(model), filename, ".jld2")

[navidcy]

When I add a @show in front of

Oceananigans.jl/src/OutputWriters/jld2_writer.jl

Lines 178 to 180 in 0b02506

	filename = auto_extension(filename, ".jld2")
	filename = with_architecture_suffix(architecture(model), filename, ".jld2")
	filepath = abspath(joinpath(dir, filename))

and use Distributed with 1 CPU things seems to be alright regardless whether I give the output filename with or without the extension...

julia> simulation.output_writers[:out] = JLD2Writer(model, model.tracers,
           filename = "output",
           schedule = IterationInterval(10));
filename = auto_extension(filename, ".jld2") = "output.jld2"
filename = with_architecture_suffix(architecture(model), filename, ".jld2") = "output_rank0.jld2"
filepath = abspath(joinpath(dir, filename)) = "/Oceananigans.jl/output_rank0.jld2"

julia> simulation.output_writers[:out] = JLD2Writer(model, model.tracers,
           filename = "output.jld2",
           schedule = IterationInterval(10));
filename = auto_extension(filename, ".jld2") = "output.jld2"
filename = with_architecture_suffix(architecture(model), filename, ".jld2") = "output_rank0.jld2"
filepath = abspath(joinpath(dir, filename)) = "/Oceananigans.jl/output_rank0.jld2"

[glwagner]

does the mwe fail for you?

[navidcy]

Actually, it doesn't on main.

And given the line

model = NonhydrostaticModel(; grid)

I'm wondering @ruoyan88 which Oceananigans version are you using. Try using the latest (which would require you to construct the model vie

model = NonhydrostaticModel(grid)

and then please confirm that the issue persists?

[navidcy]

On #main, when I save this:

using MPI, Oceananigans
MPI.Init()

arch = Distributed(CPU(); partition = Partition(2, 2))
grid = RectilinearGrid(arch, size=(32, 32, 16), extent=(1, 1, 1))
model = NonhydrostaticModel(grid)
simulation = Simulation(model, Δt=10, stop_iteration=100)

# This hangs:
simulation.output_writers[:out] = JLD2Writer(model, model.tracers,
    filename = "output.jld2",
    schedule = IterationInterval(10))

@show simulation

run!(simulation)

under mwe_output.jl and then run via

$ mpirun -np 4 julia --color=yes --project mwe_output.jl

simulation = Simulation of NonhydrostaticModel{Distributed{CPU} on 2×2×1, RectilinearGrid}(time = 0 seconds, iteration = 0)
├── Next time step: 10 seconds
├── run_wall_time: 0 seconds
├── run_wall_time / iteration: NaN days
├── stop_time: Inf days
├── stop_iteration: 100.0
├── wall_time_limit: Inf
├── minimum_relative_step: 0.0
├── callbacks: OrderedDict with 4 entries:
│   ├── stop_time_exceeded => Callback of stop_time_exceeded on IterationInterval(1)
│   ├── stop_iteration_exceeded => Callback of stop_iteration_exceeded on IterationInterval(1)
│   ├── wall_time_limit_exceeded => Callback of wall_time_limit_exceeded on IterationInterval(1)
│   └── nan_checker => Callback of NaNChecker for u on IterationInterval(100)
└── output_writers: OrderedDict with 1 entry:
│   └── out => JLD2Writer writing () to /Oceananigans.jl/output_rank1.jld2 on IterationInterval(10)
simulation = simulation = Simulation of NonhydrostaticModel{Distributed{CPU} on 2×2×1, RectilinearGrid}(time = 0 seconds, iteration = 0)
├── Next time step: 10 seconds
├── run_wall_time: 0 seconds
├── run_wall_time / iteration: NaN days
├── stop_time: Inf days
├── stop_iteration: 100.0
├── wall_time_limit: Inf
├── minimum_relative_step: 0.0
├── callbacks: OrderedDict with 4 entries:
│   ├── stop_time_exceeded => Callback of stop_time_exceeded on IterationInterval(1)
│   ├── stop_iteration_exceeded => Callback of stop_iteration_exceeded on IterationInterval(1)
│   ├── wall_time_limit_exceeded => Callback of wall_time_limit_exceeded on IterationInterval(1)
│   └── nan_checker => Callback of NaNChecker for u on IterationInterval(100)
└── output_writers: OrderedDict with 1 entry:
│   └── out => JLD2Writer writing () to /Oceananigans.jl/output_rank3.jld2 on IterationInterval(10)
Simulation of NonhydrostaticModel{Distributed{CPU} on 2×2×1, RectilinearGrid}(time = 0 seconds, iteration = 0)
├── Next time step: 10 seconds
├── run_wall_time: 0 seconds
├── run_wall_time / iteration: NaN days
├── stop_time: Inf days
├── stop_iteration: 100.0
├── wall_time_limit: Inf
├── minimum_relative_step: 0.0
├── callbacks: OrderedDict with 4 entries:
│   ├── stop_time_exceeded => Callback of stop_time_exceeded on IterationInterval(1)
│   ├── stop_iteration_exceeded => Callback of stop_iteration_exceeded on IterationInterval(1)
│   ├── wall_time_limit_exceeded => Callback of wall_time_limit_exceeded on IterationInterval(1)
│   └── nan_checker => Callback of NaNChecker for u on IterationInterval(100)
└── output_writers: OrderedDict with 1 entry:
│   └── out => JLD2Writer writing () to /Oceananigans.jl/output_rank0.jld2 on IterationInterval(10)
simulation = Simulation of NonhydrostaticModel{Distributed{CPU} on 2×2×1, RectilinearGrid}(time = 0 seconds, iteration = 0)
├── Next time step: 10 seconds
├── run_wall_time: 0 seconds
├── run_wall_time / iteration: NaN days
├── stop_time: Inf days
├── stop_iteration: 100.0
├── wall_time_limit: Inf
├── minimum_relative_step: 0.0
├── callbacks: OrderedDict with 4 entries:
│   ├── stop_time_exceeded => Callback of stop_time_exceeded on IterationInterval(1)
│   ├── stop_iteration_exceeded => Callback of stop_iteration_exceeded on IterationInterval(1)
│   ├── wall_time_limit_exceeded => Callback of wall_time_limit_exceeded on IterationInterval(1)
│   └── nan_checker => Callback of NaNChecker for u on IterationInterval(100)
└── output_writers: OrderedDict with 1 entry:
│   └── out => JLD2Writer writing () to /Oceananigans.jl/output_rank2.jld2 on IterationInterval(10)
[ Info: Initializing simulation...
[ Info: Initializing simulation...
[ Info: Initializing simulation...
[ Info: Initializing simulation...
[ Info:     ... simulation initialization complete (2.276 seconds)
[ Info:     ... simulation initialization complete (2.272 seconds)
[ Info:     ... simulation initialization complete (2.277 seconds)
[ Info: Executing initial time step...
[ Info: Executing initial time step...
[ Info: Executing initial time step...
[ Info:     ... simulation initialization complete (2.323 seconds)
[ Info: Executing initial time step...
[ Info:     ... initial time step complete (4.237 seconds).
[ Info:     ... initial time step complete (4.237 seconds).
[ Info:     ... initial time step complete (4.206 seconds).
[ Info:     ... initial time step complete (4.238 seconds).
[ Info: Simulation is stopping after running for 7.442 seconds.
[ Info: Simulation is stopping after running for 7.442 seconds.
[ Info: Simulation is stopping after running for 7.458 seconds.
[ Info: Simulation is stopping after running for 7.439 seconds.
[ Info: Model iteration 100 equals or exceeds stop iteration 100.
[ Info: Model iteration 100 equals or exceeds stop iteration 100.
[ Info: Model iteration 100 equals or exceeds stop iteration 100.
[ Info: Model iteration 100 equals or exceeds stop iteration 100.

all seems OK?

[ruoyan88]

Thanks @navidcy for testing! I'm using Oceananigans v0.104.2 (not main). I don't have time to test right now, but will try with the latest version when I get a chance and report back.

[navidcy]

If you are using v0.104.2 then I'd expect your MWE above to fail at the model = NonhydrostaticModel(; grid) line.

[navidcy]

I confirm that my example above works with v0.104.2

(base) $ git checkout v0.104.2                                                                                                                                                                                                                        
Note: switching to 'v0.104.2'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 409dc8ce1 (0.104.2) Add convenience method for 1D interpolation (#5182)
(base) $ mpirun -np 4 julia --color=yes --project mwe_output.jl         
PrecompilingPrecompiling packages...
 packages...
PrecompilingPrecompiling  packages...packages...

    Oceananigans Being precompiled by another process (pid: 56503, pidfile: /Users/navid/.julia/compiled/v1.12/Oceananigans/hU93i_nRxWE.ji.pidfile)
    Oceananigans Being precompiled by another process (pid: 56503, pidfile: /Users/navid/.julia/compiled/v1.12/Oceananigans/hU93i_nRxWE.ji.pidfile)
    Oceananigans Being precompiled by another process (pid: 56503, pidfile: /Users/navid/.julia/compiled/v1.12/Oceananigans/hU93i_nRxWE.ji.pidfile)
  10384.7 ms  ✓ Oceananigans
  1 dependency successfully precompiled in 11 seconds. 113 already precompiled.
  10518.6 ms  ✓ Oceananigans
  1 dependency successfully precompiled in 11 seconds. 113 already precompiled.
  10634.2 ms  ✓ Oceananigans
  1 dependency successfully precompiled in 11 seconds. 113 already precompiled.
  10761.7 ms  ✓ Oceananigans
  1 dependency successfully precompiled in 11 seconds. 113 already precompiled.
simulation = Simulation of NonhydrostaticModel{Distributed{CPU}, RectilinearGrid}(time = 0 seconds, iteration = 0)
├── Next time step: 10 seconds
├── run_wall_time: 0 seconds
├── run_wall_time / iteration: NaN days
├── stop_time: Inf days
├── stop_iteration: 100.0
├── wall_time_limit: Inf
├── minimum_relative_step: 0.0
├── callbacks: OrderedDict with 4 entries:
│   ├── stop_time_exceeded => Callback of stop_time_exceeded on IterationInterval(1)
│   ├── stop_iteration_exceeded => Callback of stop_iteration_exceeded on IterationInterval(1)
│   ├── wall_time_limit_exceeded => Callback of wall_time_limit_exceeded on IterationInterval(1)
│   └── nan_checker => Callback of NaNChecker for u on IterationInterval(100)
└── output_writers: OrderedDict with 1 entry:
│   └── out => JLD2Writer writing () to /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank2.jld2 on IterationInterval(10)
simulation = simulation = Simulation of NonhydrostaticModel{Distributed{CPU}, RectilinearGrid}(time = 0 seconds, iteration = 0)
├── Next time step: 10 seconds
├── run_wall_time: 0 seconds
├── run_wall_time / iteration: NaN days
├── stop_time: Inf days
├── stop_iteration: 100.0
├── wall_time_limit: Inf
├── minimum_relative_step: 0.0
├── callbacks: OrderedDict with 4 entries:
│   ├── stop_time_exceeded => Callback of stop_time_exceeded on IterationInterval(1)
│   ├── stop_iteration_exceeded => Callback of stop_iteration_exceeded on IterationInterval(1)
│   ├── wall_time_limit_exceeded => Callback of wall_time_limit_exceeded on IterationInterval(1)
│   └── nan_checker => Callback of NaNChecker for u on IterationInterval(100)
└── output_writers: OrderedDict with 1 entry:
│   └── out => JLD2Writer writing () to /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank3.jld2 on IterationInterval(10)
Simulation of NonhydrostaticModel{Distributed{CPU}, RectilinearGrid}(time = 0 seconds, iteration = 0)
├── Next time step: 10 seconds
├── run_wall_time: 0 seconds
├── run_wall_time / iteration: NaN days
├── stop_time: Inf days
├── stop_iteration: 100.0
├── wall_time_limit: Inf
├── minimum_relative_step: 0.0
├── callbacks: OrderedDict with 4 entries:
│   ├── stop_time_exceeded => Callback of stop_time_exceeded on IterationInterval(1)
│   ├── stop_iteration_exceeded => Callback of stop_iteration_exceeded on IterationInterval(1)
│   ├── wall_time_limit_exceeded => Callback of wall_time_limit_exceeded on IterationInterval(1)
│   └── nan_checker => Callback of NaNChecker for u on IterationInterval(100)
└── output_writers: OrderedDict with 1 entry:
│   └── out => JLD2Writer writing () to /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank1.jld2 on IterationInterval(10)
simulation = Simulation of NonhydrostaticModel{Distributed{CPU}, RectilinearGrid}(time = 0 seconds, iteration = 0)
├── Next time step: 10 seconds
├── run_wall_time: 0 seconds
├── run_wall_time / iteration: NaN days
├── stop_time: Inf days
├── stop_iteration: 100.0
├── wall_time_limit: Inf
├── minimum_relative_step: 0.0
├── callbacks: OrderedDict with 4 entries:
│   ├── stop_time_exceeded => Callback of stop_time_exceeded on IterationInterval(1)
│   ├── stop_iteration_exceeded => Callback of stop_iteration_exceeded on IterationInterval(1)
│   ├── wall_time_limit_exceeded => Callback of wall_time_limit_exceeded on IterationInterval(1)
│   └── nan_checker => Callback of NaNChecker for u on IterationInterval(100)
└── output_writers: OrderedDict with 1 entry:
│   └── out => JLD2Writer writing () to /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank0.jld2 on IterationInterval(10)
[ Info: Initializing simulation...
[ Info: Initializing simulation...
[ Info: Initializing simulation...
[ Info: Initializing simulation...
┌ Warning: Failed to save and serialize [:grid, :coriolis, :buoyancy, :closure] in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank3.jld2 because ArgumentError: ArgumentError: a group or dataset named x is already present within this group
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:196
┌ Warning: Failed to save and serialize [:grid, :coriolis, :buoyancy, :closure] in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank2.jld2 because ArgumentError: ArgumentError: a group or dataset named x is already present within this group
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:196
┌ Warning: Failed to save and serialize [:grid, :coriolis, :buoyancy, :closure] in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank1.jld2 because ArgumentError: ArgumentError: a group or dataset named x is already present within this group
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:196
┌ Warning: Failed to save and serialize [:grid, :coriolis, :buoyancy, :closure] in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank0.jld2 because ArgumentError: ArgumentError: a group or dataset named x is already present within this group
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:196
┌ Warning: Iteration 0 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank2.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
[ Info:     ... simulation initialization complete (1.561 seconds)
┌ Warning: Iteration 0 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank1.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
[ Info:     ... simulation initialization complete (1.546 seconds)
┌ Warning: Iteration 0 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank0.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
[ Info:     ... simulation initialization complete (1.562 seconds)
[ Info: Executing initial time step...
[ Info: Executing initial time step...
[ Info: Executing initial time step...
┌ Warning: Iteration 0 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank3.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
[ Info:     ... simulation initialization complete (1.597 seconds)
[ Info: Executing initial time step...
[ Info:     ... initial time step complete (4.323 seconds).
[ Info:     ... initial time step complete (4.319 seconds).
[ Info:     ... initial time step complete (4.296 seconds).
[ Info:     ... initial time step complete (4.322 seconds).
┌ Warning: Iteration 10 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank3.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 10 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank0.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 10 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank2.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 10 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank1.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 20 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank2.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 20 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank0.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 20 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank1.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 20 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank3.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 30 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank3.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 30 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank1.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 30 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank2.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 30 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank0.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 40 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank0.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 40 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank2.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 40 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank3.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 40 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank1.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 50 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank1.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 50 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank0.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 50 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank2.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 50 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank3.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 60 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank1.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 60 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank3.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 60 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank0.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 60 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank2.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 70 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank1.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 70 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank3.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 70 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank2.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 70 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank0.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 80 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank0.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 80 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank1.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 80 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank3.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 80 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank2.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 90 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank0.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 90 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank3.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 90 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank1.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 90 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank2.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
[ Info: Simulation is stopping after running for 6.990 seconds.
[ Info: Simulation is stopping after running for 6.999 seconds.
[ Info: Simulation is stopping after running for 6.973 seconds.
[ Info: Simulation is stopping after running for 6.986 seconds.
[ Info: Model iteration 100 equals or exceeds stop iteration 100.
[ Info: Model iteration 100 equals or exceeds stop iteration 100.
[ Info: Model iteration 100 equals or exceeds stop iteration 100.
[ Info: Model iteration 100 equals or exceeds stop iteration 100.
┌ Warning: Iteration 100 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank2.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 100 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank0.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 100 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank3.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283
┌ Warning: Iteration 100 was found in /Users/navid/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/output_rank1.jld2. Skipping output writing (for now...)
└ @ Oceananigans.OutputWriters ~/Library/CloudStorage/OneDrive-TheUniversityofMelbourne/Documents/Research/Oceananigans.jl-v1/src/OutputWriters/jld2_writer.jl:283

[ruoyan88]

Sorry for the confusion — the MWE I posted here was simplified and may not accurately reproduce the issue. My actual code that hangs uses ClimaOcean's ocean_simulation with TripolarGrid on distributed GPU (see CliMA/ClimaOcean.jl#746 (comment)). The hang disappears when I remove the .jld2 extension from the filename. I haven't been able to reproduce it with RectilinearGrid or on CPU — it may be specific to TripolarGrid + distributed GPU. Will test on main when I have access to the GPUs again.

[navidcy]

No worries. When you do confirm, then please also post the output of using Pkg; Pkg.status(); it'd be helpful to track down where the error comes from.