Restarting from a checkpoint with function boundary conditions

[sam12396]

Hi all!
First off, I have been loving the package and how easy it is to build quick tests to try new ideas.

The current problem I am having is with trying to restart a run from a checkpoint where the run uses boundary conditions defined by functions. The documentation warns that there are some complications surrounding this but I was wondering if anyone had any experience.

I have rebuilt the boundary conditions as I did in the initial set up of the model and then when I try to run the line below,

model = restore_from_checkpoint("rst_iteration187400.jld2",boundary_conditions = (u=u_bcs, T=T_bcs, S=S_bcs, v=v_bcs))

the following error is returned

MethodError: no method matching extract_boundary_conditions(::JLD2.Group{JLD2.JLDFile{JLD2.MmapIO}})

and the stack trace ultimately leads to the function 'extract_boundary_conditions' in src/Models/IncompressibleModels/incompressible_model.jl

Am I using the wrong keyword arguments for restore_from_checkpoint?
I am new to Oceananigans and Julia and appreciate any support. Thanks!

[ali-ramadhan]

Hi @sam12396! Glad you're finding the package easy to use!

Apologies for the outdated documentation. Checkpointing has been recently refactored (and improved) in PR #1082.

So you don't want to use restore_from_checkpoint anymore (we should remove this function). Instead, you pickup via the pickup keyword argument passed to run!.

If you've already added a Checkpointer to simulation.output_writers you can just do run!(simulation, pickup=true) and it'll continue running the simulation from the latest checkpoint. You can also pass a filepath or an iteration number to restore from a specific checkpoint. More details: https://clima.github.io/OceananigansDocumentation/stable/library/#Oceananigans.Simulations.run!-Tuple{Any}

This way you don't have to worry about restoring boundary conditions!

[ali-ramadhan]

Ah sorry I think this error is related to #1280

I assume you have some output writers with time interval schedules? Essentially the problem is that the model tries to take zero or negative time steps leading to blow up lol.

I have a quick fix that I'll try to include as part of v0.46.0 to be released hopefully later today or tomorrow.

Until then, I have a fix for it on the ali/no-negative-dt branch (PR #1303) which you can use in Julia with ] add Oceananigans#ali/no-negative-dt in REPL pkg mode (press ] at the REPL).

---

But to answer your questions:

1. The iteration number is larger than the checkpoint iteration since the model takes some time steps before checking for NaNs. So if you checkpointed at iteration 3500, it might take a negative time step (the bug) at iteration 3501 causing the model to blow up in NaNs, then it checks for NaNs at iteration 3600. It checks for NaNs every simulation.iteration_interval iterations.
2. The units of time is the same as model.clock.time. Usually SI units (seconds) by default but could be non-dimensional if you're running a non-dimensional model.

[sam12396]

No problem! Thanks for the info.

When the model restarts, does it start counting from 0 seconds again or does it start counting from the last time value in the checkpoint file? I am asking because I have my forcing data set up as a time series starting at 0 and I need to know if the restarted model will pick back up where it left off in the forcing (resume counting from last time step) or start at the beginning of the forcing (start counting from 0 again). I am sure I can figure this out on my own with some testing but I figured I would ask.

Thanks again! I will update the package tomorrow and test again. I am in no rush at the moment because my HPC nodes are down for maintenance.

[sam12396]

@ali-ramadhan
I updated my package to the branch you suggested and it did help with the restarting. However, a new time step issue arose that may be related.
At 14 hours of model time, the model time step jumped back to 0 and restarted counting. This would be an easy enough fix to just change the time coordinates after the run but I fear that it has reset my forcing functions. I am using boundary conditions that are dependent on time and the functions were fit to observed data so restarting the time step in the middle of the event may be repeating forcing.
Do you know if resetting the time step mid-run would reset the forcing as well? Have you run into this issue before? Thanks!

[glwagner]

It looks like using the pickup kwarg should set the model clock time:

Oceananigans.jl/src/OutputWriters/checkpointer.jl

Line 277 in 02b0e71

model.clock.time = checkpointed_clock.time

But are you saying that model.clock.time arbitrarily jumped from 14 hours to 0 during a run!? I'm not sure why or how that would happen but it's certainly an issue. Do you mind sharing your script so we can figure out what went wrong, perhaps?

[sam12396]

@glwagner
I don't mind sharing at all!

The time field in the output file the model is writing jumps from 50400 seconds to 0 seconds. The log file (also attached) shows that the timesteps written out by the progress message don't show the same jump. I have also attached the output from the run. The model is still running, it has been going for a few days and I am hoping this issue is something I am misunderstanding about the output. I guess it could be that the outputter might be appending a netcdf file from a previous run but this is the first run I have used with that filename for output so I do not think that is the case. Also, I am working in Oceananigans#ali/no-negative-dt.

sam12396_time_issue.zip
soulik_gliderINI_nowaves_means.nc.zip

Thanks for the support!