Skip to content

Commit ced83b5

Browse files
committed
Some further enhancements and fixes
1 parent 9a1ce86 commit ced83b5

File tree

3 files changed

+481
-622
lines changed

3 files changed

+481
-622
lines changed

docs/src/private_api_reference.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@ Peridynamics.apply_precracks!
5757
Peridynamics.apply_precrack!
5858
Peridynamics.point_sets_intersect
5959
Peridynamics.invreg
60+
Peridynamics.update_sim_success_from_log!
6061
```
6162

6263
## Macros

src/core/study.jl

Lines changed: 36 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,13 @@ A structure for managing parameter studies with multiple peridynamic simulations
55
`Study` type coordinates the execution of multiple simulation jobs with different parameter
66
configurations, tracks their status, and logs all results to a central logfile.
77
8+
If a logfile already exists at the study root (from a previous interrupted run), the
9+
constructor automatically reads it and initializes the `sim_success` field based on the
10+
recorded completion status. This enables seamless resumption of interrupted studies.
11+
812
# Arguments
9-
- `jobcreator::Function`: A function with signature `jobcreator(setup::NamedTuple, root::String)`
13+
- `jobcreator::Function`: A function with signature
14+
`jobcreator(setup::NamedTuple, root::String)`
1015
that creates and returns a [`Job`](@ref) object. The function receives:
1116
- `setup`: A `NamedTuple` containing the parameters for one simulation
1217
- `root`: The root directory path for the study (to construct individual job paths)
@@ -137,6 +142,18 @@ This function creates the study root directory and logfile, then submits each jo
137142
respective job directories, while a central study logfile tracks the overall progress and
138143
status of all simulations.
139144
145+
## Resuming Interrupted Runs
146+
147+
If the study logfile already exists (from a previous run), `submit!` will:
148+
- Read the logfile and refresh `study.sim_success` to detect previously completed jobs
149+
- Skip jobs that already completed successfully (marked with "skipped" in the logfile)
150+
- Only execute jobs that failed or were not yet started
151+
- Append a "RESUMED" marker with timestamp to the logfile
152+
153+
This allows you to safely resume a study after an interruption (crash, timeout, manual
154+
stop) without re-running successful simulations. Simply call `submit!(study)` again with
155+
the same study configuration.
156+
140157
# Arguments
141158
- `study::Study`: The study containing all jobs to execute
142159
@@ -160,6 +177,11 @@ status of all simulations.
160177
study = Study(create_job, setups; root="my_study")
161178
submit!(study; quiet=true)
162179
180+
# If the process was interrupted, simply call submit! again to resume:
181+
# It will skip completed jobs and only run remaining ones
182+
study = Study(create_job, setups; root="my_study") # Reads existing logfile
183+
submit!(study; quiet=true) # Resumes from where it left off
184+
163185
# Check which simulations succeeded
164186
successful_indices = findall(study.sim_success)
165187
println("Successful simulations: ", successful_indices)
@@ -231,9 +253,9 @@ function submit!(study::Study; kwargs...)
231253
study.sim_success[i] = success
232254
open(study.logfile, "a") do io
233255
if success
234-
msg *= @sprintf(" status: completed ✓ (%.2f seconds)\n\n", simtime)
256+
msg = @sprintf(" status: completed ✓ (%.2f seconds)\n\n", simtime)
235257
else
236-
msg *= " status: failed ✗\n\n"
258+
msg = " status: failed ✗\n\n"
237259
end
238260
write(io, msg)
239261
end
@@ -308,6 +330,11 @@ This function iterates through all jobs in the study and applies the user-define
308330
processing function `f` to jobs that completed successfully. Failed jobs or jobs where
309331
the processing function errors will use the `default_result` instead.
310332
333+
Before processing, this function automatically refreshes `study.sim_success` from the
334+
logfile if it exists. This ensures that jobs completed in a previous interrupted run
335+
are properly detected and processed, even if the current Julia session doesn't reflect
336+
their completion status yet.
337+
311338
# Arguments
312339
- `f::Function`: A processing function with signature `f(job::Job, setup::NamedTuple)`
313340
that returns a `NamedTuple` containing the processed results. The function receives:
@@ -323,17 +350,19 @@ the processing function errors will use the `default_result` instead.
323350
`default_result` for failed/errored cases.
324351
325352
# Behavior
353+
- Refreshes job completion status from logfile (if exists) before processing
326354
- Only processes jobs where `study.sim_success[i] == true`
327355
- Failed jobs automatically receive `default_result` (warning logged)
328-
- If processing function `f` throws an error, that job receives `default_result` (error logged)
356+
- If processing function `f` throws an error, that job receives `default_result` (error
357+
logged)
329358
- Processing is sequential (one job at a time)
330359
- All jobs in the study will have a corresponding entry in the results vector
331360
332361
# Example
333362
```julia
334-
# After running a parameter study
363+
# After running a parameter study (possibly interrupted and resumed)
335364
study = Study(create_job, setups; root="my_study")
336-
submit!(study)
365+
# Status is automatically refreshed from logfile in constructor
337366
338367
# Define a function to extract maximum displacement from results
339368
function extract_max_displacement(job::Job, setup::NamedTuple)
@@ -343,6 +372,7 @@ function extract_max_displacement(job::Job, setup::NamedTuple)
343372
end
344373
345374
default = (; E=0.0, velocity=0.0, max_displacement=NaN)
375+
# This will process all successfully completed jobs, even if from a previous run
346376
results = process_each_job(extract_max_displacement, study, default)
347377
348378
# Filter successful results

0 commit comments

Comments
 (0)