Skip to content

Commit a797dd8

Browse files
committed
reset checkpoint task list after checkpointing all tasks, in memoizer
this was done in one occurence in the DFK. in the other occurence, this happened at cleanup when no further checkpointing would happen in the relevant modes - so this change should not change user-facing behaviour
1 parent 4856e98 commit a797dd8

File tree

2 files changed

+7
-1
lines changed

2 files changed

+7
-1
lines changed

parsl/dataflow/dflow.py

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1270,7 +1270,6 @@ def cleanup(self) -> None:
12701270
def checkpoint(self) -> None:
12711271
with self._modify_checkpointable_tasks_lock:
12721272
self.memoizer.checkpoint()
1273-
self.memoizer.checkpointable_tasks = []
12741273

12751274
@staticmethod
12761275
def _log_std_streams(task_record: TaskRecord) -> None:

parsl/dataflow/memoization.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -411,3 +411,10 @@ def checkpoint(self, *, task: Optional[TaskRecord] = None) -> None:
411411
logger.debug("No tasks checkpointed in this pass.")
412412
else:
413413
logger.info("Done checkpointing {} tasks".format(count))
414+
415+
# TODO: test that tasks are not repeatedly checkpointed.
416+
# TODO: race condition here, for tasks that were added while the list was being updated
417+
# That's maybe a race condition in checkpointing in master at time of writing, and I could write a test for that?
418+
# Or maybe a failure of iteration if the list is appended to while checkpointing is happening?
419+
if not task:
420+
self.checkpointable_tasks = []

0 commit comments

Comments
 (0)