- 
                Notifications
    
You must be signed in to change notification settings  - Fork 211
 
Commit ac783f6
committed
Move a lot of checkpointing code into Memoizer, from DFK
This should not change behaviour and only move code around and
add in more wiring.
This is working towards all of checkpointing/memoizer code being
plugable, with the DFK having no knowledge of the specifics - for
example, the DFK no longer imports `pickle`, because that is now the
business of the memoizer.
Because of that goal, the parameters for Memoizer are arranged
in two ways:
Constructor parameters for Memoizer are parameters that the future user
will supply at configuration when Memoizer is exposed as a user plugin,
like other plugins.
DFK internals that should be injected into the (eventually pluggable)
memoizer (small m) are set as startup-time attributes, as for other
plugins.
Right now these two things happen right next to each other, but the
Memoizer constructor call will move into user configuration space in
a subsequent PR.
As before this PR, there is still a separation between "checkpointing"
and "memoization" that is slightly artificial. Subsequent PRs will
merge these actions together more in this Memoizer implementation.
This PR preserves a user-facing dfk.checkpoint() call, that becomes a
passthrough to Memoizer.checkpoint().  I think that is likely to
disappear in a future PR: manual checkpoint triggering will become a
feature (or non-feature) of a specific memoizer implementation and so
a method directly on that memoizer (or not).
To go alongside the existing update_memo call, called by the DFK when
a task is ready for memoization, this PR adds update_checkpoint, which
captures the slightly different notion of a task being ready for
checkpointing -- the current implementation can memoize a task that is
not yet complete because it memoizes the future, not the result, while
a checkpoint needs the actual result to be ready and so must be called
later. This latter call happens in the race-condition-prone
handle_app_update. A later PR will remove this distinction and move
everything to around the same place as update_memo. See PR #3979 for
description of fixing a related race condition related to update_memo.
See PR #3535 for broader context.1 parent 6b5fa80 commit ac783f6Copy full SHA for ac783f6
File tree
Expand file treeCollapse file tree
0 file changed
+0
-0
lines changedOpen diff view settings
Filter options
Expand file treeCollapse file tree
0 file changed
+0
-0
lines changedOpen diff view settings
0 commit comments