You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/guide/checkpoints.md
+15-7Lines changed: 15 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,10 +2,12 @@
2
2
3
3
Checkpoints allow DataChain to automatically skip re-creating datasets that were successfully saved in previous script runs. When a script fails or is interrupted, you can re-run it and DataChain will resume from where it left off, reusing datasets that were already created.
4
4
5
-
**Note:**Checkpoints are currently available only for local script runs. Support for Studio is planned for future releases.
5
+
Checkpoints are available for both local script runs and Studio executions.
6
6
7
7
## How Checkpoints Work
8
8
9
+
### Local Script Runs
10
+
9
11
When you run a Python script locally (e.g., `python my_script.py`), DataChain automatically:
10
12
11
13
1.**Creates a job** for the script execution, using the script's absolute path as the job name
@@ -16,6 +18,18 @@ When you run a Python script locally (e.g., `python my_script.py`), DataChain au
16
18
17
19
This means that if your script creates multiple datasets and fails partway through, the next run will skip recreating the datasets that were already successfully saved.
18
20
21
+
### Studio Runs
22
+
23
+
When running jobs on Studio, the checkpoint workflow is managed through the UI:
24
+
25
+
1.**Job execution** is triggered using the Run button in the Studio interface
26
+
2.**Checkpoint control** is explicit - you choose between:
27
+
-**Run from scratch**: Ignores any existing checkpoints and recreates all datasets
28
+
-**Continue from last checkpoint**: Resumes from the last successful checkpoint, skipping already-completed stages
29
+
3.**Parent-child job linking** is handled automatically by the system - no need for script path matching or job name conventions
30
+
4.**Checkpoint behavior** during execution is the same as local runs: datasets are saved at each `.save()` call and can be reused on retry
31
+
32
+
19
33
## Example
20
34
21
35
Consider this script that processes data in multiple stages:
@@ -66,7 +80,6 @@ Checkpoints are **not** used when:
- Running code as a module (e.g., `python -m mymodule`)
68
82
- The `DATACHAIN_CHECKPOINTS_RESET` environment variable is set (see below)
69
-
- Running on Studio (checkpoints support planned for future releases)
70
83
71
84
## Resetting Checkpoints
72
85
@@ -176,17 +189,12 @@ for ds in dc.datasets():
176
189
177
190
## Limitations
178
191
179
-
-**Local only:** Checkpoints currently work only for local script runs. Studio support is planned.
180
192
-**Script-based:** Code must be run as a script (not interactively or as a module).
181
193
-**Hash-based matching:** Any change to the chain will create a different hash, preventing checkpoint reuse.
182
194
-**Same script path:** The script must be run from the same absolute path for parent job linking to work.
183
195
184
196
## Future Plans
185
197
186
-
### Studio Support
187
-
188
-
Support for checkpoints on Studio is planned for future releases, which will enable checkpoint functionality for collaborative workflows and cloud-based data processing.
189
-
190
198
### UDF-Level Checkpoints
191
199
192
200
Currently, checkpoints are created only when datasets are saved using `.save()`. This means that if a script fails during a long-running UDF operation (like `.map()`, `.gen()`, or `.agg()`), the entire UDF computation must be rerun on the next execution.
0 commit comments