You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/platforms/mlp/index.md
+17-4Lines changed: 17 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -51,16 +51,29 @@ Use scratch to store datasets that will be accessed by jobs, and for job output.
51
51
Scratch is per user - each user gets separate scratch path and quota.
52
52
53
53
* The environment variable `SCRATCH=/iopsstor/scratch/cscs/$USER` is set automatically when you log into the system, and can be used as a shortcut to access scratch.
54
+
* There is an additional scratch path mounted on [Capstor][ref-alps-capstor] at `/capstor/scratch/cscs/$USER`.
54
55
55
56
!!! warning "scratch cleanup policy"
56
57
Files that have not been accessed in 30 days are automatically deleted.
57
58
58
59
**Scratch is not intended for permanent storage**: transfer files back to the capstor project storage after job runs.
59
60
60
-
!!! note
61
-
There is an additional scratch path mounted on [Capstor][ref-alps-capstor] at `/capstor/scratch/cscs/$USER`.
62
-
This filesystem should perform better for contiguous reads and writes.
63
-
Therefore, we recommend using capstor for storing checkpoint files generated by your training runs.
61
+
!!! note "file system suitability"
62
+
The Capstor scratch filesystem is based on HDDs and is optimized for large, sequential read and write operations.
63
+
We recommend using Capstor for storing **checkpoint files** and other **large, contiguous outputs** generated by your training runs.
64
+
In contrast, Iopstor uses high-performance NVMe drives, which excel at handling **IOPS-intensive workloads** involving frequent, random access. This makes it a better choice for storing **training datasets**, especially when accessed randomly during machine learning training.
65
+
66
+
### Scratch Usage Recommendations
67
+
68
+
Use Iopstor scratch (`$SCRATCH`) for:
69
+
* Training and validation datasets that are read frequently and non-sequentially.
70
+
* Workloads that perform many small, random I/O operations.
71
+
72
+
Use Capstor scratch (`/capstor/scratch/cscs/$USER`) for:
73
+
* Storing model checkpoints.
74
+
* Outputs from simulations or training jobs that involve large, contiguous I/O.
75
+
76
+
After your job completes, remember to transfer any important results to your permanent project storage.
0 commit comments