You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -51,14 +52,31 @@ Use scratch to store datasets that will be accessed by jobs, and for job output.
51
52
Scratch is per user - each user gets separate scratch path and quota.
52
53
53
54
* The environment variable `SCRATCH=/iopsstor/scratch/cscs/$USER` is set automatically when you log into the system, and can be used as a shortcut to access scratch.
55
+
* There is an additional scratch path mounted on [Capstor][ref-alps-capstor] at `/capstor/scratch/cscs/$USER`.
54
56
55
57
!!! warning "scratch cleanup policy"
56
58
Files that have not been accessed in 30 days are automatically deleted.
57
59
58
60
**Scratch is not intended for permanent storage**: transfer files back to the capstor project storage after job runs.
59
61
60
-
!!! note
61
-
There is an additional scratch path mounted on [Capstor][ref-alps-capstor] at `/capstor/scratch/cscs/$USER`, however this is not recommended for ML workloads for performance reasons.
62
+
!!! note "file system suitability"
63
+
The Capstor scratch filesystem is based on HDDs and is optimized for large, sequential read and write operations.
64
+
We recommend using Capstor for storing **checkpoint files** and other **large, contiguous outputs** generated by your training runs.
65
+
In contrast, Iopstor uses high-performance NVMe drives, which excel at handling **IOPS-intensive workloads** involving frequent, random access. This makes it a better choice for storing **training datasets**, especially when accessed randomly during machine learning training.
66
+
67
+
### Scratch Usage Recommendations
68
+
69
+
Use Iopstor scratch (`$SCRATCH`) for:
70
+
71
+
* Training and validation datasets that are read frequently and non-sequentially.
72
+
* Workloads that perform many small, random I/O operations.
73
+
74
+
Use Capstor scratch (`/capstor/scratch/cscs/$USER`) for:
75
+
76
+
* Storing model checkpoints.
77
+
* Outputs from simulations or training jobs that involve large, contiguous I/O.
78
+
79
+
After your job completes, remember to transfer any important results to your permanent project storage.
0 commit comments