Skip to content

MLP storage advice #145

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jun 10, 2025
Merged

MLP storage advice #145

Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
632e480
small change.
rubber-duck-debug Jun 6, 2025
db0070f
wording.
rubber-duck-debug Jun 6, 2025
756409f
better explanation
rubber-duck-debug Jun 6, 2025
1045448
formatting.
rubber-duck-debug Jun 6, 2025
da7d572
formatting
rubber-duck-debug Jun 6, 2025
fbf51b3
try expanding filesystem table...
rubber-duck-debug Jun 6, 2025
679cd72
typo.
rubber-duck-debug Jun 6, 2025
4a0dc09
typo.
rubber-duck-debug Jun 6, 2025
1578a3c
another typo.
rubber-duck-debug Jun 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions docs/platforms/mlp/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,9 @@ There are three main file systems mounted on the MLP clusters Clariden and Brist

| type |mount | filesystem |
| -- | -- | -- |
| Home | /users/$USER | [VAST][ref-alps-vast] |
| Home | `/users/$USER` | [VAST][ref-alps-vast] |
| Scratch | `/iopsstor/scratch/cscs/$USER` | [Iopsstor][ref-alps-iopsstor] |
| | `/capstor/scratch/cscs/$USER` | [Capstor][ref-alps-capstor] |
| Project | `/capstor/store/cscs/swissai/<project>` | [Capstor][ref-alps-capstor] |

### Home
Expand All @@ -51,14 +52,31 @@ Use scratch to store datasets that will be accessed by jobs, and for job output.
Scratch is per user - each user gets separate scratch path and quota.

* The environment variable `SCRATCH=/iopsstor/scratch/cscs/$USER` is set automatically when you log into the system, and can be used as a shortcut to access scratch.
* There is an additional scratch path mounted on [Capstor][ref-alps-capstor] at `/capstor/scratch/cscs/$USER`.

!!! warning "scratch cleanup policy"
Files that have not been accessed in 30 days are automatically deleted.

**Scratch is not intended for permanent storage**: transfer files back to the capstor project storage after job runs.

!!! note
There is an additional scratch path mounted on [Capstor][ref-alps-capstor] at `/capstor/scratch/cscs/$USER`, however this is not recommended for ML workloads for performance reasons.
!!! note "file system suitability"
The Capstor scratch filesystem is based on HDDs and is optimized for large, sequential read and write operations.
We recommend using Capstor for storing **checkpoint files** and other **large, contiguous outputs** generated by your training runs.
In contrast, Iopstor uses high-performance NVMe drives, which excel at handling **IOPS-intensive workloads** involving frequent, random access. This makes it a better choice for storing **training datasets**, especially when accessed randomly during machine learning training.

### Scratch Usage Recommendations

Use Iopstor scratch (`$SCRATCH`) for:

* Training and validation datasets that are read frequently and non-sequentially.
* Workloads that perform many small, random I/O operations.

Use Capstor scratch (`/capstor/scratch/cscs/$USER`) for:

* Storing model checkpoints.
* Outputs from simulations or training jobs that involve large, contiguous I/O.

After your job completes, remember to transfer any important results to your permanent project storage.

### Project

Expand Down