You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This adds the `lfs migrate` recommendation/example that was removed
along with the Gordon-Bell guide (#240).
This also adds an example to recursively update all files in a directory
(finding files using `lfs find`).
@mpasserini can you or someone else from the storage team comment if you
think these are ok recommendations. For example:
- (this already exists) do you think the recommended block sizes and
progressive layouts are sane recommendations?
- Are there any concerns with using e.g. `xargs -P` to migrate files in
parallel (the example is currently sequential which can be slow, but I'm
not sure if doing a parallel `lfs migrate` adds unnecessarily high load
on lustre)?
Note that I changed the short options (`-c`/`-S`/`-E`) to long options
as I find that it's easier to follow for someone like me who doesn't
know the options by heart. However, the commands do end up being longer
(wider than the regular page, and require scrolling), and I'm happy to
change that back (it's not the important part of this PR).
I'm not attempting to move or restructure the storage guide otherwise.
Link to generated docs:
https://docs.tds.cscs.ch/297/guides/storage/#lustre-tuning
Copy file name to clipboardExpand all lines: docs/guides/storage.md
+23-6Lines changed: 23 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -136,19 +136,36 @@ The simplest way to have the correct layout is to copy to a directory with the c
136
136
... so it is a good choice when reading a file sequentially or in large chunks, but if one reads shorter chunks in random order it might be better to reduce the size, the performance will be smaller, but the performance of your application might actually increase.
137
137
See the [Lustre documentation](https://doc.lustre.org/lustre_manual.xhtml#managingstripingfreespace) for more information.
138
138
139
-
140
139
!!! example "Settings for large files"
141
-
*Remember:* Settings only apply to files added to the directory after this command.
*Remember:* Settings applied with `lfs setstripe` only apply to files added to the directory after this command.
145
+
[Use `lfs migrate` to update the settings for existing files][ref-guides-storage-examples-lfs-migrate].
146
+
145
147
Lustre also supports composite layouts, switching from one layout to another at a given size `--component-end` (`-E`).
146
148
With it it is possible to create a Progressive file layout switching `--stripe-count` (`-c`), `--stripe-size` (`-S`), so that fewer locks are required for smaller files, but load is distributed for larger files.
0 commit comments