Skip to content

Commit 81ed087

Browse files
authored
Add lfs migrate examples to storage guide (#297)
This adds the `lfs migrate` recommendation/example that was removed along with the Gordon-Bell guide (#240). This also adds an example to recursively update all files in a directory (finding files using `lfs find`). @mpasserini can you or someone else from the storage team comment if you think these are ok recommendations. For example: - (this already exists) do you think the recommended block sizes and progressive layouts are sane recommendations? - Are there any concerns with using e.g. `xargs -P` to migrate files in parallel (the example is currently sequential which can be slow, but I'm not sure if doing a parallel `lfs migrate` adds unnecessarily high load on lustre)? Note that I changed the short options (`-c`/`-S`/`-E`) to long options as I find that it's easier to follow for someone like me who doesn't know the options by heart. However, the commands do end up being longer (wider than the regular page, and require scrolling), and I'm happy to change that back (it's not the important part of this PR). I'm not attempting to move or restructure the storage guide otherwise. Link to generated docs: https://docs.tds.cscs.ch/297/guides/storage/#lustre-tuning
1 parent b8f83cc commit 81ed087

File tree

1 file changed

+23
-6
lines changed

1 file changed

+23
-6
lines changed

docs/guides/storage.md

Lines changed: 23 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -136,19 +136,36 @@ The simplest way to have the correct layout is to copy to a directory with the c
136136
... so it is a good choice when reading a file sequentially or in large chunks, but if one reads shorter chunks in random order it might be better to reduce the size, the performance will be smaller, but the performance of your application might actually increase.
137137
See the [Lustre documentation](https://doc.lustre.org/lustre_manual.xhtml#managingstripingfreespace) for more information.
138138

139-
140139
!!! example "Settings for large files"
141-
*Remember:* Settings only apply to files added to the directory after this command.
142-
```console
143-
lfs setstripe --stripe-count -1 --stripe-size 4M <big_files_dir>`
140+
```bash
141+
lfs setstripe --stripe-count -1 --stripe-size 4M <big_files_dir>
144142
```
143+
144+
*Remember:* Settings applied with `lfs setstripe` only apply to files added to the directory after this command.
145+
[Use `lfs migrate` to update the settings for existing files][ref-guides-storage-examples-lfs-migrate].
146+
145147
Lustre also supports composite layouts, switching from one layout to another at a given size `--component-end` (`-E`).
146148
With it it is possible to create a Progressive file layout switching `--stripe-count` (`-c`), `--stripe-size` (`-S`), so that fewer locks are required for smaller files, but load is distributed for larger files.
147149

148150
!!! example "Good default settings"
149-
```console
150-
lfs setstripe -E 4M -c 1 -E 64M -c 4 -E -1 -c -1 -S 4M <base_dir>
151+
```bash
152+
lfs setstripe --component-end 4M --stripe-count 1 --component-end 64M --stripe-count 4 --component-end -1 --stripe-count -1 --stripe-size 4M <base_dir>
153+
```
154+
155+
[](){#ref-guides-storage-examples-lfs-migrate}
156+
!!! example "Updating settings for existing files"
157+
While `lfs setstripe` applies to newly created files, `lfs migrate` can be used to re-layout existing files.
158+
For example, to set the recommended settings above on an existing file:
159+
```bash
160+
lfs migrate --component-end 4M --stripe-count 1 --component-end 64M --stripe-count 4 --component-end -1 --stripe-count -1 --stripe-size 4M <file>
161+
```
162+
163+
Alternatively, to migrate all files recursively in a directory:
164+
```bash
165+
lfs find --type file <base_dir> | xargs lfs migrate --verbose --component-end 4M --stripe-count 1 --component-end 64M --stripe-count 4 --component-end -1 --stripe-count -1 --stripe-size 4M
151166
```
167+
The `--verbose` flag makes `lfs migrate` print the path of each file after the file has been migrated.
168+
Also note the use of `lfs find` instead of regular `find` as `lfs` can more efficiently retrieve the list of files recursively.
152169

153170
### Iopsstor vs Capstor
154171

0 commit comments

Comments
 (0)