Merge pull request #791 from AaltoSciComp/rkdarst/lustre

triton/usage/lustre: Major update

Author: simo-tuomisto

triton/usage/lustre.rst

@@ -1,4 +1,3 @@
-=========================
 Storage: Lustre (scratch)
 =========================
 
@@ -9,42 +8,40 @@ Storage: Lustre (scratch)
 Lustre is scalable high performance file system created for HPC. It
 allows MPI-IO but mainly it provides large storage capacity and high
 sequential throughput for cluster applications. Currently the total
-capacity is 2PB. The basic idea in Lustre is to spread data in each file
-over multiple storage servers. With large (larger than 1GB) files Lustre
-will significantly boost the performance.
-
-Working with small files
-~~~~~~~~~~~~~~~~~~~~~~~~
+capacity is 5PB.
 
-As Lustre is meant for large files, the performance with small (smaller
-than 10MB) files will not be optimal. If possible, try to avoid working
-with multiple small files.
+As you might expect, making a storage system capable of storing
+petabytes accessed at tens of gigabytes per second across hundreds of
+nodes and users simultaneously is quite a challenge.  It works well,
+but there are tradeoffs.  The basic idea in Lustre is to spread data
+in large files over multiple storage servers.  Small files can be a
+problem, but Triton's scratch is adjusted to mitigate it somewhat.
+With large (larger than 1GB) files Lustre will significantly boost the
+performance.
 
-**Note: Triton has a default stripe of 1 already, so it is by default
-optimized for small files (but it's still not that great).  If you use
-large files, see below.**
+.. important::
 
-If small files are needed (i.e. source codes) you can tell Lustre not to
-spread data over all the nodes. This will help in performance.
+   More often than not, when "Triton is down", it's Lustre (scratch)
+   being down.  If you do normal-sized work this usually isn't a
+   problem.  If you do big data-intensive work, please pay attention
+   to storage and ask for help early.
 
-To see the striping for any given file or directory you can use
-following command to check status
 
-::
-
-    lfs getstripe -d /scratch/path/to/dir
+Working with small files
+------------------------
 
-You can not change the striping of an existing file, but you can change
-the striping of new files created in a directory, then copy the file to
-a new name in that directory.
+As Lustre is meant for large files, the performance with small (smaller
+than 10MB) files will not be optimal. If possible, try to avoid working
+with large numbers of small files.  Large numbers is greater than
+thousands or tens of thousands.
 
-::
+.. seealso::
 
-    lfs setstripe -c 1 /scratch/path/to/dir
-    cp somefile /scratch/path/to/dir/newfile
+   * :doc:`smallfiles`, a dedicated page on handling small files.
+   * :doc:`localstorage`, a page explaining how to use compute node
+     local drives to unpack archives with many small files to get
+     better performance.
 
-Working with lots of small files
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Large datasets which consist mostly of small (<1MB) files can be slow to
 process because of network overhead associated with individual files. If
@@ -53,49 +50,32 @@ drives <localstorage>` page, see the ``tar`` example
 over there or find some other way to compact your files together into
 one.
 
-Working with large files
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-By default Lustre on Triton is configured to stripe a single file over a
-single OST. This provides the best performance for small files, serial
-programs, parallel programs where only one process is doing I/O, and
-parallel programs using a file-per-process file I/O pattern. However,
-when working with large files (>> 10 GB), particularly if they are
-accessed in parallel from multiple processes in a parallel application,
-it can be advantageous to stripe over several OST's.  In this case the
-easiest way is to create a directory for the large file(s), and set the
-striping parameters for any files subsequently created in that
-directory:
-
-::
-
-    cd $WRKDIR
-    mkdir large_file
-    lfs setstripe -c 4 large_file
 
-The above creates a directory ``large_file`` and specifies that files
-created inside that directory will be striped over 4 OST's. For really
-really large files (hundreds of GB's) accessed in parallel from very
-large MPI runs, set the stripe count to "-1" which tells the system to
-stripe over all the available OST's.
+Working with large files
+------------------------
 
-To reset back to the default settings, run
+By default Lustre on Triton is configured so that as files grow
+larger, they get `striped
+<https://en.wikipedia.org/wiki/Data_striping>`__ (split) over more
+storage servers.  This way, small files only require one server to
+serve the file (reducing latency), while large files can be streamed
+over multiple disks.
 
-::
+This page previously had instructions for how to adjust the striping
+of files yourself, but it is now automatic.
 
-    lfs setstripe -d path/to/directory
 
 Lustre: common recommendations
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-- Minimize use of ``ls -l`` and ``ls --color`` when possible
+------------------------------
 
-Several excellent recommendations are at
+Triton's Lustre is much better than it was 10 years ago, but it's
+still worth thinking about the following things:
 
--  https://www.nas.nasa.gov/hecc/support/kb/Lustre-Best-Practices_226.html
--  http://www.nics.tennessee.edu/computing-resources/file-systems/io-lustre-tips.
+Minimize use of ``ls -l`` and ``ls --color`` when possible.
 
-They are fully applicable to our case.
+Several excellent recommendations are at
+https://www.nas.nasa.gov/hecc/support/kb/Lustre-Best-Practices_226.html
+, they are fully applicable to our case.
 
 Be aware, that being a high performance filesystem Lustre still has its
 own bottlenecks, and even non-proper a usage by a single user can get
@@ -104,5 +84,5 @@ avoid those potential situations. Common Lustre troublemakers are
 ``ls -lR``, creating many small files, ``rm -rf``, small random i/o,
 heavy bulk i/o.
 
-For advanced user, these slides can be interesting:
-https://www.eofs.eu/fileadmin/lad2012/06_Daniel_Kobras_S_C_Lustre_FS_Bottleneck.pdf
+For advanced user, these slides (from 2012) can be interesting:
+https://www.eofs.eu/wp-content/uploads/2024/02/06_daniel_kobras_s_c_lustre_fs_bottleneck.pdf