Move MLP tutorials under software, add CE section to Pytorch including best practice for large-scale training (#231)

Co-authored-by: boeschf <[email protected]>
Co-authored-by: Ben Cumming <[email protected]>
Co-authored-by: Theofilos Manitaras <[email protected]>

Author: 4 people

.github/CODEOWNERS

@@ -9,6 +9,6 @@ docs/software/prgenv/linalg.md @finkandreas @msimberg
 docs/software/sciapps/cp2k.md @abussy @RMeli
 docs/software/sciapps/lammps.md @nickjbrowning
 docs/software/sciapps/gromacs.md @kanduri
-docs/software/ml @boeschf
+docs/software/ml @boeschf @henrique @lukasgd
 docs/storage @mpasserini
 docs/alps/storage.md @mpasserini

.github/actions/spelling/expect.txt

@@ -0,0 +1,4 @@
+JAX
+nvitop
+NVRTC
+placeholders

docs/access/jupyterlab.md

@@ -86,7 +86,7 @@ If the default base images do not meet your requirements, you can specify a cust
     3. Currently only required on Daint and Santis, not on Clariden
     4. Set working directory of Jupyter session (file browser root directory)
     5. Use environment settings for optimized communication 
-    6. Disable CUDA JIT cache
+    6. Avoid writing JITed binaries to the (distributed) file system, which could lead to performance issues.
     7. Async error handling when an exception is observed in NCCL watchdog: aborting NCCL communicator and tearing down process upon error
     8. Disable GPU support in MPICH, as it can lead to deadlocks when using together with NCCL
 
@@ -199,7 +199,9 @@ Examples of notebooks with `ipcmagic` can be found [here](https://github.com/
 
 While it is generally recommended to submit long-running machine learning training and inference jobs via `sbatch`, certain use cases can benefit from an interactive Jupyter environment.
 
-A popular approach to run multi-GPU ML workloads is with [`accelerate`](https://github.com/huggingface/accelerate) and [`torchrun`](https://docs.pytorch.org/docs/stable/elastic/run.html) as demonstrated in the [tutorials][ref-guides-mlp-tutorials]. In particular, the `accelerate launch` script in the [LLM fine-tuning tutorial][ref-mlp-llm-fine-tuning-tutorial] can be directly carried over to a Jupyter cell with a `%%bash` header (to run its contents interpreted by bash). For `torchrun`, one can adapt the command from the multi-node [nanotron tutorial][ref-mlp-llm-nanotron-tutorial] to run on a single GH200 node using the following line in a Jupyter cell
+A popular approach to run multi-GPU ML workloads is with [`accelerate`](https://github.com/huggingface/accelerate) and [`torchrun`](https://docs.pytorch.org/docs/stable/elastic/run.html) as demonstrated in the [tutorials][ref-tutorials-ml].
+In particular, the `accelerate launch` script in the [LLM fine-tuning tutorial][software-ml-llm-fine-tuning-tutorial] can be directly carried over to a Jupyter cell with a `%%bash` header (to run its contents interpreted by bash).
+For `torchrun`, one can adapt the command from the multi-node [nanotron tutorial][software-ml-llm-nanotron-tutorial] to run on a single GH200 node using the following line in a Jupyter cell
 
 ```bash
 !python -m torch.distributed.run --standalone --nproc_per_node=4 run_train.py ...

docs/build-install/containers.md

@@ -4,17 +4,22 @@
 Building OCI container images on Alps vClusters is supported through [Podman](https://podman.io/), an open-source container engine that adheres to OCI standards and supports rootless containers by leveraging Linux [user namespaces](https://www.man7.org/linux/man-pages/man7/user_namespaces.7.html).
 Its command-line interface (CLI) closely mirrors Docker’s, providing a consistent and familiar experience for users of established container tools.
 
+[](){#ref-build-containers-configure-podman}
 ## Preliminary step: configuring Podman's storage
 
-The first step in order to use Podman on Alps is to create a valid Container Storage configuration file at `$HOME/.config/containers/storage.conf` (or `$XDG_CONFIG_HOME/containers/storage.conf`, if you have `$XDG_CONFIG_HOME` set), according to the following minimal template:
+The first step in order to use Podman on Alps is to create a valid Container Storage configuration file in your home directory, according to the following minimal template:
 
-```toml
+```toml title="$HOME/.config/containers/storage.conf"
 [storage]
 driver = "overlay"
 runroot = "/dev/shm/$USER/runroot"
 graphroot = "/dev/shm/$USER/root"
 ```
 
+!!! warning
+    If `$XDG_CONFIG_HOME` is set, place this file at `$XDG_CONFIG_HOME/containers/storage.conf` instead.
+    See the [terminal user guide][ref-guides-terminal-arch] for further information about XDG variables.
+
 !!! warning
     In the above configuration, `/dev/shm` is used to store the container images.
     `/dev/shm` is the mount point of a [tmpfs filesystem](https://www.kernel.org/doc/html/latest/filesystems/tmpfs.html#tmpfs) and is compatible with the user namespaces used by Podman.
@@ -43,11 +48,27 @@ podman build -t <image:tag> .
 
 In general, [`podman build`](https://docs.podman.io/en/stable/markdown/podman-build.1.html) follows the Docker options convention.
 
+!!! info "Debugging the container build"
+    If the container build fails, you can run an interactive shell using the image from the last successfully built layer with
+
+    ```bash
+    podman run -it --rm -e NVIDIA_VISIBLE_DEVICES=void <last-layer-hash> bash # (1)!
+    ```
+
+    1. Setting `NVIDIA_VISIBLE_DEVICES` in the environment is required specifically to run NGC containers with podman
+
+    replacing `<last-layer-hash>` with the actual hash output in the build job and interactively test the failing command.
+
+
 ## Importing images in the Container Engine
 
 An image built using Podman can be easily imported as a squashfs archive in order to be used with our Container Engine solution.
 It is important to keep in mind that the import has to take place in the same job allocation where the image creation took place, otherwise the image is lost due to the temporary nature of `/dev/shm`.
 
+!!! info "Preliminary configuration: Lustre settings for container images"
+    Container images are stored in a single [SquashFS]() file, that is typically between 1-20 GB in size (particularly for large ML containers).
+    To ensure good performance for jobs on multiple nodes, take the time to configure the target directory using `lfs setstripe` according to [best practices for Lustre][ref-guides-storage-lustre] before importing the container image, or using `lfs migrate` to fix files that are already imported.
+
 To import the image:
 
 ```
@@ -62,7 +83,6 @@ image = "/<path to image directory>/<image_name.sqsh>"
 mounts = ["/capstor/scratch/cscs/<username>:/capstor/scratch/cscs/<username>"]
 workdir = "/capstor/scratch/cscs/<username>"
 ```
-
 ## Pushing Images to a Container Registry
 
 In order to push an image to a container registry, you first need to follow three steps:

docs/clusters/clariden.md

@@ -65,6 +65,8 @@ Alternatively, [uenv][ref-uenv] are also available on Clariden. Currently deploy
     uenv start namd/3.0:v3@daint
     ```
 
+For detailed instructions and best practices with ML frameworks, please refer to the dedicated pages under [ML software][ref-software-ml].
+
 ## Running Jobs on Clariden
 
 ### Slurm

docs/guides/mlp_tutorials/index.md

@@ -1,9 +1,3 @@
-!!! info ""
-    This is the new CSCS documentation site, which replaces the [CSCS Knowledge Base](https://confluence.cscs.ch/display/KB).
-
-    The migration of old documentation is still not fully complete.
-    If you find documentation that is missing, please create a ticket on the documentation's [GitHub issue tracker](https://github.com/eth-cscs/cscs-docs/issues).
-
 # CSCS Documentation
 
 <div class="grid cards" markdown>
@@ -66,32 +60,26 @@ The Alps Research infrastructure hosts multiple platforms and clusters targeting
 
 </div>
 
-[](){#ref-get-in-touch}
-## Get in Touch
+## Tutorials and Guides
 
-If you cannot find the information that you need in the documentation, help is available.
+Learn by doing with our guides and tutorials.
 
 <div class="grid cards" markdown>
+-   :fontawesome-solid-layer-group: __Tutorials__
 
--   :fontawesome-solid-headset: __Get Help__
-
-    Contact the CSCS Service Desk for help.
-
-    [:octicons-arrow-right-24: Service Desk](https://jira.cscs.ch/plugins/servlet/desk)
+    Hands on tutorials that show how to implement workflows on Alps.
 
--   :fontawesome-regular-comments: __Chat__
+    [:octicons-arrow-right-24: Machine Learning][ref-tutorials-ml]
 
-    Discuss Alps with other users and CSCS staff on Slack.
+-   :fontawesome-solid-mountain-sun: __Guides__
 
-    [:octicons-arrow-right-24: CSCS User Slack](https://cscs-users.slack.com/)
+    Guides with practical advice, hints and tips for key topics.
 
-<div class="grid cards" markdown>
--   :fontawesome-solid-hammer: __Contribute__
+    [:octicons-arrow-right-24: Using storage effectively][ref-guides-storage]
 
-    The source for the documentation is hosted on GitHub.
+    [:octicons-arrow-right-24: Accessing internet and external services][ref-guides-internet-access]
 
-    [:octicons-arrow-right-24: Contribute to the docs ](contributing/index.md)
-</div>
+    [:octicons-arrow-right-24: Using and configuring the terminal][ref-guides-terminal]
 
 </div>
 
@@ -142,3 +130,32 @@ If you cannot find the information that you need in the documentation, help is a
 
 </div>
 
+[](){#ref-get-in-touch}
+## Get in Touch
+
+If you cannot find the information that you need in the documentation, help is available.
+
+<div class="grid cards" markdown>
+
+-   :fontawesome-solid-headset: __Get Help__
+
+    Contact the CSCS Service Desk for help.
+
+    [:octicons-arrow-right-24: Service Desk](https://jira.cscs.ch/plugins/servlet/desk)
+
+-   :fontawesome-regular-comments: __Chat__
+
+    Discuss Alps with other users and CSCS staff on Slack.
+
+    [:octicons-arrow-right-24: CSCS User Slack](https://cscs-users.slack.com/)
+
+<div class="grid cards" markdown>
+-   :fontawesome-solid-hammer: __Contribute__
+
+    The source for the documentation is hosted on GitHub.
+
+    [:octicons-arrow-right-24: Contribute to the docs ](contributing/index.md)
+</div>
+
+</div>
+

docs/index.md

@@ -3,6 +3,15 @@
 
 The Machine Learning Platform (MLP) provides compute, storage and expertise to the machine learning and AI community in Switzerland, with the main user being the [Swiss AI Initiative](https://www.swiss-ai.org/).
 
+<div class="grid cards" markdown>
+-   :fontawesome-solid-mountain: [__Tutorials__][ref-tutorials-ml]
+
+    Tutorials on how to set up and configure a machine learning environment in order to run LLM workloads such as inference, fine-tuning and multi-node training can be found in the [tutorials section][ref-tutorials-ml].
+
+    Also check out the [PyTorch documentation][ref-software-ml-pytorch] for information about how to run PyTorch.
+
+</div>
+
 ## Getting started
 
 ### Getting access
@@ -89,6 +98,3 @@ Project is per project - each project gets a project folder with project-specifi
 * hard limits on capacity and inodes prevent users from writing to project if the quota is reached - you can check quota and available space by running the [`quota`][ref-storage-quota] command on a login node or ela 
 * it is not recommended to write directly to the project path from jobs.
 
-## Guides and tutorials
-
-Tutorials for fine-tuning and running inference of LLMs as well as training an LLM with Nanotron can be found in the [MLP Tutorials][ref-guides-mlp-tutorials] page.

docs/platforms/mlp/index.md

@@ -14,7 +14,7 @@ When using e.g. the `default` view of `prgenv-gnu` the `aws-ofi-nccl` plugin wil
 Alternatively, loading the `aws-ofi-nccl` module with the `modules` view also makes the plugin available in the environment.
 The environment variables described below must be set to ensure that NCCL uses the plugin.
 
-While the container engine sets these automatically when using the NCCL hook, the following environment variables should always be set for correctness and optimal performance when using NCCL:
+While the container engine sets these automatically when using the NCCL hook, the following environment variables should always be set for correctness and optimal performance when using NCCL with uenv:
 
 ```bash
 --8<-- "docs/software/communication/nccl_env_vars"

docs/software/communication/nccl.md

@@ -2,22 +2,33 @@
 # Machine learning applications and frameworks
 
 CSCS supports a wide range of machine learning (ML) applications and frameworks on its systems.
-Most ML workloads are containerized to ensure portability, reproducibility, and ease of use across environments.
+Most ML workloads are containerized to ensure portability, reproducibility, and ease of use across systems.
 
 Users can choose between running containers, using provided uenv software stacks, or building custom Python environments tailored to their needs.
 
-## Running machine learning applications with containers
+First time users are recommended to consult the [LLM tutorials][ref-tutorials-ml] to get familiar with the concepts of the Machine Learning platform in a series of hands-on examples. 
+
+## Running ML applications with containers (recommended)
 
 Containerization is the recommended approach for ML workloads on Alps, as it simplifies software management and maximizes compatibility with other systems.
 
-* Users are encouraged to build their own containers, starting from popular sources such as the [Nvidia NGC Catalog](https://catalog.ngc.nvidia.com/containers), which offers a variety of pre-built images optimized for HPC and ML workloads.
+Users are encouraged to build their own containers, starting from popular sources such as the [Nvidia NGC Catalog](https://catalog.ngc.nvidia.com/containers), which offers a variety of pre-built images optimized for HPC and ML workloads.
 Examples include:
-    * [PyTorch NGC container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch)
-    * [TensorFlow NGC container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorflow)
-* For frequently changing dependencies, consider creating a virtual environment (venv) mounted into the container.
+
+* [PyTorch NGC container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch) ([Release Notes](https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/index.html))
+* [JAX NGC container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/jax) ([Release Notes](https://docs.nvidia.com/deeplearning/frameworks/jax-release-notes/index.html))
+* [TensorFlow NGC container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorflow) (deprecated since 25.02, see [Release Notes](https://docs.nvidia.com/deeplearning/frameworks/tensorflow-release-notes/index.html))
+
+Documented best practices are available for:
+
+* [PyTorch][ref-ce-pytorch]
+
+!!! note "Extending a container with a virtual environment"
+    For frequently changing Python dependencies during development, consider creating a Virtual Environment (venv) on top of the packages in the container (see [this example][ref-ce-pytorch-venv]).
 
 Helpful references:
 
+* Introduction to concepts of the Machine Learning platform: [LLM tutorials][ref-tutorials-ml]
 * Running containers on Alps: [Container Engine Guide][ref-container-engine]
 * Building custom container images: [Container Build Guide][ref-build-containers]
 
@@ -30,17 +41,18 @@ Available ML-related uenvs:
 
 * [PyTorch][ref-uenv-pytorch] — available on [Clariden][ref-cluster-clariden] and [Daint][ref-cluster-daint]
 
-To extend these environments with additional Python packages, it is recommended to create a Python Virtual Environment (venv).
-See this [PyTorch venv example][ref-uenv-pytorch-venv] for details.
-
-!!! note
-    While many Python packages provide pre-built binaries for common architectures, some may require building from source.
+!!! note "Extending a uenv with a virtual environment"
+    To extend these environments with additional Python packages, it is recommended to create a Python Virtual Environment (venv) layered on top of the packages in the uenv.
+    See this [PyTorch venv example][ref-uenv-pytorch-venv] for details.
 
 ## Building custom Python environments
 
 Users may also choose to build entirely custom software stacks using Python package managers such as `uv` or `conda`.
 Most ML libraries are available via the [Python Package Index (PyPI)](https://pypi.org/).
 
+!!! note
+    While many Python packages provide pre-built binaries for common architectures, some may require building from source.
+
 To ensure optimal performance on CSCS systems, we recommend starting from an environment that already includes:
 
 * CUDA, cuDNN