Refactor for ML discoverability (#246)

Co-authored-by: Mikael Simberg <[email protected]>

Author: bcumming

.github/actions/spelling/allow.txt

@@ -230,6 +230,7 @@ opls
 osts
 osu
 papi
+paraview
 parmetis
 petsc
 pme

docs/build-install/index.md

@@ -2,30 +2,51 @@
 # Building and Installing Software
 
 CSCS provides commonly used software and tools on Alps, however many use cases will require first installing software on a system before you can start working.
+Modern HPC applications and software stacks are often very complicated, and there is no one-size-fits-all method for building and installing them.
 
 <div class="grid cards" markdown>
 
-- :fontawesome-solid-earth-americas: __[Unsupported software][ref-software-installation-guides]__ – guides on how to install commonly requested or difficult to install software.
+-   :fontawesome-solid-earth: [__Programming Environments__][ref-software-prgenvs]
 
-</div>
+    Programming environments are your first option if you want to install an application (and its dependencies) from source, or set up a Python/Julia environment.
 
-Modern HPC applications and software stacks are often very complicated, and there is no one-size-fits-all method for building and installing them.
+    CSCS provides the following uenv:
 
-<div class="grid cards" markdown>
+    [:octicons-arrow-right-24: prgenv-gnu][ref-uenv-prgenv-gnu]
+
+    [:octicons-arrow-right-24: prgenv-nvfortran][ref-uenv-prgenv-nvfortran]
+
+    [:octicons-arrow-right-24: linalg][ref-uenv-linalg]
+
+    [:octicons-arrow-right-24: julia][ref-uenv-julia]
 
-- :fontawesome-solid-earth-americas: __[uenv][ref-uenv]__ – uenv provide isolated software environments for applications and developers.
+    And containers are used to deploy:
+
+    [:octicons-arrow-right-24: Cray Programming Environment][ref-cpe]
 
 </div>
 
 <div class="grid cards" markdown>
 
-- :fontawesome-brands-docker: __[Podman][ref-build-containers]__ – for building software in containers.
+-   :fontawesome-solid-truck-fast: __Packaging and Deployment__
+
+    How to create containers or uenv, and how to share them with your colleagues and community.
+
+    [:octicons-arrow-right-24: build containers with podman][ref-build-containers]
+
+    [:octicons-arrow-right-24: use the uenv build service][ref-uenv-build]
 
 </div>
 
 <div class="grid cards" markdown>
 
-- :fontawesome-brands-python: __[Python][ref-build-python]__ – create a virtual environment using python in a uenv.
+- :fontawesome-solid-hammer: __Software Building Guides__
+
+    How to create containers or uenv, and how to share them with your colleagues and community.
+
+    [:octicons-arrow-right-24: building software using uenv][ref-build-uenv]
+
+    Coming soon: how to install Python software stacks.
 
 </div>
 

docs/build-install/pip.md

@@ -1,4 +1,4 @@
-[](){#ref-building-uenv}
+[](){#ref-build-uenv}
 # Building with uenv
 
 Uenv are user environments that provide scientific applications, libraries and tools on [Alps][ref-alps]. 
@@ -7,7 +7,7 @@ This article explains how to use them to build software.
 For more documentation on how to find, download and use uenv in your workflow,
 see the [uenv documentation][ref-uenv].
 
-[](){#ref-building-uenv-spack}
+[](){#ref-build-uenv-spack}
 ## Building software using Spack
 
 Each uenv is tightly coupled with [Spack] and can be used as an upstream [Spack] instance, because

docs/build-install/uenv.md

@@ -98,7 +98,7 @@ To build images, see the [guide to building container images on Alps][ref-build-
 
     CSCS will continue to support and update uenv and container engine, and users are encouraged to update their workflows to use these methods at the first opportunity.
 
-    The CPE is still installed on Daint, however it will receive no support or updates, and will be replaced with a container in a future update.
+    The CPE is still installed on Daint, however it will receive no support or updates, and will be [replaced with a container][ref-cpe] in a future update.
 
 ## Running jobs on Daint
 

docs/clusters/daint.md

@@ -122,7 +122,7 @@ To build images, see the [guide to building container images on Alps][ref-build-
 
     CSCS will continue to support and update uenv and the Container Engine, and users are encouraged to update their workflows to use these methods at the first opportunity.
 
-    The CPE is deprecated and will be removed completely at a future date.
+    The CPE is still installed on Eiger, however it will receive no support or updates, and will be [replaced with a container][ref-cpe] in a future update.
 
 ## Running jobs on Eiger
 

docs/clusters/eiger.md

@@ -514,3 +514,27 @@ For example, to include the recommended NCCL environment variables, do the follo
     ```bash title="Recommended NCCL environment variables"
     --8<-- "docs/software/communication/nccl_env_vars"
     ```
+
+## Documentation structure
+
+Here we describe a high-level overview of the documentation layout and organisation.
+
+!!! under-construction
+    This section is mostly incomplete, and will be expanded over time.
+
+Note that the directory layout, where markdown files are stored in the repository, does not strictly reflect the section of the documentation where the content is displayed because:
+
+* the URL of a page is decided by its location in the directory tree, not in the table of contents.
+  If a page is moved in the ToC, we are conservative about moving the file, so that urls don't break.
+* pages can be included in multiple locations in the ToC (not a feature that we use very often).
+
+### Tutorials
+
+All tutorials are stored in the `/docs/tutorials` directory.
+Currently we only have ML tutorials in `/docs/tutorials/ml`.
+
+There is no top level "Tutorials" section, instead tutorial content can be included directly in the docs where most appropriate.
+The ML tutorials, for example, are alongside the PyTorch documentation in the Applications and Frameworks material.
+
+!!! note "rationale"
+    Group all tutorial content together in the directory structure so that the url of specific tutorials won't change when they are moved around.

docs/contributing/index.md

@@ -0,0 +1,37 @@
+# Alps environments
+
+Once you have logged into a [cluster][ref-alps-clusters] on Alps, you will want to set up your environment and find the software and tools provided by CSCS.
+This part of the documentation will give guidance on how to set up your shell, and how to access software in uenv and container environments.
+
+<div class="grid cards" markdown>
+
+-   :fontawesome-solid-layer-group: __Set up your login environment__
+
+    A good spot to start is to read our guide on configuring your shell.
+
+    [:octicons-arrow-right-24: Using the terminal][ref-platform-hpcp]
+
+</div>
+
+<div class="grid cards" markdown>
+
+-   :fontawesome-solid-layer-group: __Setting up your development and simulation environments__
+
+    CSCS provides two ways to access software, and create the environment used to develop and run.
+
+    Scientific HPC workloads are a good fit for uenv -- a tool developed by CSCS to deliver scientific software stacks on Alps.
+
+    [:octicons-arrow-right-24: uenv][ref-uenv]
+
+    The container engine runtime is recommended for machine learning workflows.
+    It can also be a very good choice for Python environments installed using pip, uv and conda.
+
+    [:octicons-arrow-right-24: container engine][ref-container-engine]
+
+</div>
+
+
+CSCS provides ready to use environments in uenv and containers:
+
+* [Programming environments][ref-software-prgenvs];
+* [Applications and software][ref-software].

docs/environments/index.md

@@ -1,27 +1,30 @@
 # CSCS Documentation
 
-<div class="grid cards" markdown>
--   <p style="text-align:center">Visit <a href="https://status.cscs.ch/">status.cscs.ch</a> for the status of Alps systems, and the latest announcements.</p>
-</div>
-
-The Alps Research infrastructure hosts multiple platforms and clusters targeting different communities
+The Alps Research Infrastructure hosts multiple platforms and clusters targeting different communities.
+A good spot to get started is with the documentation for the platform that your project is running on.
 
 <div class="grid cards" markdown>
 
 -   :fontawesome-solid-layer-group: __Platforms__
 
     Projects at CSCS are granted access to [clusters][ref-alps-clusters], which are managed by platforms.
-    Start by finding the platform for the cluster that you want to use.
 
-    [:octicons-arrow-right-24: Platforms overview][ref-alps-platforms]
+    [:octicons-arrow-right-24: HPC Platform (__Daint__, __Eiger__)][ref-platform-hpcp]
+
+    [:octicons-arrow-right-24: Machine Learning Platform (__Clariden__, __Bristen__)][ref-platform-mlp]
 
-    Go straight to the documentation for the platform that hosts your project:
+    [:octicons-arrow-right-24: Climate and Weather Platform (__Santis__)][ref-platform-cwp]
 
-    [:octicons-arrow-right-24: HPC Platform (Daint, Eiger)][ref-platform-hpcp]
+    For an overview of the different platforms:
 
-    [:octicons-arrow-right-24: Machine Learning Platform (Clariden)][ref-platform-mlp]
+    [:octicons-arrow-right-24: Platforms overview][ref-alps-platforms]
 
-    [:octicons-arrow-right-24: Climate and Weather Platform (Santis)][ref-platform-cwp]
+</div>
+
+Alps is a general-purpose compute and data Research Infrastructure (RI) open to the broad community of researchers in Switzerland and the rest of the world.
+Find out more about Alps...
+
+<div class="grid cards" markdown>
 
 -   :fontawesome-solid-mountain-sun: __Alps__
 
@@ -37,19 +40,13 @@ The Alps Research infrastructure hosts multiple platforms and clusters targeting
 
     [:octicons-arrow-right-24: Alps Storage](alps/storage.md)
 
--   :fontawesome-solid-layer-group: __Accounts and Projects__
-
-    The first step is to get an account and a project
-
-    [:octicons-arrow-right-24: Accounts and Projects][ref-account-management]
-
 -   :fontawesome-solid-key: __Logging In__
 
-    Once you have an account, you can set up multi factor authentication
+    Once you have an account, you can set up multi factor authentication:
 
     [:octicons-arrow-right-24: Setting up MFA][ref-mfa]
 
-    Then access CSCS services
+    Then access CSCS services:
 
     [:octicons-arrow-right-24: Accessing CSCS Web Services][ref-access-web]
 
@@ -58,6 +55,20 @@ The Alps Research infrastructure hosts multiple platforms and clusters targeting
     [:octicons-arrow-right-24: FirecREST API][ref-firecrest]
 
 </div>
+<div class="grid cards" markdown>
+
+-   :fontawesome-solid-layer-group: __Accounts and Projects__
+
+    If you are a new user, or working on a project for the first time:
+
+    [:octicons-arrow-right-24: Accounts and Projects][ref-account-management]
+
+</div>
+
+<div class="grid cards" markdown>
+-   <p style="text-align:center">Visit <a href="https://status.cscs.ch/">status.cscs.ch</a> for the status of Alps systems, and the latest announcements.</p>
+</div>
+
 
 ## Tutorials and Guides
 
@@ -78,8 +89,6 @@ Learn by doing with our guides and tutorials.
 
     [:octicons-arrow-right-24: Accessing internet and external services][ref-guides-internet-access]
 
-    [:octicons-arrow-right-24: Using and configuring the terminal][ref-guides-terminal]
-
 </div>
 
 ## Tools and Services

docs/index.md

@@ -0,0 +1,14 @@
+# to replace all instances of ref-index-page with ref-index
+#  patchref ref-index-page ref-index
+
+value=$1
+patch=$2
+echo "replacing $value in"
+for f in $(rg "$value" -l)
+do
+    echo " $f"
+done
+for f in $(rg "$value" -l)
+do
+    sed -i '' "s|$value|$patch|g" $f
+done