debuggers and performance analysis tools (#46)

Author: jgphpc

docs/index.md

@@ -68,7 +68,7 @@ The Alps Research infrastructure hosts multiple platforms and clusters targeting
 [](){#ref-get-in-touch}
 ## Get in Touch
 
-If you can't find the information that you need in the documentation, help is available.
+If you cannot find the information that you need in the documentation, help is available.
 
 <div class="grid cards" markdown>
 

docs/services/index.md

@@ -6,9 +6,8 @@
     * slurm
     * uenv
     * container engine
-    * debuggers and profilers
 
-    We document the tools and their interfaces here, but we don't put all the documentation for a tool here.
+    We document the tools and their interfaces here, but we do not put all the documentation for a tool here.
 
     * e.g Documentation on how to build software using uenv, is in another section.
     * e.g Documentation on how to use Podman to build containers, is in another section.

docs/software/devtools/index.md

@@ -0,0 +1,34 @@
+[](){#ref-software-devtools}
+# Debugging and Performance Analysis tools
+
+Debugging and Performance Analysis tools can assist users in developing and optimizing scientific parallel applications, especially in a high-performance computing (HPC) environment.
+Efficient tools can significantly improve workflows and save valuable computational resources.
+
+CSCS provides debuggers and performance analysis tools on Alps Clusters.
+
+!!! note "get in touch"
+    If you have issues or questions about debugging or performance analysis tools, please do not hesitate to [contact us][ref-get-in-touch].
+
+[](){#ref-devtools-debug}
+## Debugging
+
+Parallel debugging tools designed for parallel and distributed applications can help you diagnose issues and verify the correctness of your code - whether you are using MPI, OpenMP, or accelerated programming models.
+
+Learning to debug a code effectively will not only help you quickly resolve issues but also build a deeper understanding of how your code interacts with the underlying hardware.
+In this section we introduce the various debugging tools available at CSCS.
+
+* [Linaro Forge DDT][ref-devtools-ddt]
+
+[](){#ref-devtools-perf}
+## Performance Analysis
+
+Performance analysis tools are essential to gain insight into how an application leverages a distributed system with CPUs and GPUs and should be integrated to the development and optimization workflow of the application.
+This ensures that computational resources are utilized to their fullest potential.
+
+Learning to analyze the performance of an applications effectively is crucial to build a deeper understanding of how your code interacts with the underlying hardware.
+In this section we introduce the various performance analysis solutions available at CSCS.
+If you have issues or questions about performance analysis tools, please do not hesitate to [contact us][ref-get-in-touch].
+
+* [Linaro Forge MAP][ref-devtools-map]
+* [NVIDIA Nsight Developer Tools][ref-devtools-nsight]
+

docs/software/devtools/linaro-ddt.md

@@ -0,0 +1,81 @@
+[](){#ref-devtools-ddt}
+# Linaro DDT
+
+DDT allows source-level debugging of Fortran, C, C++ and Python codes.
+It can be used for debugging serial, multi-threaded (OpenMP), multi-process (MPI) and accelerated (CUDA, OpenACC) programs running on research and production systems, including the CSCS Alps system.
+DDT can be executed either with its graphical user interface or from the command-line.
+
+!!! note
+    Linaro DDT is provided in the `linaro-forge` uenv.
+    Before using DDT, please read the [`linaro-forge` documentation][ref-uenv-linaro], which explains how to download and set up the latest version and set it up.
+
+## User guide
+
+The following guid will walk through the steps required to build and debug an application using DDT.
+
+### Set up the user environment and build the executable
+
+Once the uenv is loaded and activated, the program to debug must be compiled with the `-g` (for CPU) and `-G` (for GPU) debugging flags.
+For example, we can build a CUDA test with a user environment:
+
+```terminal
+uenv start prgenv-gnu:24.11:v1 --view=default
+nvcc -c -arch=sm_90 -g -G test_gpu.cu
+mpicxx -g test_cpu.cpp test_gpu.o -o myexe
+```
+
+### Launch Linaro DDT
+
+To use the DDT client with uenv, it must be launched in `Manual Launch` mode
+(assuming that it is connected to Alps via `Remote Launch`):
+
+=== "on local machine"
+
+    Start DDT, and connect to the target cluster using the drop down menu for `Remote Launch`.
+
+    Click on `Manual launch`, set the number of processes to listen to, then wait for the slurm job to start (see the "on Alps" tab).
+
+    <img src="https://raw.githubusercontent.com/jgphpc/cornerstone-octree/ddt/scripts/img/ddt/0.png" width="600" />
+
+=== "on Alps"
+
+    Log into the system and launch with the `srun` command:
+
+    ```terminal
+    # start a session with both the PE used to build your application
+    # and the linaro-forge uenv mounted
+    > uenv start prgenv-gnu/24.11:v1,linaro-forge/24.1.1:v1 --view=prgenv-gnu:default
+    > source /user-tools/activate
+
+    > srun -N1 -n4 -t15 -pdebug ./cuda_visible_devices.sh   ddt-client   ./myexe
+    ```
+
+### Start debugging
+
+By default, DDT will pause execution on the call to `MPI_Init`:
+<img src="https://raw.githubusercontent.com/jgphpc/cornerstone-octree/ddt/scripts/img/ddt/1.png" width="600" />
+
+There are two mechanisms for controlling program execution:
+
+=== "Breakpoint"
+
+    Breakpoint(s) can be set by clicking in the margin to the left of the line number:
+
+    <img src="https://raw.githubusercontent.com/jgphpc/cornerstone-octree/ddt/scripts/img/ddt/3.png" width="600" />
+
+=== "Stop at"
+
+    Execution can be paused in every CUDA kernel launch by activating the default breakpoints from the Control menu:
+
+    <img src="https://raw.githubusercontent.com/jgphpc/cornerstone-octree/ddt/scripts/img/ddt/4.png" width="400" />
+
+
+This screenshot shows a debugging session on 128 gpus:
+
+![DDTgpus](https://raw.githubusercontent.com/jgphpc/cornerstone-octree/ddt/scripts/img/ddt/5.png)
+
+More informations regarding how to use Linaro DDT are provided in the Forge [User Guide](https://docs.linaroforge.com/latest/html/forge/index.html).
+
+## Troubleshooting
+
+See the troubleshooting guide for the [`linaro-forge` uenv][ref-uenv-linaro-troubleshooting].

docs/software/devtools/linaro-forge.md

@@ -0,0 +1,160 @@
+[](){#ref-uenv-linaro}
+# Linaro Forge
+
+[Linaro Forge](https://docs.linaroforge.com/latest/html/forge/index.html) is a suite of profiling and debugging tools, that includes the DDT debugger and the MAP profiler.
+
+!!! note
+    We have separate user guides for the tools provided by the `linaro-forge` uenv.
+    The documentation here shows how to download the uenv, and how to set up your environment.
+
+    Once you are set up, follow the specific guides:
+
+    * the [DDT debugger][ref-devtools-ddt]
+    * the [MAP profiler][ref-devtools-map].
+
+## Quickstart guide
+
+The Linaro uenv is named `linaro-forge`, and the available versions can be determined using the `uenv image find` command:
+
+!!! example "finding available linaro-forge versions"
+
+    ```
+    > uenv image find linaro-forge
+    uenv                    arch   system  id                size(MB)  date
+    linaro-forge/24.1.1:v1  gh200  daint   e0e79f5c3e6a8ee0  365       2025-02-12
+
+    > uenv image pull linaro-forge/24.1.1:v1
+    pulling e0e79f5c3e6a8ee0 100.00% --- 365/365 (0.00 MB/s)
+    ```
+
+This uenv is configured to be mounted in the `/user-tools` path so that they can be used alongside application and development uenv mounted at `/user-environment`.
+
+When using alongside another uenv, start a uenv session with both uenv.
+In the following example, the `prgenv-gnu` and `linaro-forge` uenv will be mounted at `/user-environment` and `/user-tools`  respectively:
+
+```bash
+> uenv start prgenv-gnu/24.11,linaro-forge/24.1.1 \
+    --view=prgenv-gnu:default,forge
+
+# test that everything has been mounted correctly
+# (will give warnings if there are problems)
+> uenv status
+
+# check that ddt is in the path
+> ddt --version
+Linaro DDT Part of Linaro Forge.
+Copyright (c) 2023-2024 Linaro Limited. All rights reserved.
+Version: 24.1.1
+```
+
+!!! note
+    The `linaro-forge` uenv is always mounted at the `/user-tools` mount point, and a script `/user-tools/activate` is provided to load both ddt and map into your environment, without needing to use a view.
+
+    ```bash
+    > uenv start linaro-forge/14.1.1
+    > source /user-tools/activate
+    > ddt --version
+    Linaro DDT Part of Linaro Forge.
+    Copyright (c) 2023-2024 Linaro Limited. All rights reserved.
+    Version: 24.1.1
+    ```
+
+### Install and configure the Linaro client on your local machine
+
+We recommend installing the [desktop client](https://www.linaroforge.com/downloadForge) on your local workstation/laptop.
+It can be downloaded for a selection of operating systems.
+The client can be configured to connect with the debug jobs running on Alps, offering a better user experience compared to running with X11 forwarding.
+Once installed, the client needs to be configured to connect to the vCluster on which you are working.
+
+First, start the client on your laptop:
+
+=== "Linux"
+
+    The path will change if you have installed a different version, or if it has been installed in a non-standard installation location.
+
+    ```bash
+    $HOME/linaro/forge/24.1.1/bin/ddt
+    ```
+
+=== "MacOS"
+
+    The path will change if you have installed a different version, or if it has been installed in a non-standard installation location.
+
+    ```bash
+    open /Applications/Linaro\ Forge\ Client\ 24.1.1.app/
+    ```
+
+Next, configure a connection to the target system.
+Open the *Remote Launch* menu and click on *configure* then *Add*.
+Examples of the settings are below.
+
+=== "Daint"
+
+    | Field       | Value                                   |
+    | ----------- | --------------------------------------- |
+    | Connection  | `daint`                                  |
+    | Host Name   | `[email protected] [email protected]`  |
+    | Remote Installation Directory | `uenv run linaro-forge/24.1.1:/user-tools -- /user-tools/env/forge/` |    
+
+=== "Santis"
+
+    | Field       | Value                                   |
+    | ----------- | --------------------------------------- |
+    | Connection  | `santis`                                |
+    | Host Name   | `[email protected] [email protected]`  |
+    | Remote Installation Directory | `uenv run linaro-forge/24.1.1:/user-tools -- /user-tools/env/forge/` |
+
+=== "Clariden"
+
+    | Field       | Value                                   |
+    | ----------- | --------------------------------------- |
+    | Connection  | `clariden`                                |
+    | Host Name   | `[email protected] [email protected]`  |
+    | Remote Installation Directory | `uenv run linaro-forge/24.1.1:/user-tools -- /user-tools/env/forge/` |
+
+=== "Eiger"
+
+    | Field       | Value                                   |
+    | ----------- | --------------------------------------- |
+    | Connection  | `eiger`                                |
+    | Host Name   | `[email protected] [email protected]`  |
+    | Remote Installation Directory | `uenv run linaro-forge/24.1.1:/user-tools -- /user-tools/env/forge/` |
+
+!!! tip
+
+    It is recommended to log into Alps using `ela.cscs.ch` as a ssh Jump host, as explained [here][ref-ssh-config].
+    In that case, you can remove `[email protected]` from the Linaro client configuration above.
+
+Some notes on the examples above:
+
+* SSH forwarding via `ela.cscs.ch` is used to access the cluster;
+* replace the username `cscsusername` with your CSCS user name that you would normally use to open an SSH connection to CSCS;
+* `Remote Installation Path` is pointing to the install directory of ddt inside the image;
+* private keys should be the ones generated for CSCS MFA, and this field does not need to be set if you have added the key to your [SSH agent][ref-ssh-agent].
+
+Once configured, test and save the configuration:
+
+1. check whether the configuration is correct, click `Test Remote Launch`.
+2. Click on `ok` and `close` to save the configuration.
+3. You can now connect by going to `Remote Launch` and choose the `Alps` entry.
+   If the client fails to connect, look at the error message, check your SSH
+   configuration and make sure you can ssh without the client.
+
+[](){#ref-uenv-linaro-troubleshooting}
+## Troubleshooting
+
+Notes about known issues.
+
+!!! warning "The proxy type is invalid for this operation"
+
+    If the tool fails to launch with the following error message: 
+
+        Error communicating with Licence Server velan.cscs.ch:
+        The proxy type is invalid for this operation
+        Attempting again while ignoring proxies.
+
+    Proxy environment variables need to be set to let the tool connect to the license server, as explained in [Compute node proxy configuration][ref-guides-internet-access].
+
+!!! note "AMD GPU support"
+
+    CSCS does not currently have a Linaro license for AMD gpus.

docs/software/devtools/linaro-map.md

@@ -0,0 +1,48 @@
+[](){#ref-devtools-map}
+# Linaro Forge MAP and Performance Reports
+
+MAP can be used for profiling serial, multi-threaded (OpenMP), multi-process (MPI) and accelerated (Cuda, OpenACC) programs running on research and production systems, including the CSCS Alps system.
+MAP can be executed either with its graphical user interface or from the command-line.
+
+Linaro MAP can be used to profile an application either by the GUI or by the CLI.
+In the first case the user can set the profiling configuration using the GUI and then see the results.
+In the latter (recommended) case, the user can use the MAP executable to launch the application they want to profile which will generate a report file that can then be opened from the locally installed [client](https://docs.linaroforge.com/latest/html/forge/forge/installing/mac_install.html).
+
+!!! note
+    Linaro Map is provided in the `linaro-forge` uenv.
+    Before using Map, please read the [`linaro-forge` documentation][ref-uenv-linaro], which explains how to download and set up the latest version and set it up.
+
+## Linaro Forge MAP
+
+We will focus here on the profiling using MAP from the CLI but the same configuration applies in the other case as well.
+To debug an MPI application on Alps the following script is necessary:
+
+```bash
+> map -n <num_of_procs> --mpi=slurm --mpiargs="<slurm_arguments>" \
+  --profile <executable> <executable_arguments>
+```
+
+This will generate a profile report in a binary file with suffix `.map`.
+
+To open this file we can open the Linaro Forge Client on our local machine, navigate to the `Linaro MAP` tab, connect to the corresponding `Remote` and then select `LOAD PROFILE DATA FILE` to locate the file.
+
+After loading the report file we will be in the home of Linaro MAP.
+
+<img src="https://raw.githubusercontent.com/iomaganaris/alps-uenv/refs/heads/linaro_map_docs_archive/docs/images/map-home.png" width="800" />
+
+## Linaro Forge Performance Reports
+
+Linaro MAP also allows the generation of a high level Performance Report in HTML format that shows key metrics of the profiled application.
+To see this we can click in the toolbar `Reports > View HTML Performance Report in browser`.
+
+This will look like the following:
+
+<center>
+<img src="https://raw.githubusercontent.com/iomaganaris/alps-uenv/refs/heads/linaro_map_docs_archive/docs/images/perf-report.png" width="300">
+</center>
+
+More informations regarding how to use Linaro MAP and Performance Reports are provided in the Forge [User Guide](https://docs.linaroforge.com/latest/html/forge/index.html).
+
+## Troubleshooting
+
+See the troubleshooting guide for the [`linaro-forge` uenv][ref-uenv-linaro-troubleshooting].

docs/software/devtools/nvidia-nsight.md

@@ -0,0 +1,42 @@
+[](){#ref-devtools-nsight}
+# NVIDIA Nsight
+
+## NVIDIA Nsight Systems
+
+[NVIDIA Nsight Systems](https://developer.nvidia.com/nsight-systems) is a system-wide performance analysis tool that enables developers to gain a deep understanding of how their applications utilize computing resources, such as CPUs, GPUs, memory, and I/O.
+The tool provides a unified view of application performance across the entire system, capturing detailed trace information that allows users to analyze how different components interact and where performance issues might arise.
+
+A key advantage of Nsight Systems is its ability to provide detailed traces of GPU activity, offering deeper insights into GPU utilization.
+It features a timeline-based visualization, enabling developers to inspect the execution flow, pinpoint latencies, and correlate events across different system components.
+As a sampling profiler, it can be easily used to profile applications written in C, C++, Python, Fortran, or Julia by wrapping the application with the Nsight Systems profiler executable.
+
+!!! note
+    NVIDIA Nsight Systems is available with any [uenv][ref-uenv] that comes with a CUDA compiler, for instance [`prgenv-gnu`][ref-uenv-prgenv-gnu].
+
+## NVIDIA Nsight Compute
+
+[NVIDIA Nsight Compute](https://developer.nvidia.com/nsight-compute) is a performance analysis tool specifically designed for optimizing GPU-accelerated applications.
+It focuses on providing detailed metrics and insights into the performance of CUDA kernels, helping developers identify performance bottlenecks and improve the efficiency of their GPU code.
+Nsight Compute offers a kernel-level profiler with customizable reports, enabling in-depth analysis of memory usage, compute utilization, and instruction throughput.
+As a sampling profiler, it can be easily used to profile applications written in C, C++, Python, Fortran, or Julia by wrapping the application with the Nsight Compute profiler executable.
+
+!!! note
+    NVIDIA Nsight Systems is available with any [uenv][ref-uenv] that comes with a CUDA compiler, for instance [`prgenv-gnu`][ref-uenv-prgenv-gnu].
+
+### Known Issues and Common Problems
+
+??? warning "CrashReporter: Qt initialization failed, Failed to load Qt platform plugin: xcb"
+    While we recommend using `ncu` instead of `ncu-ui`, it is possible to use X11 to launch ncu-ui.
+    However, this will fail with the following error message: `Failed to load Qt platform plugin: "xcb"`.
+
+    To workaround this issue, you can follow these instructions:
+
+    ```bash
+    ssh -Y -J <user>@ela.cscs.ch <user>@daint.alps.cscs.ch
+    # the -Y ssh flag enables trusted X11 forwarding.
+    wget https://jfrog.svc.cscs.ch/artifactory/cscs-reframe-tests/nvidia/ncu_deps.tar.gz
+    tar xf ncu_deps.tar.gz
+    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PWD/usr/lib64
+    ncu-ui &
+    ```
+