Merge pull request #301 from vkucera/tools

Add Tools section

Author: ddobrigk

docs/datamodel/README.md

@@ -1,5 +1,5 @@
 ---
-sort: 9
+sort: 10
 title: Appendix: data model reference
 ---
 

docs/gettingstarted/gitbasics.md

@@ -48,11 +48,7 @@ Selected sections:
 - [git - the simple guide](https://rogerdudler.github.io/git-guide/)
 - [Oh Shit, Git!?!](https://ohshitgit.com/)
 
-## Tips
-
-### Tools
-
-[Visual Studio Code (VS Code)](https://code.visualstudio.com/) – Integrated development environment
+## Tools
 
 [Meld](https://meldmerge.org/) – Visual diff and merge tool that can be used with Git commands `git difftool` and `git mergetool`
 

docs/gettingstarted/installing.md

@@ -25,20 +25,20 @@ ALICE uses aliBuild to build software. aliBuild:
 
 According to your operating system, please follow the prerequisites below. You will find a list of packages to install and configurations to perform.
 
-##### Primary supported platform:
+##### Primary supported platform
 
 * [CentOS 7](#prereq-for-centos7)
 * [CentOS/AlmaLinux 8](#prereq-for-centos8)
 * [AlmaLinux 9](#prereq-for-alma9)
 
-##### Platforms supported on a best-effort basis:
+##### Platforms supported on a best-effort basis
 
 * [macOS Sonoma and Sequoia (14.0, 15.0)](#prereq-for-macos)
 * [Ubuntu (20.04 LTS, 22.04 LTS, 24.04 LTS)](#prereq-for-ubuntu)
 * [Fedora](#prereq-for-fedora)
 * Linux Mint
-    * Follow the instructions for the Ubuntu version your Linux Mint version is based on.
-    * Specify the corresponding Ubuntu architecture when running the `aliBuild` command using the `-a` option (e.g. `-a ubuntu2004_x86-64` for Ubuntu 20.04). Use the `-a` option also with the `alienv` command.
+  * Follow the instructions for the Ubuntu version your Linux Mint version is based on.
+  * Specify the corresponding Ubuntu architecture when running the `aliBuild` command using the `-a` option (e.g. `-a ubuntu2004_x86-64` for Ubuntu 20.04). Use the `-a` option also with the `alienv` command.
 
 If your operating system is *not* in any list, it does not mean our software won't work on it; it will be just more difficult for you to get support for it.
 
@@ -47,6 +47,7 @@ Only in case you cannot install aliBuild in the way described above, you can ins
 <h6 id="prereq-for-centos7"> aliBuild prerequisites for CentOS7 </h6>
 
 With root permission, i.e. `sudo` or as `root`install the prerequisits using:
+
 ```bash
 cat << EOF > /etc/yum.repos.d/alice-system-deps.repo
 [alice-system-deps]
@@ -64,6 +65,7 @@ yum install -y alibuild
 <h6 id="prereq-for-centos8"> aliBuild prerequisites for CentOS8 </h6>
 
 With root permission, i.e. `sudo` or as `root` install the prerequisits using:
+
 ```bash
 yum install -y epel-release
 yum install -y dnf-plugins-core
@@ -86,6 +88,7 @@ yum install -y alice-o2-full-deps alibuild
 <h6 id="prereq-for-alma9"> aliBuild prerequisites for AlmaLinux9 </h6>
 
 With root permission, i.e. `sudo` or as `root`install the prerequisits using:
+
 ```bash
 dnf install -y epel-release dnf-plugins-core
 dnf update -y
@@ -120,7 +123,6 @@ Xcode bundles the necessary tools to build software in the apple ecosystem inclu
 * Install Homebrew using the [instructions on their webpage](https://brew.sh/).
 * Once installed detect any problems regarding Homebrew and your system using: `brew doctor`
 
-
 **Uninstall ROOT**<br>
 If you have an existing ROOT installation on your system, this will interfere with the aliBuild installation, and will cause difficult-to-debug compilation errors.
 
@@ -138,7 +140,6 @@ Note that Homebrew does not run as root. Do not prepend sudo to any of the follo
 ```echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> ~/.zprofile```
 * Close Terminal and reopen it to apply changes.
 
-
 <h6 id="prereq-for-ubuntu">aliBuild prerequisites for Ubuntu</h6>
 
 ALICE software on Ubuntu is supported on a best effort basis. There is no guarantee that software builds or runs correctly. Support requests might have low priority. We were able to successfully build on:
@@ -150,13 +151,17 @@ ALICE software on Ubuntu is supported on a best effort basis. There is no guaran
 **Install required system packages**<br>
 
 With root permissions, i.e. sudo, update your package sources:
+
 ```bash
 sudo apt update -y
 ```
+
 With root permissions, i.e. `sudo`, install the following packages:
+
 ```bash
 sudo apt install -y curl libcurl4-gnutls-dev build-essential gfortran libmysqlclient-dev xorg-dev libglu1-mesa-dev libfftw3-dev libxml2-dev git unzip autoconf automake autopoint texinfo gettext libtool libtool-bin pkg-config bison flex libperl-dev libbz2-dev swig liblzma-dev libnanomsg-dev rsync lsb-release environment-modules libglfw3-dev libtbb-dev python3-dev python3-venv python3-pip graphviz libncurses-dev software-properties-common gtk-doc-tools
 ```
+
 **Installing aliBuild**<br>
 
 AliBuild, our build tool, is installed as a standard ubuntu package, provided you enable the alisw PPA repository. This is done with:
@@ -170,6 +175,7 @@ sudo apt install python3-alibuild
 <h6 id="prereq-for-fedora">aliBuild prerequisites for Fedora</h6>
 
 With root permissions, i.e. `sudo` or as `root` install the prerequisites using:
+
 ```bash
 yum install -y epel-release
 yum install -y dnf-plugins-core
@@ -192,10 +198,12 @@ yum install -y alice-o2-full-deps alibuild
 ### Configure aliBuild
 
 After you are done installing alibuild you need to configure it by adding the two following lines to your `~/.bashrc`, `~/.bash_profile`, `~/.zshrc` or `~/.zprofile` (depending on your operating system and configuration):
+
 ```bash
 export ALIBUILD_WORK_DIR="$HOME/alice/sw"
 eval "$(alienv shell-helper)"
 ```
+
 The first line tells what directory is used as "build cache", the second line installs a "shell helper" that makes easier to run certain aliBuild-related commands.
 
 You need to close and reopen your terminal for the change to be effective. The directory `~/alice/sw` will be created the first time you run aliBuild.
@@ -204,6 +212,8 @@ Note that this directory tends to grow in size over time, and it is the one you
 
 When `aliBuild`is installed on your computer and your prerequisits are statisfied, you can move to the next step.
 
+See also [shell rc utilities](../tools/README.md#shell-rc-utilities).
+
 ## Prepare your source code
 
 We assume your work area is `~/alice`.
@@ -318,6 +328,8 @@ For example: `ninja PWGCF/Tasks/install`
 ninja install > ../log 2>&1 && echo "Good" || echo "Bad"
 ```
 
+See also [shell rc utilities](../tools/README.md#shell-rc-utilities).
+
 A specific executable can be built in the staging directory `stage/bin` with
 
 ```bash

docs/tools/README.md

@@ -0,0 +1,137 @@
+---
+sort: 8
+title: Tools
+---
+
+# Tools
+
+Tools which are not part of the O2(Physics) analysis framework and can make your work with it much more effective.
+
+## Setup diagnostic tool
+
+This simple Bash script prints a summary of some basic information about your system and your O2Physics installation setup.
+
+It can be useful to provide this summary when [requesting support](../troubleshooting/README.md#reporting-problems) concerning a problem with your analysis framework.
+
+Download the [`summarise_o2p_setup.sh`](summarise_o2p_setup.sh) script and run it with `bash` in your `alice` directory.
+
+## Dependency finder
+
+Tool to explore input/output dependencies between O2Physics workflows and tables.
+
+See the dedicated [Dependency finder](dependencyFinder.md) page.
+
+## O2 linter
+
+Tool to detect O2-specific (and some common C++) issues in the code.
+
+See the dedicated [O2 linter](o2linter.md) page.
+
+## Pre-commit hooks
+
+[Git hooks](https://git-scm.com/book/ms/v2/Customizing-Git-Git-Hooks) are scripts hooked to certain Git commands.
+Pre-commit hooks run when you execute `git commit` and are used to validate the staged changes that are about to be committed.
+If any hook fails, the commit is not made.
+
+In the O2Physics repository, pre-commit hooks are [configured](https://github.com/AliceO2Group/O2Physics/blob/master/.pre-commit-config.yaml) to format C++ code with [clang-format](https://clang.llvm.org/docs/ClangFormat.html) and lint it (check for issues) with [cpplint](https://github.com/cpplint/cpplint).
+This way you can make sure that your changes will pass the C++ formatting check and the cpplint check in [MegaLinter](https://megalinter.io/) when you make your pull request.
+
+### How to use pre-commit hooks
+
+1. [Install pre-commit](https://pre-commit.com/#installation) (typically `pip install pre-commit`).
+1. Enter the `O2Physics` directory.
+1. [Install the Git hook scripts](https://pre-commit.com/#3-install-the-git-hook-scripts): `pre-commit install`.
+
+Next time you execute `git commit`, the hooks will run automatically.
+
+- If the clang-format hook fails, it means your staged files have been formatted.
+- If the cpplint hook fails, the linter has found issues that need to be fixed.
+- Updated files need to be staged with `git add` before attempting `git commit` again.
+
+## [clang-tidy](https://clang.llvm.org/extra/clang-tidy/)
+
+`clang-tidy` is a clang-based C++ linter tool for diagnosing and fixing typical programming errors, like style violations or bugs.
+(See the [list of checks](https://clang.llvm.org/extra/clang-tidy/checks/list.html).)
+
+### Prerequisites for using clang-tidy
+
+- You need to have O2Physics successfully compiled.
+- Verify that there is a valid symbolic link `compile_commands.json` in the O2Physics directory pointing to the `alice/sw/BUILD/.../O2Physics` directory.
+
+### Tips
+
+#### Cleaning `#include`s
+
+The [`misc-include-cleaner`](https://clang.llvm.org/extra/clang-tidy/checks/misc/include-cleaner.html) check can fix missing and unused `#include`s.
+This helps to apply the [Include What You Use](https://github.com/AliceO2Group/O2Physics/issues/8357) principle which allows to maintain header dependencies clean.
+
+#### Sorting `#include`s
+
+The [`llvm-include-order`](https://clang.llvm.org/extra/clang-tidy/checks/llvm/include-order.html) check sorts `#include`s from most specific to least specific to ensure that the headers does not have any hidden dependencies.
+
+Unfortunately, the [LLVM include style](https://llvm.org/docs/CodingStandards.html#include-style) is incompatible with the [Google include style](https://google.github.io/styleguide/cppguide.html#Names_and_Order_of_Includes) applied by `cpplint`.
+
+#### Testing (and fixing) many files at once
+
+Here is an example of how to run the `misc-include-cleaner` check in parallel on all `.h` and `.cxx` files in the current directory.
+
+```bash
+parallel "clang-tidy --fix -checks=-*,misc-include-cleaner {}; echo \"{} \$?\"" ::: $(find -name "*.h" -o -name "*.cxx") > "clang-tidy.log"
+```
+
+The [`parallel`](https://www.gnu.org/software/parallel/) command is used to parallelise the execution of the `clang-tidy` command for all files.
+
+For each file, `clang-tidy` will first try to compile it and then run the enabled check(s) and fix found problems (the `--fix` option).
+The messages are redirected into `clang-tidy.log`.
+The file name and the exit code are printed below the output of `clang-tidy` so that you can get the list of files for which `clang-tidy` failed with `grep " 1$" "clang-tidy.log"`.
+
+## [Visual Studio Code (VS Code)](https://code.visualstudio.com/)
+
+Integrated development environment
+
+See [how to run O2 linter from VS Code](o2linter.md#in-visual-studio-code)
+
+### Useful extensions
+
+- [clangd](https://marketplace.visualstudio.com/items?itemName=llvm-vs-code-extensions.vscode-clangd) - C/C++ completion, navigation, and insights
+- [json](https://marketplace.visualstudio.com/items?itemName=ZainChen.json) - Json for Visual Studio Code
+- [Markdown All in One](https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one) - All you need to write Markdown (keyboard shortcuts, table of contents, auto preview and more)
+- [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python) - Python language support with extension access points for IntelliSense (Pylance), Debugging (Python Debugger), linting, formatting, refactoring, unit tests, and more.
+- [Ruff](https://marketplace.visualstudio.com/items?itemName=charliermarsh.ruff) - A Visual Studio Code extension with support for the Ruff linter and formatter.
+- [ShellCheck](https://marketplace.visualstudio.com/items?itemName=timonwong.shellcheck) - Integrates ShellCheck into VS Code, a linter for Shell scripts.
+- [YAML](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml) - YAML Language Support by Red Hat, with built-in Kubernetes syntax support
+- [Remote - SSH](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh) - Open any folder on a remote machine using SSH and take advantage of VS Code's full feature set.
+- [PDF Viewer](https://marketplace.visualstudio.com/items?itemName=mathematic.vscode-pdf) - Portable document format (PDF) viewer for Visual Studio Code.
+
+## Shell rc utilities
+
+You can use the [`bashrc-alice.sh`](bashrc-alice.sh) file to add ALICE-related utilities in your Bash environment.
+
+The file provides:
+
+- variables needed to [configure aliBuild](../gettingstarted/installing.md#configure-alibuild)
+- utility functions for [recompiling with `ninja`](../gettingstarted/installing.md#building-partially-for-development-using-ninja)
+- utility functions for [debugging compilation and runtime failures](../troubleshooting/README.md#finding-problems)
+
+Add the [`bashrc-alice.sh`](bashrc-alice.sh) file in your home directory (`~`) and source it in your Bash environment by adding the following line in the `~/.bashrc` file.
+
+```bash
+source bashrc-alice.sh
+```
+
+## [Run 3 validation framework](https://github.com/AliceO2Group/Run3AnalysisValidation)
+
+The Run 3 validation framework is a tool for an easy execution, testing and validation of any O2Physics analysis code on large local data samples.
+
+It is extensively configurable and provides similar features as AliHyperloop, such as:
+
+- Dataset management
+- Support of linked-derived-data input
+- Automatic JSON configuration based on input properties
+- Automatic workflow topology generation based on "wagon" dependencies
+- Job parallelisation
+- Table saving
+- Output merging
+- Diagnostics
+
+Among several other utilities, it includes a [maintenance script](https://github.com/AliceO2Group/Run3AnalysisValidation?tab=readme-ov-file#keep-your-repositories-and-installations-up-to-date-and-clean) for automated maintenance of Git repositories and aliBuild packages.

docs/tools/bashrc-alice.sh

@@ -0,0 +1,86 @@
+#!/bin/bash
+
+# alice
+
+# aliBuild
+export ALIBUILD_WORK_DIR="$HOME/alice/sw"
+eval "$(alienv shell-helper)"
+
+# aliases
+alias loadali='alienv enter AliPhysics/latest'
+alias loado2='alienv enter O2/latest'
+alias loado2p='alienv enter O2Physics/latest'
+alias root='root -l'
+
+# Recompile an aliBuild development package with ninja.
+# Usage: recompile <package> <branch> <target>
+# Arguments <branch> and <target> are optional.
+# Command "recompile O2Physics" will invoke "ninja install" for the "master" branch of O2Physics.
+# Command "recompile O2Physics my-branch" will invoke "ninja install" for the "my-branch" branch of O2Physics.
+# Command "recompile O2Physics my-branch Common" will invoke "ninja Common/install" for the "my-branch" branch of O2Physics.
+recompile() {
+  # set -o xtrace # to print out each command
+  [ "$1" ] || { echo "Provide a package name"; return 1; }
+  package="$1"
+  branch="master"
+  [ "$2" ] && branch="$2"
+  target=""
+  target_name="all"
+  [ "$3" ] && { target="$3/"; target_name="$3"; }
+  dir_pwd=$(pwd)
+  dir_build="$ALIBUILD_WORK_DIR/BUILD/${package}-latest-${branch}/${package}"
+  log="$(dirname "$dir_build")/log"
+  log_err="$(dirname "$dir_build")/log_err"
+  cd "$dir_build" || { echo "Could not enter $dir_build"; return 1; }
+  direnv allow || { echo "Failed to allow direnv"; return 1; }
+  eval "$(direnv export "$SHELL")"
+  echo "Recompiling ${package}_${branch}_${target_name}..."
+  start=$(date +%s)
+  ninja "${target}install" > "$log" 2>&1
+  ec=$?
+  end=$(date +%s)
+  echo "Compilation exited with: $ec"
+  echo "See the log at: $log"
+  if [ "$ec" != "0"  ]; then
+    grep -e "FAILED:" -e "error:" "$log" > "$log_err"
+    echo "See the errors at: $log_err"
+  fi
+  echo "Took $((end - start)) seconds."
+  cd "$dir_pwd" || return 1
+  # set +o xtrace
+  return $ec
+}
+
+# Recompile O2 with ninja.
+recompile-o2() { recompile "O2" "$1" "$2"; }
+# Recompile O2Physics with ninja.
+recompile-o2p() { recompile "O2Physics" "$1" "$2"; }
+
+# Find the workflow that produces a given table.
+# Limited functionality. Use find_dependencies.py for a full search.
+find-o2-table-producer() {
+  # Check that we are inside the O2 or the O2Physics directory.
+  [[ "$PWD/" != *"/O2"*"/"* ]] && { echo "You must be inside the O2 or the O2Physics directory."; return 1; }
+  [ ! "$1" ] && { echo "Provide a table name."; return 1; }
+  # Find files that produce the table.
+  table="$1"
+  echo "Table $table is produced in:"
+  files=$(grep -r -i --include="*.cxx" "<aod::$table>" | grep -E 'Produces|Spawns' | cut -d: -f1 | sort -u)
+  for f in $files; do
+    # Extract the workflow name from the CMakeLists.txt in the same directory.
+    wf=$(grep -B 1 "$(basename "$f")" "$(dirname "$f")/CMakeLists.txt" | head -n 1 | cut -d\( -f2)
+    echo "$wf in $f"
+  done
+}
+
+# Find compilation error messages in a compilation log.
+debug-o2-compile() {
+  [ "$1" ] || { echo "Provide a log file"; return 1; }
+  grep -n -e "FAILED:" -e "error:" -e "warning:" "$1"
+}
+
+# Find runtime error messages in an execution log.
+debug-o2-run() {
+  [ "$1" ] || { echo "Provide a log file"; return 1; }
+  grep -n -e "\\[ERROR\\]" -e "\\[FATAL\\]" -e "segmentation" -e "Segmentation" -e "SEGMENTATION" -e "command not found" -e "Program crashed" -e "Error:" -e "Error in " -e "\\[WARN\\]" -e "Warning in " "$1"
+}