Update / add documentation for FST

Author: davidrohr

prodtests/full-system-test/documentation/dpl-workflow-options.md

@@ -0,0 +1,55 @@
+# Configuration options
+You can use the following options to change the workflow behavior:
+- `DDMODE` (default `processing`) : Must be `processing` (synchronous processing) or `processing-disk` (synchronous processing + storing of raw time frames to disk, note that this is the raw time frame not the CTF!). The `DDMODE` `discard` and `disk` are not compatible with the synchronous processing workflow, you must use the `no-processing.desc` workflow instead!.
+- `WORKFLOW_DETECTORS` (default `ALL`) : Comma-separated list of detectors for which the processing is enabled. If these are less detectors than participating in the run, data of the other detectors is ignored. If these are more detectors than participating in the run, the processes for the additional detectors will be started but will not do anything.
+- `WORKFLOW_DETECTORS_QC` (default `ALL`) : Comma-separated list of detectors for which to run QC, can be a subset of `WORKFLOW_DETECTORS` (for standalone detectors QC) and `WORKFLOW_DETECTORS_MATCHING` (for matching/vertexing QC). If a detector (matching/vertexing step) is not listed in `WORKFLOW_DETECTORS` (`WORKFLOW_DETECTORS_MATCHING`), the QC is automatically disabled for that detector. Only active if the `WORKFLOW_PARAMETER=QC` is set.
+- `WORKFLOW_DETECTORS_CALIB` (default `ALL`) : Comma-separated list of detectors for which to run calibration, can be a subset of `WORKFLOW_DETECTORS`. If a detector is not listed in `WORKFLOW_DETECTORS`, the calibration is automatically disabled for that detector. Only active if the `WORKFLOW_PARAMETER=CALIB` is set.
+- `WORKFLOW_DETECTORS_FLP_PROCESSING` (default `TOF` for sync processing on EPN, `NONE` otherwise) : Signals that these detectors have processing on the FLP enabled. The corresponding steps are thus inactive in the EPN epl-workflow, and the raw-proxy is configured to receive the FLP-processed data instead of the raw data in that case.
+- `WORKFLOW_DETECTORS_RECO` (default `ALL`) : Comma-separated list of detectors for which to run reconstruction.
+- `WORKFLOW_DETECTORS_CTF` (default `ALL`) : Comma-separated list of detectors to include in CTF.
+- `WORKFLOW_DETECTORS_MATCHING` (default selected corresponding to default workflow for sync or async mode respectively) : Comma-separated list of matching / vertexing algorithms to run. Use `ALL` to enable all of them. Currently supported options (see LIST_OF_GLORECO in common/setenv.h): `ITSTPC`, `TPCTRD`, `ITSTPCTRD`, `TPCTOF`, `ITSTPCTOF`, `MFTMCH`, `PRIMVTX`, `SECVTX`.
+- `WORKFLOW_EXTRA_PROCESSING_STEPS` Enable additional processing steps not in the preset for the SYNC / ASYNC mode. Possible values are: `MID_RECO` `MCH_RECO` `MFT_RECO` `FDD_RECO` `FV0_RECO` `ZDC_RECO` `ENTROPY_ENCODER` `MATCH_ITSTPC` `MATCH_TPCTRD` `MATCH_ITSTPCTRD` `MATCH_TPCTOF` `MATCH_ITSTPCTOF` `MATCH_MFTMCH` `MATCH_MFTMCH` `MATCH_PRIMVTX` `MATCH_SECVTX`. (Here `_RECO` means full async reconstruction, and can be used to enable it also in sync mode.)
+- `WORKFLOW_PARAMETERS` (default `NONE`) : Comma-separated list, enables additional features of the workflow. Currently the following features are available:
+  - `GPU` : Performs the TPC processing on the GPU, otherwise everything is processed on the CPU.
+  - `CTF` : Write the CTF to disk (CTF creation is always enabled, but if this parameter is missing, it is not stored).
+  - `EVENT_DISPLAY` : Enable JSON export for event display.
+  - `QC` : Enable QC.
+  - `CALIB` : Enable calibration (not yet working!)
+- `RECO_NUM_NODES_OVERRIDE` (default `0`) : Overrides the number of EPN nodes used for the reconstruction (`0` or empty means default).
+- `MULTIPLICITY_FACTOR_RAWDECODERS` (default `1`) : Scales the number of parallel processes used for raw decoding by this factor.
+- `MULTIPLICITY_FACTOR_CTFENCODERS` (default `1`) : Scales the number of parallel processes used for CTF encoding by this factor.
+- `MULTIPLICITY_FACTOR_REST` (default `1`) : Scales the number of other reconstruction processes by this factor.
+- `QC_JSON_EXTRA` (default `NONE`) : extra QC jsons to add (if does not fit to those defined in WORKFLOW_DETECTORS_QC & (WORKFLOW_DETECTORS | WORKFLOW_DETECTORS_MATCHING)
+Most of these settings are configurable in the AliECS GUI. But some of the uncommon settings (`WORKFLOW_DETECTORS_FLP_PROCESSING`, `WORKFLOW_DETECTORS_CTF`, `WORKFLOW_DETECTORS_RECO`, `WORKFLOW_DETECTORS_MATCHING`, `WORKFLOW_EXTRA_PROCESSING_STEPS`, advanced `MULTIPLICITY_FACTOR` settings) can only be set via the "Additional environment variables field" in the GUI using bash syntax, e.g. `WORKFLOW_DETECTORS_FLP_PROCESSING=TPC`.
+
+# Process multiplicity factors
+- The production workflow has internally a default value how many instances of a process to run in parallel (which was tuned for Pb-Pb processing)
+- Some critical processes for synchronous pp processing are automatically scaled by the inverse of the number of nodes, i.e. the multiplicity is increased by a factor of 2 if 125 instead of 250 nodes are used, to enable the processing using only a subset of the nodes.
+- Factors can be provided externally to scale the multiplicity of processes further. All these factors are multiplied.
+  - One factor can be provided based on the type of the processes: raw decoder (`MULTIPLICITY_FACTOR_RAWDECODERS`), CTF encoder (`MULTIPLICITY_FACTOR_CTFENCODERS`), or other reconstruction process (`MULTIPLICITY_FACTOR_REST`)
+  - One factor can be provided per detector via `MULTIPLICITY_FACTOR_DETECTOR_[DET]` using the 3 character detector representation, or `MATCH` for the global matching and vertexing workflows.
+  - One factor can be provided per process via `MULTIPLICITY_FACTOR_PROCESS_[PROCESS_NAME]`. In the process name, dashes `-` must be replaced by underscores `_`.
+- The multiplicity of an individual process can be overridden externally (this is an override, no scaling factor) by using `MULTIPLICITY_PROCESS_[PROCESS_NAME]`. In the process name, dashes `-` must be replaced by underscores `_`.
+- For example, creating the workflow with `MULTIPLICITY_FACTOR_RAWDECODERS=2 MULTIPLICITY_FACTOR_DETECTOR_ITS=3 MULTIPLICITY_FACTOR_PROCESS_mft_stf_decoder=5` will scale the number of ITS raw decoders by 6, of other ITS processes by 3, of other raw decoders by 2, and will run exactly 5 `mft-stf-decoder` processes.
+
+# Additional custom control variables
+For user modification of the workflow settings, the folloing *EXTRA* environment variables exist:
+- `ARGS_ALL_EXTRA` : Extra command line options added to all workflows
+- `ALL_EXTRA_CONFIG` : Extra config key values added to all workflows
+- `GPU_EXTRA_CONFIG` : Extra options added to the configKeyValues of the GPU workflow
+- `ARGS_EXTRA_PROCESS_[WORKFLOW_NAME]` : Extra command line arguments for the workflow binary `WORKFLOW_NAME`. Dashes `-` must be replaced by underscores `_` in the name! E.g. `ARGS_EXTRA_PROCESS_o2_tof_reco_workflow='--output-type clusters'`
+- `CONFIG_EXTRA_PROCESS_[WORKFLOW_NAME]` : Extra `--configKeyValues` arguments for the workflow binary `WORKFLOW_NAME`. Dashes `-` must be replaced by underscores `_` in the name! E.g. `CONFIG_EXTRA_PROCESS_o2_gpu_reco_workflow='GPU_proc.debugLevel=1;GPU_proc.ompKernels=0;'`
+
+**IMPORTANT:** When providing additional environment variables please always use single quotes `'` instead of double quotes `"`, because otherwise there can be issues with whitespaces. E.g. `ARGS_EXTRA_PROCESS_o2_eve_display='--filter-time-min 0 --filter-time-max 120'` does work while `ARGS_EXTRA_PROCESS_o2_eve_display="--filter-time-min 0 --filter-time-max 120"` does not.
+
+In case the CTF dictionaries were created from the data drastically different from the one being compressed, the default memory allocation for the CTF buffer might be insufficient. One can apply scaling factor to the buffer size estimate (default=1.5) of particular detector by defining variable e.g. `TPC_ENC_MEMFACT=3.5`
+
+# File input for ctf-reader / raw-tf-reader
+- The variable `$INPUT_FILE_LIST` can be a comma-seperated list of files, or a file with a file-list of CTFs/raw TFs.
+- The variable `$INPUT_FILE_COPY_CMD` can provide a custom copy command (default is to fetch the files from EOS).
+
+# Remarks on QC
+The JSON files for the individual detectors are merged into one JSON file, which is cached during the run on the shared EPN home folder.
+The default JSON file per detector is defined in `qc-workflow.sh`.
+JSONs per detector can be overridden by exporting `QC_JSON_[DETECTOR_NAME]`, e.g. `QC_JSON_TPC`, when creating the workflow.
+The global section of the merged qc JSON config is taken from qc-sync/qc-global.json

prodtests/full-system-test/documentation/env-variables.md

@@ -0,0 +1,51 @@
+The `setenv-sh` script sets the following environment options
+* `NTIMEFRAMES`: Number of time frames to process.
+* `TFDELAY`: Delay in seconds between publishing time frames (1 / rate).
+* `NGPUS`: Number of GPUs to use, data distributed round-robin.
+* `GPUTYPE`: GPU Tracking backend to use, can be CPU / CUDA / HIP / OCL / OCL2.
+* `SHMSIZE`: Size of the global shared memory segment.
+* `DDSHMSIZE`: Size of shared memory unmanaged region for DataDistribution Input.
+* `GPUMEMSIZE`: Size of allocated GPU memory (if GPUTYPE != CPU)
+* `HOSTMEMSIZE`: Size of allocated host memory for GPU reconstruction (0 = default).
+  * For `GPUTYPE = CPU`: TPC Tracking scratch memory size. (Default 0 -> dynamic allocation.)
+  * Otherwise : Size of page-locked host memory for GPU processing. (Defauls 0 -> 1 GB.)
+* `CREATECTFDICT`: Create CTF dictionary.
+* `SAVECTF`: Save the CTF to a root file.
+  * 0: Read `ctf_dictionary.root` as input.
+  * 1: Create `ctf_dictionary.root`. Note that this was already done automatically if the raw data was simulated with `full_system_test.sh`.
+* `SYNCMODE`: Run only reconstruction steps of the synchronous reconstruction.
+  * Note that there is no `ASYNCMODE` but instead the `CTFINPUT` option already enforces asynchronous processing.
+* `NUMAGPUIDS`: NUMAID-aware GPU id selection. Needed for the full EPN configuration with 8 GPUs, 2 NUMA domains, 4 GPUs per domain.
+  In this configuration, 2 instances of `dpl-workflow.sh` must run in parallel.
+  To be used in combination with `NUMAID` to select the id per workflow.
+  `start_tmux.sh` will set up these variables automatically.
+* `NUMAID`: SHM segment id to use for shipping data as well as set of GPUs to use (use `0` / `1` for 2 NUMA domains, 0 = GPUS `0` to `NGPUS - 1`, 1 = GPUS `NGPUS` to `2 * NGPUS - 1`)
+* 0: Runs all reconstruction steps, of sync and of async reconstruction, using raw data input.
+* 1: Runs only the steps of synchronous reconstruction, using raw data input.
+* `EXTINPUT`: Receive input from raw FMQ channel instead of running o2-raw-file-reader.
+  * 0: `dpl-workflow.sh` can run as standalone benchmark, and will read the input itself.
+  * 1: To be used in combination with either `datadistribution.sh` or `raw-reader.sh` or with another DataDistribution instance.
+* `CTFINPUT`: Read input from CTF ROOT file. This option is incompatible to EXTINPUT=1. The CTF ROOT file can be stored via SAVECTF=1.
+* `NHBPERTF`: Time frame length (in HBF)
+* `GLOBALDPLOPT`: Global DPL workflow options appended to o2-dpl-run.
+* `EPNPIPELINES`: Set default EPN pipeline multiplicities.
+  Normally the workflow will start 1 dpl device per processor.
+  For some of the CPU parts, this is insufficient to keep step with the GPU processing rate, e.g. one ITS-TPC matcher on the CPU is slower than the TPC tracking on multiple GPUs.
+  This option adds some multiplicies for CPU processes using DPL's pipeline feature.
+  The settings were tuned for EPN processing with 4 GPUs (i.e. the default multiplicities are per NUMA domain).
+  The multiplicities are scaled with the `NGPUS` setting, i.e. with 1 GPU only 1/4th are applied.
+  You can pass an option different to 1, and than it will be applied as factor on top of the multiplicities.
+  It is auto-selected by `start-tmux.sh`.
+* `SEVERITY`: Log verbosity (e.g. info or error, default: info)
+* `INFOLOGGER_SEVERITY`: Min severity for messages sent to Infologger. (default: `$SEVERITY`)
+* `SHMTHROW`: Throw exception when running out of SHM memory.
+  It is suggested to leave this enabled (default) on tests on the laptop to get an actual error when it runs out of memory.
+  This is disabled in `start_tmux.sh`, to avoid breaking the processing while there is a chance that another process might free memory and we can continue.
+* `NORATELOG`: Disable FairMQ Rate Logging.
+* `INRAWCHANNAME`: FairMQ channel name used by the raw proxy, must match the name used by DataDistribution.
+* `WORKFLOWMODE`: run (run the workflow (default)), print (print the command to stdout), dds (create partial DDS topology)
+* `FILEWORKDIR`: directory for all input / output files. E.g. grp / geometry / dictionaries etc. are read from here, and dictionaries / ctf / etc. are written to there.
+  Some files have more fine grained control via other environment variables (e.g. to store the CTF to somewhere else). Such variables are initialized to `$FILEWORKDIR` by default but can be overridden.
+* `EPNSYNCMODE`: Specify that this is a workflow running on the EPN for synchronous processing, e.g. logging goes to InfoLogger, DPL metrics to to the AliECS monitoring, etc.
+* `BEAMTYPE`: Beam type, must be PbPb, pp, pPb, cosmic, technical.
+* `IS_SIMULATED_DATA` : 1 for MC data, 0 for RAW data.

prodtests/full-system-test/documentation/full-system-test-as-stress-test.md

@@ -0,0 +1,33 @@
+This is a quick summary how to run the full system test (FST) as stress test on the EPN. (For the full FST documentation, see https://github.com/AliceO2Group/AliceO2/blob/dev/prodtests/full-system-test/documentation/full-system-test-setup.md and https://github.com/AliceO2Group/AliceO2/blob/dev/prodtests/full-system-test/documentation/full-system-test.md)
+
+# Preparing the data set
+- I usually try to keep an up-to-date data set that can be used in `/home/drohr/alitest/tmp-fst*`. The folder with the highest number is the latest dataset. However, data formats are still evolving, and it requires rerunning the simulation regularly. I.e. please try my latest data set, if it doesn't work, please generate a new one as described below.
+- Short overview how to generate a FST Pb-Pb 128 orbit data set:
+  - The O2 binaries installed on the EPN via RPMs use the `o2-dataflow` defaults and cannot run the simulation, and also they lack readout. Thus you need to build `O2PDPSuite` and `Readout` (the version matching the O2PDPSuite RPM you want to use for running the test) yourself with `alibuild` on an EPN: `aliBuild --defaults o2 build O2PDPSuite Readout --jobs 32 --debug`. The flag `--jobs` configures the number of parallel jobs and can be changed.
+  - Enter the O2PDPSuite environment either vie `alienv enter O2PDPSuite/latest Readout/latest`.
+  - Go to an empty directory.
+  - Run the FST simulation via: `NEvents=650 NEventsQED=10000 SHMSIZE=128000000000 TPCTRACKERSCRATCHMEMORY=40000000000 SPLITTRDDIGI=0 GENERATE_ITSMFT_DICTIONARIES=1 $O2_ROOT/prodtests/full_system_test.sh`
+  - Get a current matbud.root (e.g. from here https://alice.its.cern.ch/jira/browse/O2-2288) and place it in that folder.
+  - Create a timeframe file from the raw files: `$O2_ROOT/prodtests/full-system-test/convert-raw-to-tf-file.sh`.
+  - Prepare the ramdisk folder: `mv raw/timeframe raw/timeframe-org; mkdir raw/timeframe-tmpfs; ln -s timeframe-tmpfs raw/timeframe`
+
+# Running the full system test
+- Enter the environment! On an EPN do `module load O2PDPSuite` (this will load the latest O2 software installed on that EPN).
+- Go into the folder with the data set (you might need to create one, see above).
+- Prepare the ramdisk with the data: `sudo mount -t tmpfs tmpfs raw/timeframe-tmpfs; sudo cp raw/timeframe-org/* raw/timeframe`
+  - (NOTE that the ramdisk might already be present from previous tests, or in a different folder. Check the mounted tmpfs filesystems (`mount | grep tmpfs`), and don't mount multiple of them since memory is critical!)
+  - If you do not have root permissions and cannot create a ramdisk, the test will also work without. In that case you should decrease the publishing rate below to `TFDELAY=5`.
+- Make sure disk caches are cleared: as ROOT do: `echo 1 > /proc/sys/vm/drop_caches`
+- In order to run the Full System Test, the workflow must be able to access the CCDB. Normally, if you run as user, you must make sure to have an alien token present. On the EPN, one can use the EPN-internal CCDB server instead, which does not require alien access. If you use the `start-tmux.sh`, the env variables are set automatically to access the EPN-internal CCDB server.
+- Start the FST with 2 NUMA domains: `TFDELAY=2.5 NTIMEFRAMES=1000000 $O2_ROOT/prodtests/full-system-test/start_tmux.sh dd`
+
+This will start a tmux session with 3 shells, the upper 2 shells are the 2 DPL workflows, one per NUMA domain, for the processing. The lower shell is the input with DataDistribution's StfBuilder. Leave it running and check that the StfBuilder doesn't complain that its buffer is full. Then the EPN can sustain the rate.
+
+# **NOTE**
+- Attached to this ticket is a screenshot of how the console should look like:
+  - The DD console (on the bottom) should not show warnings about full buffers.
+  - The other 2 consoles (1 per NUMA domain) should show the processing times per TF for the GPU reconstruction:
+    ```
+    [2974450:gpu-reconstruction_t3]: [10:50:38][INFO] GPU Reoncstruction time for this TF 26.77 s (cpu), 17.8823 s (wall)
+    ```
+    This should be 17 to 18 seconds, and you should see it for all 4 GPUs on both NUMA domains (`reconstruction_t0` to `reconstruction_t3`)