Running the correction

Launching the program

Always first enter AliRoot, and then the virtual environment.

To run the correction software, inside the tpcwithdnn/ directory type:

python steer_analysis.py

If you see some warnings like:

cling::DynamicLibraryManager::loadLibrary(): libGpad.so.6.16: cannot open shared object file: No such file or directory

they can be safely ignored.

To print all available command line options, type

python steer_analysis.py --help

Stages of the algorithm

Full correction consists of 3 stages:

prediction of global distortion fluctuations with Boosted Decision Trees (XGBoost library)
correction of the distortion fluctuations predicted by BDT
prediction of residual (local) distortion fluctuations with U-Net - a deep convolutional neural network

Specific stages can be (de-)activated with active flag under each category in config_model_parameters.yml.

NOTE: What is called 'event' in the code and in the instruction is not a single collision, but a 'snapshot' - a set of measurements (3D maps) for a given time point. Such a single measurement reflects actually the overlapping of many collisions.

You can define which steps of the an analysis you want to run in default.yml. Multiple steps can be active:

dotrain - train model from zero
doapply - use the trained model to make predictions
doplot - create quality assurance plots of prediction results
dobayes - perform Bayesian optimization to find the best model configuration - currently implemented only for BDT
doprofile - compare models trained with different numbers of events

The remaining options are for the ND Validation.

Configuration

The parameters of the ML analysis can be configured in config_model_parameters.yml. Most often used arguments:

dirmodel, dirapply, dirplots - directories where the trained models, prediction results and plots should be saved (the paths can be relative)
dirinput_bias, dirinput_nobias - paths to directories where the biased / unbiased input datasets are stored
- the unbiased input dataset is the original one created from MC data with the true expected density and distortion fluctuations.
- for the biased input dataset, the density of the random maps is constructed from existing maps, choosing randomly density z-slices and biasing them with harmonics. This method provided a more uniform distributions of relative density fluctuations across all random maps instead of the expected gaussian distribution. The method is described in https://indico.cern.ch/event/916148/contributions/3851980/attachments/2032772/3402755/Bias_presentation.pdf. This method is not used for the most recent training data.
grid_phi, grid_r, grid_z - grid granularity, usually 180x33x33 or higher
z_range - only space points with z_min <= z < z_max will be processed by the algorithm
opt_predout - the direction of distortions (dr, drphi, dz) to correct - currently only one direction can be processed at the same time
train_events, validation_events, apply_events - number of events for train / validation / apply, specified separately for 1D (BDT, NN) and 3D (CNN) fluctuations. You can specify multiple numbers, but the lists of values for train / validation / apply should be of equal length. Then, the program will run for each triple. If doprofile is specified, the program will output plots with prediction results (mean, std dev., mean + std dev.) gathered for each triple.

1D fluctuation correction (Boosted Decision Trees, neural network)

Three types of models can be chosen: random forest (RF) (XGBRFRegressor), xgboost (XGB) (XGBRegressor) or dense neural network (NN). For the BDTs, the default configuration uses the approximate 'hist' tree method, the fastest available in XGBoost. In config_model_parameters.yml, the model to use is given by the parameter param['xgboost']['xgbtype'] (the name refers to xgb for historical reasons) and has to assume on of the following values: XGB, RF, NN.

The general parameters for the 1D fluctuation correction can be configured in config_model_parameters.yml under 'xgboost', e.g.

downsample - whether to use the downsampling
downsample_npoints - number of random space points per map to downsample
plot_train, train_npoints - whether to plot the learning curve and with how many points
num_fourier_coeffs_train, num_fourier_coeffs_apply – number of Fourier coefficients to take from the 1D IDC train/apply(=predict) input

The model-specific parameters for the BDT and NN models, respectively, are given by param['xgboost']['params'] and param['xgboost']['nn_params'], respectively. For the BDTs, they come from the XGBoost Scikit-Learn API. Their meaning is described on the XGBoost page.

3D fluctuation correction (UNet)

The parameters are set in config_model_parameters.yml under 'dnn'.

filters - number of channels (filters) in the first convolutional block (the 3rd dimension of a 3D convolution)
pooling - type of pooling function: max - max pooling, avg - average pooling, conv - not an actual pooling but 3D convolution
depth - depth of the network = number of convolutional blocks / levels
batch normalization - whether to use batch normalization
dropout - fraction of dropout
batch_size - size of a batch
shuffle - whether to shuffle
epochs - number of epochs
lossfun - loss function
metrics - metrics, values measured besides the loss function that do not affect training
adamlr - learning rate for Adam optimizer

ND validation

ND validation parameters are explained in Validation.

Command line

Some parameters are available for a quick setup on the command line. You can check them with:

python steer_analysis.py -h

Correction output

debug: on the console
models: dirmodel
- XGBoost: JSON
- U-Net: JSON, network weights: h5
predictions: dirval
- a single ROOT file with histograms
prediction and profile plots: dirplot
indices of events in train / validation / apply partitions: dirmodel
- the indices are picked up by ND validator if any of these partitions is chosen for the ND validation
BDT feature importance plots and debug: dirmodel
dirmodel, dirval, dirplot are taken from config_model_parameters.yml