Skip to content

Commit ccdd1cb

Browse files
ariostasVourMa
ariostas
and
VourMa
committed
Improvements to the README
Co-authored-by: Manos Vourliotis <[email protected]>
1 parent 46da965 commit ccdd1cb

File tree

1 file changed

+85
-191
lines changed

1 file changed

+85
-191
lines changed

‎RecoTracker/LSTCore/standalone/README.md‎

Lines changed: 85 additions & 191 deletions
Original file line numberDiff line numberDiff line change
@@ -1,53 +1,93 @@
1-
# TrackLooper
1+
# How to set up standalone LST
22

3-
4-
## Quick Start
5-
6-
7-
### Setting up LSTPerformanceWeb (only for lnx7188 and lnx4555)
3+
## Setting up LSTPerformanceWeb (only for lnx7188 and lnx4555)
84

95
For lnx7188 and lnx4555 this needs to be done once
106

117
cd /cdat/tem/${USER}/
128
git clone [email protected]:SegmentLinking/LSTPerformanceWeb.git
139

14-
### Setting up container (only for lnx7188)
10+
## Setting up container (only for lnx7188)
1511

1612
For lnx7188 this needs to be done before compiling or running the code:
1713

1814
singularity shell --nv --bind /mnt/data1:/data --bind /data2/segmentlinking/ --bind /opt --bind /nfs --bind /mnt --bind /usr/local/cuda/bin/ --bind /cvmfs /cvmfs/unpacked.cern.ch/registry.hub.docker.com/cmssw/el8:x86_64
1915

20-
### Setting up the code
16+
## Setting up LST
17+
18+
There are two way to set up LST as a standalone, either by setting up a full CMSSW area, which provides a unified setup for standalone and CMSSW tests, or by `sparse-checkout` only the relevant package and using them independent of CMSSW. A CVMFS-less setup is also provided for the second option.
19+
20+
### Setting up LST within CMSSW (preferred option)
21+
22+
```bash
23+
CMSSW_VERSION=CMSSW_14_2_0_pre4 # Change with latest/preferred CMSSW version
24+
cmsrel ${CMSSW_VERSION}
25+
cd ${CMSSW_VERSION}/src/
26+
cmsenv
27+
git cms-init
28+
# If necessary, add the remote [email protected]:SegmentLinking/cmssw.git
29+
# and checkout a development/feature branch
30+
git cms-addpkg RecoTracker/LST RecoTracker/LSTCore
31+
# If modifying some dependencies, run `git cms-checkdeps -a -A`
32+
scram b -j 12
33+
cd RecoTracker/LSTCore/standalone
34+
```
35+
36+
The data files for LST will be fetched from CVMFS. However, if new data files are needed, the need to be manually placed (under `$CMSSW_BASE/external/$SCRAM_ARCH/data/RecoTracker/LSTCore/data/`). This is done by running:
2137

22-
git clone [email protected]:SegmentLinking/TrackLooper.git
23-
cd TrackLooper/
24-
# Source one of the commands below, depending on the site
25-
source setup.sh # if on UCSD or Cornell
26-
source setup_hpg.sh # if on Florida
38+
```bash
39+
mkdir -p $CMSSW_BASE/external/$SCRAM_ARCH/data/RecoTracker/LSTCore/
40+
cd $CMSSW_BASE/external/$SCRAM_ARCH/data/RecoTracker/LSTCore/
41+
git clone [email protected]:cms-data/RecoTracker-LSTCore.git data
42+
<modify the files or checkout a different branch>
43+
cd -
44+
```
2745

28-
### Running the code
46+
### Setting up LST outside of CMSSW
2947

30-
sdl_make_tracklooper -mc
31-
sdl_<backend> -i PU200 -o LSTNtuple.root
48+
For this setup, dependencies are still provided from CMSSW through CVMFS but no CMSSW area is setup. This is done by running the following commands.
49+
50+
``` bash
51+
LST_BRANCH=master # Change to the development branch
52+
git clone --filter=blob:none --no-checkout --depth 1 --sparse --branch ${LST_BRANCH} https://github.com/SegmentLinking/cmssw.git TrackLooper
53+
cd TrackLooper
54+
git sparse-checkout add RecoTracker/LSTCore
55+
git checkout
56+
cd RecoTracker/LSTCore/standalone/
57+
```
58+
59+
As in the sectino above, the data files are fetched from CVMFS, but they can also be copied manually under `RecoTracker/LSTCore/data/`.
60+
61+
62+
## Running the code
63+
64+
Each time the standalone version of LST is to be used, the following command should be run from the `RecoTracker/LSTCore/standalone` directory:
65+
```bash
66+
source setup.sh
67+
```
68+
69+
For running the code:
70+
71+
lst_make_tracklooper -m
72+
lst_<backend> -i PU200 -o LSTNtuple.root
3273
createPerfNumDenHists -i LSTNtuple.root -o LSTNumDen.root
33-
lst_plot_performance.py LSTNumDen.root -t "myTag"
34-
# python3 efficiency/python/lst_plot_performance.py LSTNumDen.root -t "myTag" # if you are on cgpu-1 or Cornell
74+
lst_plot_performance.py LSTNumDen.root -t "myTag" # or
75+
python3 efficiency/python/lst_plot_performance.py LSTNumDen.root -t "myTag" # if you are on cgpu-1 or Cornell
3576

3677
The above can be even simplified
3778

38-
sdl_run -f -mc -s PU200 -n -1 -t myTag
79+
lst_run -f -m -s PU200 -n -1 -t myTag
3980

4081
The `-f` flag can be omitted when the code has already been compiled. If multiple backends were compiled, then the `-b` flag can be used to specify a backend. For example
4182

42-
sdl_run -b cpu -s PU200 -n -1 -t myTag
83+
lst_run -b cpu -s PU200 -n -1 -t myTag
4384

44-
## Command explanations
85+
### Command explanations
4586

4687
Compile the code with option flags. If none of `C,G,R,A` are used, then it defaults to compiling for CUDA and CPU.
4788

48-
sdl_make_tracklooper -mc
89+
lst_make_tracklooper -m
4990
-m: make clean binaries
50-
-c: run with the cmssw caching allocator
5191
-C: compile CPU backend
5292
-G: compile CUDA backend
5393
-R: compile ROCm backend
@@ -56,7 +96,7 @@ Compile the code with option flags. If none of `C,G,R,A` are used, then it defau
5696

5797
Run the code
5898

59-
sdl_<backend> -n <nevents> -v <verbose> -w <writeout> -s <streams> -i <dataset> -o <output>
99+
lst_<backend> -n <nevents> -v <verbose> -w <writeout> -s <streams> -i <dataset> -o <output>
60100

61101
-i: PU200; muonGun, etc
62102
-n: number of events; default: all
@@ -106,186 +146,40 @@ Comparing two different runs
106146
-t "mywork" \
107147
--compare
108148

109-
## CMSSW Integration
110-
This is the a complete set of instruction on how the TrackLooper code
111-
can be linked as an external tool in CMSSW:
112-
113-
### Build TrackLooper
114-
```bash
115-
git clone [email protected]:SegmentLinking/TrackLooper.git
116-
cd TrackLooper/
117-
# Source one of the commands below, depending on the site
118-
source setup.sh # if on UCSD or Cornell
119-
source setup_hpg.sh # if on Florida
120-
sdl_make_tracklooper -mc
121-
cd ..
122-
```
123-
124-
### Set up `TrackLooper` as an external
125-
```bash
126-
mkdir workingFolder # Create the folder you will be working in
127-
cd workingFolder
128-
cmsrel CMSSW_14_1_0_pre3
129-
cd CMSSW_14_1_0_pre3/src
130-
cmsenv
131-
git cms-init
132-
git remote add SegLink [email protected]:SegmentLinking/cmssw.git
133-
git fetch SegLink CMSSW_14_1_0_pre3_LST_X
134-
git cms-addpkg RecoTracker Configuration
135-
git checkout CMSSW_14_1_0_pre3_LST_X
136-
#To include both the CPU library and GPU library into CMSSW, create 3 xml files (headers file has no library).
137-
#Before writing the following xml file, check that libsdl_cpu.so and libsdl_gpu.so can be found under the ../../../TrackLooper/SDL/ folder.
138-
cat <<EOF >lst_headers.xml
139-
<tool name="lst_headers" version="1.0">
140-
<client>
141-
<environment name="LSTBASE" default="$PWD/../../../TrackLooper"/>
142-
<environment name="INCLUDE" default="\$LSTBASE"/>
143-
</client>
144-
<runtime name="LST_BASE" value="\$LSTBASE"/>
145-
</tool>
146-
EOF
147-
cat <<EOF >lst_cpu.xml
148-
<tool name="lst_cpu" version="1.0">
149-
<client>
150-
<environment name="LSTBASE" default="$PWD/../../../TrackLooper"/>
151-
<environment name="LIBDIR" default="\$LSTBASE/SDL"/>
152-
<environment name="INCLUDE" default="\$LSTBASE"/>
153-
</client>
154-
<runtime name="LST_BASE" value="\$LSTBASE"/>
155-
<lib name="sdl_cpu"/>
156-
</tool>
157-
EOF
158-
cat <<EOF >lst_cuda.xml
159-
<tool name="lst_cuda" version="1.0">
160-
<client>
161-
<environment name="LSTBASE" default="$PWD/../../../TrackLooper"/>
162-
<environment name="LIBDIR" default="\$LSTBASE/SDL"/>
163-
<environment name="INCLUDE" default="\$LSTBASE"/>
164-
</client>
165-
<runtime name="LST_BASE" value="\$LSTBASE"/>
166-
<lib name="sdl_cuda"/>
167-
</tool>
168-
EOF
169-
scram setup lst_headers.xml
170-
scram setup lst_cpu.xml
171-
scram setup lst_cuda.xml
172-
cmsenv
173-
git cms-checkdeps -a -A
174-
scram b -j 12
175-
```
176-
177-
### Run the LST reconstruction in CMSSW
178-
A simple test configuration of the LST reconstruction can be run with the command:
179-
```bash
180-
cmsRun RecoTracker/LST/test/LSTAlpakaTester.py
181-
```
149+
# How to set up CMSSW with LST
182150

183-
For a more complete workflow, one can run a modified version of the 21034.1 workflow.
184-
To get the commands of this workflow, one can run:
185-
```bash
186-
runTheMatrix.py -w upgrade -n -e -l 21034.1
187-
```
151+
## Setting up the area
188152

189-
For convenience, the workflow has been run for 100 events and the output is stored here:
190-
```bash
191-
/data2/segmentlinking/CMSSW_14_1_0_pre0/step2_21034.1_100Events.root
192-
```
153+
Follow the instructions in the ["Setting up LST within CMSSW" section](#setting-up-lst-within-cmssw-preferred-option).
193154

194-
For enabling the LST reconstruction in the CMSSW tracking workflow, a modified step3 needs to be run.
195-
This is based on the step3 command of the 21034.1 workflow with the following changes:
196-
- Remove the `--pileup_input` and `--pileup` flags.
197-
- The number of threads and streams for the job can be optionally controlled by the `--nThreads` and `--nStreams` command line options respectively (`1` ends up being the actual default value for both, and more info can be found by running `cmsDriver.py --help`).
198-
- Add at the end of the command: `--procModifiers gpu,trackingLST,trackingIters01 --no_exec`
199-
200-
Run the command and modify the output configuration file with the following:
201-
- If want to run a cpu version, remove the ```gpu``` in the line defining the `process` object:
202-
```python
203-
process = cms.Process('RECO',...,gpu,...)
204-
```
205-
- Add the following lines below the part where the import of the standard configurations happens:
206-
```python
207-
process.load('Configuration.StandardSequences.Accelerators_cff')
208-
process.load("HeterogeneousCore.AlpakaCore.ProcessAcceleratorAlpaka_cfi")
209-
```
210-
- Modify the input and output file names accordingly, as well as the number of events.
211-
212-
Then, run the configuration file with `cmsRun`.
213-
214-
To get the DQM files, one would have to run step4 of the 21034.1 workflow with the following modifications:
215-
- Add `--no_exec` to the end of command and then run it.
216-
- Modify the output configuration file by changing the input file (the one containing `inDQM` from the previous step) and number of events accordingly.
155+
## Run the LST reconstruction in CMSSW (read to the end, before running)
217156

218-
Running the configuration file with `cmsRun`, the output file will have a name starting with `DQM`. The name is the same every time this step runs,
219-
so it is good practice to rename the file, e.g. to `tracking_Iters01LST.root`.
220-
The MTV plots can be produced with the command:
221-
```bash
222-
makeTrackValidationPlots.py --extended tracking_Iters01LST.root
223-
```
224-
Comparison plots can be made by including multiple ROOT files as arguments.
157+
Two complete workflows have been implemented within CMSSW to run a two-iteration, tracking-only reconstruction with LST:
158+
- 24834.703 (CPU)
159+
- 24834.704 (GPU)
225160

226-
**Note:** In case one wants to run step2 as well, similar modifications as in step4 (`--no_exec` flag and input file/number of events) need to be applied. Moreover, the PU files have better be modified to point to local ones. This can be done by inserting a dummy file when running the command (set the argument of the `--pileup_input` flag to `file:file.root`), and then change the PU input files in the configuration to the following line (by means of replacing the corresponding line in the configuration):
227-
```python
228-
process.mix.input.fileNames = cms.untracked.vstring(['file:/data2/segmentlinking/PUSamplesForCMSSW1263/CMSSW_12_3_0_pre5/RelValMinBias_14TeV/GEN-SIM/123X_mcRun4_realistic_v4_2026D88noPU-v1/066fc95d-1cef-4469-9e08-3913973cd4ce.root', 'file:/data2/segmentlinking/PUSamplesForCMSSW1263/CMSSW_12_3_0_pre5/RelValMinBias_14TeV/GEN-SIM/123X_mcRun4_realistic_v4_2026D88noPU-v1/07928a25-231b-450d-9d17-e20e751323a1.root', 'file:/data2/segmentlinking/PUSamplesForCMSSW1263/CMSSW_12_3_0_pre5/RelValMinBias_14TeV/GEN-SIM/123X_mcRun4_realistic_v4_2026D88noPU-v1/26bd8fb0-575e-4201-b657-94cdcb633045.root', 'file:/data2/segmentlinking/PUSamplesForCMSSW1263/CMSSW_12_3_0_pre5/RelValMinBias_14TeV/GEN-SIM/123X_mcRun4_realistic_v4_2026D88noPU-v1/4206a9c5-44c2-45a5-aab2-1a8a6043a08a.root', 'file:/data2/segmentlinking/PUSamplesForCMSSW1263/CMSSW_12_3_0_pre5/RelValMinBias_14TeV/GEN-SIM/123X_mcRun4_realistic_v4_2026D88noPU-v1/55a372bf-a234-4111-8ce0-ead6157a1810.root', 'file:/data2/segmentlinking/PUSamplesForCMSSW1263/CMSSW_12_3_0_pre5/RelValMinBias_14TeV/GEN-SIM/123X_mcRun4_realistic_v4_2026D88noPU-v1/59ad346c-f405-4288-96d7-795f81c43fe8.root', 'file:/data2/segmentlinking/PUSamplesForCMSSW1263/CMSSW_12_3_0_pre5/RelValMinBias_14TeV/GEN-SIM/123X_mcRun4_realistic_v4_2026D88noPU-v1/7280f5ec-b71d-4579-a730-7ce2de0ff906.root', 'file:/data2/segmentlinking/PUSamplesForCMSSW1263/CMSSW_12_3_0_pre5/RelValMinBias_14TeV/GEN-SIM/123X_mcRun4_realistic_v4_2026D88noPU-v1/b93adc85-715f-477a-afc9-65f3241933ee.root', 'file:/data2/segmentlinking/PUSamplesForCMSSW1263/CMSSW_12_3_0_pre5/RelValMinBias_14TeV/GEN-SIM/123X_mcRun4_realistic_v4_2026D88noPU-v1/c7a0aa46-f55c-4b01-977f-34a397b71fba.root', 'file:/data2/segmentlinking/PUSamplesForCMSSW1263/CMSSW_12_3_0_pre5/RelValMinBias_14TeV/GEN-SIM/123X_mcRun4_realistic_v4_2026D88noPU-v1/e77fa467-97cb-4943-884f-6965b4eb0390.root'])
229-
```
161+
We will use the second one in the example below. To get the commands of this workflow, one can run:
230162

231-
### Inclusion of LST in other CMSSW packages
232-
Including the line
233-
```
234-
<use name="lst"/>
235-
```
236-
in the relevant package `BuildFile.xml` allows for
237-
including our headers in the code of that package.
163+
runTheMatrix.py -w upgrade -n -e -l 24834.704
238164

239-
## Running LST in a CVMFS-less setup
165+
For convenience, the workflow has been run for 100 events and the output is stored here:
240166

241-
The setup scripts included in this repository assume that the [CernVM File System (CVMFS)](https://cernvm.cern.ch/fs/) is installed. This provides a convenient way to fetch the required dependencies, but it is not necessary to run LST in standalone mode. Here, we briefly describe how to build and run it when CVMFS is not available.
167+
/data2/segmentlinking/step2_29834.1_100Events.root
242168

243-
The necessary dependencies are CUDA, ROOT, the Boost libraries, Alpaka, and some CMSSW headers. CUDA, ROOT, and Boost, are fairly standard libraries and are available from multiple package managers. For the remaining necessary headers you will need to clone the [Alpaka](https://github.com/alpaka-group/alpaka) and [CMSSW](https://github.com/cms-sw/cmssw) repositories. The Alpaka repository is reasonably sized, but the CMSSW one extremely large, especially considering that we only need a tiny fraction of its files to build LST. We can get only the Alpaka interface headers from CMSSW by running the following commands.
169+
The input files in each step may need to be properly adjusted to match the ones produced by the previous step/provided externally, hence it is better to run the commands with the `--no_exec` option included.
244170

245-
``` bash
246-
git clone --filter=blob:none --no-checkout --depth 1 --sparse --branch CMSSW_14_1_X https://github.com/cms-sw/cmssw.git
247-
cd cmssw
248-
git sparse-checkout add HeterogeneousCore/AlpakaInterface
249-
git checkout
250-
```
171+
Running the configuration file with `cmsRun`, the output file will have a name starting with `DQM`. The name is the same every time this step runs,
172+
so it is good practice to rename the file, e.g. to `step4_24834.704.root`.
173+
The MTV plots can be produced with the command:
251174

252-
Then all that is left to do is set some environment variables. We give an example of how to do this in lnx7188/cgpu-1.
175+
makeTrackValidationPlots.py --extended step4_24834.704.root
253176

254-
```bash
255-
# These two lines are only needed to set the right version of gcc and nvcc. They are not needed for standard installations.
256-
export PATH=/cvmfs/cms.cern.ch/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin:/cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/external/el8_amd64_gcc12/bin:$PATH
257-
export LD_LIBRARY_PATH=/cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/biglib/el8_amd64_gcc12:/cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12:/cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/external/el8_amd64_gcc12/lib:/cvmfs/cms.cern.ch/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/lib64:/cvmfs/cms.cern.ch/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/lib:$LD_LIBRARY_PATH
258-
259-
# These are the lines that you need to manually change for a CVMFS-less setup.
260-
# In this example we use cvmfs paths since that is where the dependencies are in lnx7188/cgpu1, but they can point to local directories.
261-
export BOOST_ROOT=/cvmfs/cms.cern.ch/el8_amd64_gcc12/external/boost/1.80.0-60a217837b5db1cff00c7d88ec42f53a
262-
export ALPAKA_ROOT=/cvmfs/cms.cern.ch/el8_amd64_gcc12/external/alpaka/1.1.0-7d0324257db47fde2d27987e7ff98fb4
263-
export CUDA_HOME=/cvmfs/cms.cern.ch/el8_amd64_gcc12/external/cuda/12.4.1-06cde0cd9f95a73a1ea05c8535f60bde
264-
export ROOT_ROOT=/cvmfs/cms.cern.ch/el8_amd64_gcc12/lcg/root/6.30.07-21947a33e64ceb827a089697ad72e468
265-
export CMSSW_BASE=/cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3
266-
267-
# These lines are needed to account for some extra environment variables that are exported in the setup script.
268-
export LD_LIBRARY_PATH=$PWD/SDL/cuda:$PWD/SDL/cpu:$PWD:$LD_LIBRARY_PATH
269-
export PATH=$PWD/bin:$PATH
270-
export PATH=$PWD/efficiency/bin:$PATH
271-
export PATH=$PWD/efficiency/python:$PATH
272-
export TRACKLOOPERDIR=$PWD
273-
export TRACKINGNTUPLEDIR=/data2/segmentlinking/CMSSW_12_2_0_pre2/
274-
export LSTOUTPUTDIR=.
275-
source $PWD/code/rooutil/thisrooutil.sh
276-
277-
# After this, you can compile and run LST as usual.
278-
sdl_run -f -mc -s PU200 -n -1 -t myTag
279-
```
177+
Comparison plots can be made by including multiple ROOT files as arguments.
280178

281179
## Code formatting and checking
282180

283-
The makefile in the `SDL` directory includes phony targets to run `clang-format` and `clang-tidy` on the code using the formatting and checks used in CMSSW. The following are the available commands.
181+
Using the first setup option above, it is prefered to run the checks provided by CMSSW using the following commands.
284182

285-
- `make format`
286-
Formats the code in the `SDL` directory using `clang-format` following the rules specified in `.clang-format`.
287-
- `make check`
288-
Runs `clang-tidy` on the code in the `SDL` directory to performs the checks specified in `.clang-tidy`.
289-
- `make check-fix`
290-
Same as `make check`, but fixes the issues that it knows how to fix.
291-
183+
```
184+
scram b -j 12 code-checks >& c.log && scram b -j 12 code-format >& f.log
185+
```

0 commit comments

Comments
 (0)