Skip to content

Commit e4121d2

Browse files
Sichao25Sichao25
authored
Add maintainer guide and pcms build instructions (#204)
* create maintainer guide and update build instruction * update spack based build instructions to spack v1.0 * add pointer to the maintainence scripts
1 parent 95e24d4 commit e4121d2

File tree

6 files changed

+570
-32
lines changed

6 files changed

+570
-32
lines changed

‎MAINTAINER.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
## Creating a Release
2+
3+
0. select the new version number following https://semver.org/:
4+
5+
> Given a version number MAJOR.MINOR.PATCH, increment the:
6+
>
7+
> - MAJOR version when you make incompatible API changes
8+
> - MINOR version when you add functionality in a backward compatible manner
9+
> - PATCH version when you make backward compatible bug fixes
10+
11+
12+
1. create a github issue documenting significant release changes; review the commit log and closed issues to find them
13+
14+
```
15+
This issue is to document functionality and features added to pcms since the #.#.# release (SHA1 of prior release):
16+
17+
New functionality or feature support:
18+
19+
- <feature> (SHA1,issueNumber)
20+
- ...
21+
22+
Bug Fixes:
23+
24+
- <feature> (SHA1,issueNumber)
25+
- ...
26+
27+
Other Updates and Improvements:
28+
29+
- <feature> (SHA1,issueNumber)
30+
- ...
31+
```
32+
33+
2. apply the issue/PR label 'v#.#.#' to significant issues and PR that are part of the release
34+
3. increase the pcms version # in [CMakeLists.txt](https://github.com/Sichao25/pcms/blob/yus/format_check_test/CMakeLists.txt#L4) in the `develop` branch
35+
4. commit; include the issue # in the commit message
36+
37+
```
38+
pcms version #.#.#
39+
see issue #<###>
40+
```
41+
42+
5. push
43+
6. create the tag `git tag -a v#.#.# -m "pcms version #.#.#"`
44+
7. push the tag `git push origin v#.#.#`
45+
46+
47+
## Maintain CI/CD
48+
49+
Current CI/CD contains:
50+
- Cmake build and test using GitHub Actions triggered on pushes and pull requests to the `develop`,`main` branch. The workflow file is [`.github/workflows/cmake-test.yml`](https://github.com/SCOREC/pcms/blob/develop/.github/workflows/cmake-test.yml).
51+
- Format check using clang-format and cmake-format. Workflow file is [`.github/workflows/format-check.yml`](https://github.com/SCOREC/pcms/blob/develop/.github/workflows/format-check.yml).
52+
- Static code analysis using clang-tidy. Combined in the main cmake workflow at [`.github/workflows/cmake-test.yml`](https://github.com/SCOREC/pcms/blob/develop/.github/workflows/cmake-test.yml).
53+
- Weekly globus build/test in supercomputer Perlmutter. Workflow file is in [`.github/workflows/perlmutter/`](https://github.com/SCOREC/pcms/tree/develop/.github/workflows/perlmutter) directory.
54+
- Nightly build in [CDash dashboard](https://my.cdash.org/index.php?project=SCOREC). The build script is in [`cdash/`](https://github.com/SCOREC/pcms/tree/develop/cdash) directory. The script is triggered by a cron job on a SCOREC workstation daily at early morning.
55+
56+
To maintain the CI/CD pipeline, ensure that any changes to the build or test processes are reflected in those workflow file. Regularly review the workflow for updates to dependencies or tools used in the pipeline. When adding new steps to the CI/CD process, ensure they are properly integrated and tested to avoid disruptions. Try to avoid third party actions for security and maintenance reasons.

‎README.md‎

Lines changed: 168 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@ Adios2-based xgc_coupler for XGC and GENE
66

77
- CMake 3.19+
88
- MPI
9-
- FFTW 3.3.8+
9+
- Kokkos
10+
- KokkosKernels
1011
- redev 3.0.0+ (https://github.com/SCOREC/redev)
1112
- Omega\_h 10.2.0+ with MPI enabled (https://github.com/SCOREC/omega_h)
1213
- Catch2 2.\* (for unit tests) (https://github.com/catchorg/Catch2/tree/v2.13.8)
@@ -17,40 +18,181 @@ Adios2-based xgc_coupler for XGC and GENE
1718
SCOREC Rhel7 environment
1819

1920
```
20-
module unuse /opt/scorec/spack/lmod/linux-rhel7-x86_64/Core
21-
module use /opt/scorec/spack/v0154_2/lmod/linux-rhel7-x86_64/Core
22-
module load \
23-
gcc/10.1.0 \
24-
mpich \
25-
cmake/3.20.0 \
26-
fftw \
27-
gdb
21+
module use /opt/scorec/spack/rhel9/v0201_4/lmod/linux-rhel9-x86_64/Core/
22+
module load gcc/12.3.0-iil3lno
23+
module load mpich/4.1.1-xpoyz4t
24+
module load cmake/3.26.3-2duxfcd
25+
// no need to load CUDA if not using GPUs
26+
module load cuda/12.1.1-zxa4msk
2827
```
2928

30-
Build, install, and test
29+
### Build dependencies with CPU
3130

3231
```
33-
git clone git@github.com:SCOREC/wdmapp_testcases.git #test data
34-
git clone git@github.com:SCOREC/wdmapp_coupling.git
35-
36-
cmake -S wdmapp_coupling -B buildWdmCpl \
37-
-Dredev_ROOT=/path/to/redev/install \
38-
-DOmega_h_ROOT=/path/to/omegah/install \
39-
-DCMAKE_INSTALL_PREFIX=$PWD/buildWdmCpl/install \
40-
-DPCMS_TEST_DATA_DIR=$PWD/wdmapp_testcases \
41-
-DCatch2_ROOT=/path/to/catch2/install
42-
43-
cmake --build buildWdmCpl --target install
44-
45-
ctest --test-dir buildWdmCpl --output-on-failure
32+
git clone --branch 4.6.01 --depth 1 git@github.com:kokkos/kokkos.git
33+
cmake -S kokkos -B build-kokkos \
34+
-DCMAKE_INSTALL_PREFIX=build-kokkos/install \
35+
-DCMAKE_CXX_STANDARD=17 \
36+
-DKokkos_ENABLE_SERIAL=ON \
37+
-DKokkos_ENABLE_OPENMP=OFF \
38+
-DKokkos_ENABLE_CUDA=OFF \
39+
-DKokkos_ENABLE_CUDA_LAMBDA=OFF \
40+
-DKokkos_ENABLE_CUDA_CONSTEXPR=OFF \
41+
-DBUILD_SHARED_LIBS=OFF
42+
cmake --build build-kokkos --target install
43+
44+
git clone --branch 4.6.01 --depth 1 git@github.com:kokkos/kokkos-kernels.git
45+
cmake -S kokkos-kernels -B build-kokkos-kernels \
46+
-DCMAKE_INSTALL_PREFIX=build-kokkos-kernels/install \
47+
-DCMAKE_CXX_STANDARD=17 \
48+
-DKokkos_ROOT=$PWD/build-kokkos/install/lib64/cmake \
49+
-DBUILD_SHARED_LIBS=OFF
50+
cmake --build build-kokkos-kernels --target install
51+
52+
git clone git@github.com:SCOREC/omega_h.git
53+
cmake -S omega_h -B build-omega_h \
54+
-DCMAKE_INSTALL_PREFIX=$PWD/build-omega_h/install \
55+
-DCMAKE_C_COMPILER=mpicc \
56+
-DCMAKE_CXX_COMPILER=mpicxx \
57+
-DBUILD_SHARED_LIBS=OFF \
58+
-DOmega_h_USE_Kokkos=ON \
59+
-DOmega_h_USE_CUDA=OFF \
60+
-DOmega_h_USE_MPI=ON \
61+
-DMPIEXEC_EXECUTABLE=mpirun \
62+
-DBUILD_TESTING=OFF \
63+
-DKokkos_PREFIX=$PWD/build-kokkos/install/lib64/cmake
64+
cmake --build build-omega_h --target install
65+
66+
git clone git@github.com:ornladios/ADIOS2.git
67+
cmake -S ADIOS2 -B build-ADIOS2 \
68+
-DCMAKE_INSTALL_PREFIX=build-ADIOS2/install \
69+
-DADIOS2_USE_CUDA=OFF \
70+
-DADIOS2_USE_ZFP=off
71+
cmake --build build-ADIOS2 --target install
72+
73+
git clone git@github.com:UO-OACISS/perfstubs.git
74+
cmake -S perfstubs -B build-perfstubs \
75+
-DCMAKE_CXX_COMPILER=mpicxx \
76+
-DCMAKE_INSTALL_PREFIX=build-perfstubs/install
77+
cmake --build build-perfstubs --target install
78+
79+
git clone git@github.com:catchorg/Catch2
80+
cmake -S Catch2 -B build-Catch2 \
81+
-DCMAKE_INSTALL_PREFIX=$PWD/build-Catch2/install
82+
cmake --build build-Catch2 --target install
83+
84+
git clone git@github.com:SCOREC/redev.git
85+
cmake -S redev -B build-redev \
86+
-DCMAKE_CXX_COMPILER=mpicxx \
87+
-DMPIEXEC_EXECUTABLE=mpirun \
88+
-DCMAKE_BUILD_TYPE=Release \
89+
-DBUILD_SHARED_LIBS=OFF \
90+
-DCMAKE_INSTALL_PREFIX=build-redev/install \
91+
-DADIOS2_DIR=$PWD/build-ADIOS2/install/lib64/cmake/adios2 \
92+
-Dperfstubs_DIR=$PWD/build-perfstubs/install/lib/cmake
93+
cmake --build build-redev --target install
94+
```
95+
### Build dependencies with CUDA
96+
```
97+
git clone --branch 4.6.01 --depth 1 git@github.com:kokkos/kokkos.git
98+
cmake -S kokkos -B build-kokkos \
99+
-DCMAKE_INSTALL_PREFIX=build-kokkos/install \
100+
-DCMAKE_CXX_STANDARD=17 \
101+
-DCMAKE_BUILD_TYPE="Release" \
102+
-DCMAKE_CXX_COMPILER=$PWD/kokkos/bin/nvcc_wrapper \
103+
-DKokkos_ARCH_AMPERE80=ON \
104+
-DKokkos_ENABLE_SERIAL=ON \
105+
-DKokkos_ENABLE_OPENMP=off \
106+
-DKokkos_ENABLE_CUDA=on \
107+
-DKokkos_ENABLE_CUDA_LAMBDA=on \
108+
-DKokkos_ENABLE_CUDA_CONSTEXPR=on \
109+
-DKokkos_ENABLE_DEBUG=off
110+
cmake --build build-kokkos --target install
111+
112+
git clone --branch 4.6.01 --depth 1 git@github.com:kokkos/kokkos-kernels.git
113+
cmake -S kokkos-kernels -B build-kokkos-kernels \
114+
-DCMAKE_INSTALL_PREFIX=build-kokkos-kernels/install \
115+
-DCMAKE_CXX_STANDARD=17 \
116+
-DKokkos_ROOT=$PWD/build-kokkos/install/lib64/cmake \
117+
-DBUILD_SHARED_LIBS=off
118+
cmake --build build-kokkos-kernels --target install
119+
120+
git clone git@github.com:SCOREC/omega_h.git
121+
cmake -S omega_h -B build-omega_h \
122+
-DCMAKE_INSTALL_PREFIX=build-omega_h/install \
123+
-DCMAKE_BUILD_TYPE=Release \
124+
-DBUILD_SHARED_LIBS=off \
125+
-DOmega_h_USE_Kokkos=ON \
126+
-DOmega_h_USE_CUDA=on \
127+
-DOmega_h_CUDA_ARCH=80 \
128+
-DOmega_h_USE_MPI=on \
129+
-DMPIEXEC_EXECUTABLE=mpirun \
130+
-DBUILD_TESTING=off \
131+
-DCMAKE_C_COMPILER=mpicc \
132+
-DCMAKE_CXX_COMPILER=mpicxx \
133+
-DKokkos_PREFIX=$PWD/build-kokkos/install/lib64/cmake
134+
cmake --build build-omega_h --target install
135+
136+
git clone git@github.com:ornladios/ADIOS2.git
137+
cmake -S ADIOS2 -B build-ADIOS2 \
138+
-DCMAKE_INSTALL_PREFIX=build-ADIOS2/install \
139+
-DADIOS2_USE_CUDA=on \
140+
-DCMAKE_BUILD_TYPE=Release \
141+
-DCMAKE_CXX_COMPILER=mpicxx \
142+
-DCMAKE_C_COMPILER=mpicc \
143+
-DADIOS2_USE_ZFP=off
144+
cmake --build build-ADIOS2 --target install
145+
146+
git clone git@github.com:UO-OACISS/perfstubs.git
147+
cmake -S perfstubs -B build-perfstubs \
148+
-DCMAKE_CXX_COMPILER=mpicxx \
149+
-DCMAKE_INSTALL_PREFIX=build-perfstubs/install
150+
cmake --build build-perfstubs --target install
151+
152+
git clone git@github.com:SCOREC/redev.git
153+
cmake -S redev -B build-redev \
154+
-DCMAKE_CXX_COMPILER=mpicxx \
155+
-DMPIEXEC_EXECUTABLE=mpirun \
156+
-DCMAKE_BUILD_TYPE=Release \
157+
-DBUILD_SHARED_LIBS=OFF \
158+
-DCMAKE_INSTALL_PREFIX=build-redev/install \
159+
-DADIOS2_DIR=$PWD/build-ADIOS2/install/lib64/cmake/adios2 \
160+
-Dperfstubs_DIR=$PWD/build-perfstubs/install/lib/cmake
161+
cmake --build build-redev --target install
162+
163+
git clone git@github.com:catchorg/Catch2
164+
cmake -S Catch2 -B build-Catch2 \
165+
-DCMAKE_INSTALL_PREFIX=$PWD/build-Catch2/install
166+
cmake --build build-Catch2 --target install
167+
```
168+
### Build, install, and test pcms
169+
```
170+
git clone git@github.com:jacobmerson/pcms_testcases.git // test data
171+
git clone git@github.com:SCOREC/pcms.git
172+
173+
cmake -S pcms -B build-pcms \
174+
-DCMAKE_C_COMPILER=mpicc \
175+
-DCMAKE_CXX_COMPILER=mpicxx \
176+
-DCMAKE_Fortran_COMPILER=mpifort \
177+
-DCMAKE_BUILD_TYPE="Debug" \
178+
-DPCMS_TIMEOUT=100 \
179+
-Dredev_DIR=$PWD/build-redev/install/lib64/cmake/redev \
180+
-DOmega_h_DIR=$PWD/build-omega_h/install/lib64/cmake/Omega_h/ \
181+
-Dperfstubs_DIR=$PWD/build-perfstubs/install/lib/cmake \
182+
-DCatch2_DIR=$PWD/build-Catch2/install/lib64/cmake/Catch2/ \
183+
-DKokkosKernels_DIR=$PWD/build-kokkos-kernels/install/lib64/cmake/KokkosKernels/ \
184+
-DPCMS_TEST_DATA_DIR=$PWD/pcms_testcases
185+
cmake --build build-pcms -j 8
186+
187+
ctest --test-dir build-pcms --output-on-failure
46188
```
47189

48190
## Spack based build
49191
1. Install spack
50192
```console
51193
$ mkdir /lore/$USER/spack
52194
$ cd /lore/$USER/spack
53-
$ git clone -c feature.manyFiles=true -b releases/v0.20 https://github.com/spack/spack.git
195+
$ git clone -c feature.manyFiles=true -b releases/v1.0 https://github.com/spack/spack.git
54196
$ . spack/share/spack/setup-env.sh
55197
```
56198
We can also add the spack setup line into the `~/.bashrc` with `echo ". spack/share/spack/setup-env.sh" >> ~/.bashrc". This will load the spack setup script every time we start our terminal session.
@@ -59,16 +201,10 @@ ctest --test-dir buildWdmCpl --output-on-failure
59201
The following commands will add the pcms recipe files to spack. They are not currently installed inthe upstream spack repository.
60202
```console
61203
$ git clone https://github.com/jacobmerson/pcms-spack.git
62-
$ spack repo add pcms-spack/pcms
204+
$ spack repo add pcms-spack/spack_repo/pcms
63205
```
64206

65-
3. Add Spack binary mirror
66-
Addding the binary mirrors will avoid some compilation by downloading prebuilt binaries when available.
67-
```console
68-
$ spack mirror add v0.20.1 https://binaries.spack.io/v0.20.1
69-
$ spack buildcache keys --install --trust
70-
```
71-
5. Install PCMS repo
207+
3. Install PCMS repo
72208
```console
73209
$ mkdir /lore/$USER/pcms-coupler
74210
$ cd /lore/$USER/pcms-coupler

‎cdash/CTestConfig.cmake‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
## This file should be placed in the root directory of your project.
2+
## Then modify the CMakeLists.txt file in the root directory of your
3+
## project to incorporate the testing dashboard.
4+
##
5+
## # The following are required to submit to the CDash dashboard:
6+
## ENABLE_TESTING()
7+
## INCLUDE(CTest)
8+
9+
set(CTEST_PROJECT_NAME SCOREC)
10+
set(CTEST_NIGHTLY_START_TIME 3:00:00 EDT)
11+
12+
if(CMAKE_VERSION VERSION_GREATER 3.14)
13+
set(CTEST_SUBMIT_URL http://my.cdash.org/submit.php?project=SCOREC)
14+
else()
15+
set(CTEST_DROP_METHOD "http")
16+
set(CTEST_DROP_SITE "my.cdash.org")
17+
set(CTEST_DROP_LOCATION "/submit.php?project=SCOREC")
18+
endif()
19+
20+
set(CTEST_DROP_SITE_CDASH TRUE)

‎cdash/Project.xml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
<Project name="SCOREC">
2+
<SubProject name="master"> </SubProject>
3+
<SubProject name="develop"> </SubProject>
4+
<SubProject name="develop-into-master"> </SubProject>
5+
<SubProject name="omega_h"> </SubProject>
6+
<SubProject name="pcms"> </SubProject>
7+
<SubProject name="pumi-pic"> </SubProject>
8+
<SubProject name="meshfields"> </SubProject>
9+
<SubProject name="EnGPar"> </SubProject>
10+
</Project>

0 commit comments

Comments
 (0)