Skip to content

Commit 08e58eb

Browse files
sbryngelsonsbryngelson
authored
Merge branch 'master' into Diffusion
2 parents 6e619b2 + 6f58eec commit 08e58eb

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

53 files changed

+1801
-1936
lines changed

‎.github/copilot-instructions.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# GitHub Copilot Instructions – Pull-Request Reviews for MFC
2+
3+
These instructions guide **GitHub Copilot Code Review** and **Copilot Chat** when they evaluate pull requests in this repository.
4+
5+
---
6+
7+
## 1 Project Context (always include)
8+
9+
* **Project:** MFC (exascale many-physics solver) written in **modern Fortran 2008+**, generated with **Fypp**.
10+
* **Directory layout:**
11+
* Sources in `src/`, tests in `tests/`, examples in `examples/`.
12+
* Most source files are templated `.fpp`; CMake transpiles them to `.f90`.
13+
* **Fypp macros** are in `src/<subprogram>/include/`, where `<subprogram>` is `simulation`, `common`, `pre_process`, or `post_process`. Review these first.
14+
* Only `simulation` (plus its `common` dependencies) is GPU-accelerated with **OpenACC**.
15+
16+
> **Copilot, when reviewing:**
17+
> * Treat the codebase as free-form Fortran 2008+ with `implicit none`, explicit `intent`, and standard intrinsics.
18+
> * Prefer `module … contains … subroutine foo()` over legacy constructs; flag uses of `COMMON`, file-level `include`, or other global state.
19+
20+
---
21+
22+
## 2 Incremental-Change Workflow
23+
24+
Copilot, when reviewing:
25+
* Encourage small, buildable commits
26+
27+
---
28+
29+
## 3 Style & Naming Conventions (*.fpp / *.f90)
30+
31+
| Element | Rule |
32+
|---------|------|
33+
| Indentation | 2 spaces; continuation lines align beneath &. |
34+
| Case | Lower-case keywords & intrinsics (do, end subroutine, …). |
35+
| Modules | m_<feature> (e.g. m_transport). |
36+
| Public subroutines | s_<verb>_<noun> (s_compute_flux). |
37+
| Public functions | f_<verb>_<noun>. |
38+
| Routine size | subroutine ≤ 500 lines, helper ≤ 150, function ≤ 100, file ≤ 1000. |
39+
| Arguments | ≤ 6; else use a derived-type params struct. |
40+
| Forbidden | goto (except legacy), COMMON, save globals. |
41+
| Variables | Every arg has explicit intent; use dimension/allocatable/pointer as appropriate. |
42+
| Errors | Call s_mpi_abort(<msg>), never stop or error stop. |
43+
44+
Copilot, when reviewing:
45+
* Flag violations of any cell above.
46+
* Suggest refactors when size or argument limits are exceeded.
47+
* Ensure private helpers stay inside their defining module and avoid nested procedures.
48+
49+
---
50+
51+
## 4 OpenACC Guidelines (for GPU kernels)
52+
53+
Wrap tight loops:
54+
55+
```fortran
56+
!$acc parallel loop gang vector default(present) reduction(...)
57+
```
58+
59+
* Add collapse(n) when safe.
60+
* Declare loop-local variables with private(...).
61+
* Allocate large arrays with managed or move them into a persistent !$acc enter data region at start-up.
62+
* Avoid stop/error stop inside device code.
63+
* Code must compile with Cray ftn, NVIDIA nvfortran, GNU gfortran, and Intel ifx/ifort.
64+
65+
---
66+
67+
## 5 Review Checklist (what Copilot should verify)
68+
69+
1. Buildability: PR includes build instructions or CI passes the staged build.
70+
2. Tests: Focused tests are added/updated.
71+
3. Style compliance: All rules in §3 are satisfied.
72+
4. Module hygiene: No new global state; proper namespacing.
73+
5. Performance: GPU code follows §4; no large host/device transfers in hot loops.
74+
6. Documentation: Updated in-code comments and, when needed, README or docs site.
75+
7. Regressions: No changes to outputs of golden tests without clear justification.

‎.github/workflows/bench.yml‎

Lines changed: 44 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
name: 'Benchmark'
22

3-
on:
3+
on:
4+
pull_request:
45
pull_request_review:
56
types: [submitted]
67
workflow_dispatch:
@@ -22,16 +23,40 @@ jobs:
2223
filters: ".github/file-filter.yml"
2324

2425
self:
25-
name: Georgia Tech | Phoenix (NVHPC)
26-
if: github.repository == 'MFlowCode/MFC' && needs.file-changes.outputs.checkall == 'true' && ${{ github.event.review.state == 'approved' }}
26+
name: "${{ matrix.name }} (${{ matrix.device }})"
27+
if: ${{ github.repository == 'MFlowCode/MFC' && needs.file-changes.outputs.checkall == 'true' && (
28+
(github.event_name == 'pull_request_review' && github.event.review.state == 'approved') ||
29+
(github.event_name == 'pull_request' && github.event.pull_request.user.login == 'sbryngelson')
30+
) }}
2731
needs: file-changes
2832
strategy:
29-
matrix:
30-
device: ['cpu', 'gpu']
3133
fail-fast: false
34+
matrix:
35+
include:
36+
- cluster: phoenix
37+
name: Georgia Tech | Phoenix (NVHPC)
38+
group: phoenix
39+
labels: gt
40+
flag: p
41+
device: cpu
42+
build_script: ""
43+
- cluster: phoenix
44+
name: Georgia Tech | Phoenix (NVHPC)
45+
group: phoenix
46+
labels: gt
47+
flag: p
48+
device: gpu
49+
build_script: ""
50+
- cluster: frontier
51+
name: Oak Ridge | Frontier (CCE)
52+
group: phoenix
53+
labels: frontier
54+
flag: f
55+
device: gpu
56+
build_script: "bash .github/workflows/frontier/build.sh gpu bench"
3257
runs-on:
33-
group: phoenix
34-
labels: gt
58+
group: ${{ matrix.group }}
59+
labels: ${{ matrix.labels }}
3560
timeout-minutes: 1400
3661
env:
3762
ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION: node16
@@ -49,15 +74,22 @@ jobs:
4974
ref: master
5075
path: master
5176

77+
- name: Setup & Build
78+
if: matrix.build_script != ''
79+
run: |
80+
(cd pr && ${{ matrix.build_script }}) &
81+
(cd master && ${{ matrix.build_script }}) &
82+
wait %1 && wait %2
83+
5284
- name: Bench (Master v. PR)
5385
run: |
54-
(cd pr && bash .github/workflows/phoenix/submit-bench.sh .github/workflows/phoenix/bench.sh ${{ matrix.device }}) &
55-
(cd master && bash .github/workflows/phoenix/submit-bench.sh .github/workflows/phoenix/bench.sh ${{ matrix.device }}) &
86+
(cd pr && bash .github/workflows/${{ matrix.cluster }}/submit-bench.sh .github/workflows/${{ matrix.cluster }}/bench.sh ${{ matrix.device }}) &
87+
(cd master && bash .github/workflows/${{ matrix.cluster }}/submit-bench.sh .github/workflows/${{ matrix.cluster }}/bench.sh ${{ matrix.device }}) &
5688
wait %1 && wait %2
5789
5890
- name: Generate & Post Comment
5991
run: |
60-
(cd pr && . ./mfc.sh load -c p -m g)
92+
(cd pr && . ./mfc.sh load -c ${{ matrix.flag }} -m g)
6193
(cd pr && ./mfc.sh bench_diff ../master/bench-${{ matrix.device }}.yaml ../pr/bench-${{ matrix.device }}.yaml)
6294
6395
- name: Print Logs
@@ -68,9 +100,9 @@ jobs:
68100
69101
- name: Archive Logs
70102
uses: actions/upload-artifact@v4
71-
if: always()
103+
if: always()
72104
with:
73-
name: logs-${{ matrix.device }}
105+
name: ${{ matrix.cluster }}-${{ matrix.device }}
74106
path: |
75107
pr/bench-${{ matrix.device }}.*
76108
pr/build/benchmarks/*

‎.github/workflows/frontier/bench.sh‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/bin/bash
2+
3+
n_ranks=12
4+
5+
if [ "$job_device" = "gpu" ]; then
6+
gpus=$(rocm-smi --showid | awk '{print $1}' | grep -Eo '[0-9]+' | uniq | tr '\n' ' ')
7+
n_ranks=$(echo "$gpus" | wc -w) # number of GPUs on node
8+
gpu_ids=$(echo "$gpus" | tr ' ' '\n' | tr '\n' ' ' | sed 's/ $//') # GPU IDs from rocm-smi
9+
device_opts="--gpu -g $gpu_ids"
10+
fi
11+
12+
if [ "$job_device" = "gpu" ]; then
13+
./mfc.sh bench --mem 12 -j $n_ranks -o "$job_slug.yaml" -- -c frontier $device_opts -n $n_ranks
14+
else
15+
./mfc.sh bench --mem 1 -j $(nproc) -o "$job_slug.yaml" -- -c frontier $device_opts -n $n_ranks
16+
fi

‎.github/workflows/frontier/build.sh‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,17 @@
11
#!/bin/bash
22

33
build_opts=""
4-
if [ "$1" == "gpu" ]; then
4+
if [ "$1" = "gpu" ]; then
55
build_opts="--gpu"
66
fi
77

88
. ./mfc.sh load -c f -m g
9-
./mfc.sh test --dry-run -j 8 $build_opts
9+
10+
if [ "$2" == "bench" ]; then
11+
for dir in benchmarks/*/; do
12+
dirname=$(basename "$dir")
13+
./mfc.sh run "$dir/case.py" --case-optimization -j 8 --dry-run $build_opts
14+
done
15+
else
16+
./mfc.sh test --dry-run -j 8 $build_opts
17+
fi

‎.github/workflows/frontier/submit-bench.sh‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
#!/bin/bash
2+
3+
set -e
4+
5+
usage() {
6+
echo "Usage: $0 [script.sh] [cpu|gpu]"
7+
}
8+
9+
if [ ! -z "$1" ]; then
10+
sbatch_script_contents=`cat $1`
11+
else
12+
usage
13+
exit 1
14+
fi
15+
16+
if [ "$2" = "cpu" ]; then
17+
sbatch_device_opts="\
18+
#SBATCH -n 32 # Number of cores required"
19+
elif [ "$2" = "gpu" ]; then
20+
sbatch_device_opts="\
21+
#SBATCH -n 8 # Number of cores required"
22+
else
23+
usage; exit 1
24+
fi
25+
26+
27+
job_slug="`basename "$1" | sed 's/\.sh$//' | sed 's/[^a-zA-Z0-9]/-/g'`-$2"
28+
29+
sbatch <<EOT
30+
#!/bin/bash
31+
#SBATCH -JMFC-$job_slug # Job name
32+
#SBATCH -A CFD154 # charge account
33+
#SBATCH -N 1 # Number of nodes required
34+
$sbatch_device_opts
35+
#SBATCH -t 03:59:00 # Duration of the job (Ex: 15 mins)
36+
#SBATCH -o$job_slug.out # Combined output and error messages file
37+
#SBATCH -p extended # Extended partition for shorter queues
38+
#SBATCH -W # Do not exit until the submitted job terminates.
39+
40+
set -e
41+
set -x
42+
43+
cd "\$SLURM_SUBMIT_DIR"
44+
echo "Running in $(pwd):"
45+
46+
job_slug="$job_slug"
47+
job_device="$2"
48+
49+
. ./mfc.sh load -c f -m g
50+
51+
$sbatch_script_contents
52+
53+
EOT
54+

‎.github/workflows/frontier/submit.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,10 @@ else
1313
exit 1
1414
fi
1515

16-
if [ "$2" == "cpu" ]; then
16+
if [ "$2" = "cpu" ]; then
1717
sbatch_device_opts="\
1818
#SBATCH -n 32 # Number of cores required"
19-
elif [ "$2" == "gpu" ]; then
19+
elif [ "$2" = "gpu" ]; then
2020
sbatch_device_opts="\
2121
#SBATCH -n 8 # Number of cores required"
2222
else

‎.github/workflows/frontier/test.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
gpus=`rocm-smi --showid | awk '{print $1}' | grep -Eo '[0-9]+' | uniq | tr '\n' ' '`
44
ngpus=`echo "$gpus" | tr -d '[:space:]' | wc -c`
55

6-
if [ "$job_device" == "gpu" ]; then
6+
if [ "$job_device" = "gpu" ]; then
77
./mfc.sh test --max-attempts 3 -j $ngpus -- -c frontier
88
else
99
./mfc.sh test --max-attempts 3 -j 32 -- -c frontier

‎.github/workflows/phoenix/bench.sh‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,19 +2,26 @@
22

33
n_ranks=12
44

5-
if [ "$job_device" == "gpu" ]; then
5+
if [ "$job_device" = "gpu" ]; then
66
n_ranks=$(nvidia-smi -L | wc -l) # number of GPUs on node
77
gpu_ids=$(seq -s ' ' 0 $(($n_ranks-1))) # 0,1,2,...,gpu_count-1
88
device_opts="--gpu -g $gpu_ids"
99
fi
1010

11-
mkdir -p /storage/scratch1/6/sbryngelson3/mytmp_build
12-
export TMPDIR=/storage/scratch1/6/sbryngelson3/mytmp_build
11+
tmpbuild=/storage/scratch1/6/sbryngelson3/mytmp_build
12+
currentdir=$tmpbuild/run-$(( RANDOM % 900 ))
13+
mkdir -p $tmpbuild
14+
mkdir -p $currentdir
1315

14-
if ["$job_device" == "gpu"]; then
16+
export TMPDIR=$currentdir
17+
18+
if [ "$job_device" = "gpu" ]; then
1519
./mfc.sh bench --mem 12 -j $(nproc) -o "$job_slug.yaml" -- -c phoenix-bench $device_opts -n $n_ranks
1620
else
1721
./mfc.sh bench --mem 1 -j $(nproc) -o "$job_slug.yaml" -- -c phoenix-bench $device_opts -n $n_ranks
1822
fi
1923

24+
sleep 10
25+
rm -rf "$currentdir" || true
26+
2027
unset TMPDIR

‎.github/workflows/phoenix/submit-bench.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,9 +25,9 @@ sbatch_gpu_opts="\
2525
#SBATCH -G2\
2626
"
2727

28-
if [ "$2" == "cpu" ]; then
28+
if [ "$2" = "cpu" ]; then
2929
sbatch_device_opts="$sbatch_cpu_opts"
30-
elif [ "$2" == "gpu" ]; then
30+
elif [ "$2" = "gpu" ]; then
3131
sbatch_device_opts="$sbatch_gpu_opts"
3232
else
3333
usage

‎.github/workflows/phoenix/submit.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,9 +25,9 @@ sbatch_gpu_opts="\
2525
#SBATCH -G2\
2626
"
2727

28-
if [ "$2" == "cpu" ]; then
28+
if [ "$2" = "cpu" ]; then
2929
sbatch_device_opts="$sbatch_cpu_opts"
30-
elif [ "$2" == "gpu" ]; then
30+
elif [ "$2" = "gpu" ]; then
3131
sbatch_device_opts="$sbatch_gpu_opts"
3232
else
3333
usage

0 commit comments

Comments
 (0)