merge conflict resolution

Author: Benjamin Wilfong

.cursor/rules/mfc-agent-rules.mdc

@@ -4,29 +4,27 @@ alwaysApply: true
 ---
 
 # 0  Purpose & Scope
-Consolidated guidance for the MFC exascale, many-physics solver.  
-Written primarily for Fortran/Fypp; the OpenACC and style sections matter only when
-`.fpp` / `.f90` files are in view.
+Consolidated guidance for the MFC exascale, many-physics solver.
+Written primarily for Fortran/Fypp; the GPU and style sections matter only when `.fpp` / `.f90` files are in view.
 
 ---
 
 # 1  Global Project Context (always)
-- **Project**: *MFC* is modern Fortran 2008+ generated with **Fypp**.  
-  - Sources `src/`, tests `tests/`, examples `examples/`.  
-  - Most sources are `.fpp`; CMake transpiles them to `.f90`.  
-- **Fypp macros** live in `src/<subprogram>/include/` you should scan these first.  
-  `<subprogram>` ∈ {`simulation`,`common`,`pre_process`,`post_process`}.  
-- Only `simulation` (+ its `common` calls) is GPU-accelerated via **OpenACC**.  
-- Assume free-form Fortran 2008+, `implicit none`, explicit `intent`, and modern
-  intrinsics.  
-- Prefer `module … contains … subroutine foo()`; avoid `COMMON` blocks and
-  file-level `include` files.  
-- **Read the full codebase and docs *before* changing code.**  
-  Docs: <https://mflowcode.github.io/documentation/md_readme.html> and the respository root `README.md`.  
+- **Project**: *MFC* is modern Fortran 2008+ generated with **Fypp**.
+  - Sources `src/`, tests `tests/`, examples `examples/`.
+  - Most sources are `.fpp`; CMake transpiles them to `.f90`.
+- **Fypp macros** live in `src/<subprogram>/include/` you should scan these first.
+  `<subprogram>` ∈ {`simulation`,`common`,`pre_process`,`post_process`}.
+- Only `simulation` (+ its `common` calls) is GPU-accelerated via **OpenACC** or **OpenMP**.
+- Assume free-form Fortran 2008+, `implicit none`, explicit `intent`, and modern intrinsics.
+- Prefer `module … contains … subroutine foo()`; avoid `COMMON` blocks and file-level `include` files.
+- **Read the full codebase and docs *before* changing code.**
+  - Docs: <https://mflowcode.github.io/documentation/md_readme.html> and the repository root `README.md`.
 
 ### Incremental-change workflow
-1. Draft a step-by-step plan.  
-2. After each step, build:  
+
+1. Draft a step-by-step plan.
+2. After each step, build:
    ```bash
    ./mfc.sh build -t pre_process simulation -j $(nproc)
     ```
@@ -49,34 +47,131 @@ Written primarily for Fortran/Fypp; the OpenACC and style sections matter only w
   * Subroutine → `s_<verb>_<noun>` (e.g. `s_compute_flux`)
   * Function   → `f_<verb>_<noun>`
 * Private helpers stay in the module; avoid nested procedures.
-* **Size limits**: subroutine ≤ 500 lines, helper ≤ 150, function ≤ 100,
-  module/file ≤ 1000.
-* ≤ 6 arguments per routine; otherwise pass a derived-type “params” struct.
+* **Size limits**: subroutine ≤ 500 lines, helper ≤ 150, function ≤ 100, module/file ≤ 1000.
+* ≤ 6 arguments per routine; otherwise pass a derived-type "params" struct.
 * No `goto` (except unavoidable legacy); no global state (`COMMON`, `save`).
-* Every variable: `intent(in|out|inout)` + appropriate `dimension` / `allocatable`
-  / `pointer`.
+* Every variable: `intent(in|out|inout)` + appropriate `dimension` / `allocatable` / `pointer`.
 * Use `s_mpi_abort(<msg>)` for errors, not `stop`.
-* Mark OpenACC-callable helpers that are called from OpenACC parallel loops immediately after declaration:
+* Mark GPU-callable helpers that are called from GPU parallel loops immediately after declaration:
   ```fortran
   subroutine s_flux_update(...)
-    !$acc routine seq
+    $:GPU_ROUTINE(function_name='s_flux_update', parallelism='[seq]')
     ...
   end subroutine
   ```
 
 ---
 
-# 3  OpenACC Programming Guidelines (for kernels)
+# 3  File & Module Structure
 
-Wrap tight loops with
+- **File Naming**:
+  - `.fpp` files: Fypp preprocessed files that get translated to `.f90`
+  - Modules are named with `m_` prefix followed by feature name: `m_helper_basic`, `m_viscous`
+  - Primary program file is named `p_main.fpp`
+
+- **Module Layout**:
+  - Start with Fypp include for macros: `#:include 'macros.fpp'`
+  - Header comments using `!>` style documentation
+  - `module` declaration with name matching filename
+  - `use` statements for dependencies
+  - `implicit none` statement
+  - `private` declaration followed by explicit `public` exports
+  - `contains` section
+  - Implementation of subroutines and functions
+
+---
+
+# 4  Fypp Macros
+
+- **Fypp Directives**:
+  - Start with `#:` (e.g., `#:include`, `#:def`, `#:enddef`)
+  - Macros defined in `include/*.fpp` files
+  - Used for code generation, conditional compilation, and GPU offloading
+
+---
 
+# 5  FYPP Macros for GPU Acceleration Programming Guidelines (for GPU kernels)
+
+- Do not use OpenACC or OpenMP directives directly.
+- Instead, use the FYPP macros contained in `src/common/include/parallel_macros.fpp`
+- Documentation on how to use the Fypp macros for GPU offloading is available at https://mflowcode.github.io/documentation/md_gpuParallelization.html
+
+Wrap tight loops with
 ```fortran
-!$acc parallel loop gang vector default(present) reduction(...)
+$:GPU_PARALLEL_FOR(private='[...]', copy='[...]')
 ```
-* Add `collapse(n)` to merge nested loops when safe.
-* Declare loop-local variables with `private(...)`.
+* Add `collapse=n` to merge nested loops when safe.
+* Declare loop-local variables with `private='[...]'`.
 * Allocate large arrays with `managed` or move them into a persistent
-  `!$acc enter data` region at start-up.
+  `$:GPU_ENTER_DATA(...)` region at start-up.
 * **Do not** place `stop` / `error stop` inside device code.
-* Must compile with Cray `ftn` and NVIDIA `nvfortran` for GPU offloading; also build CPU-only with
+* Must compile with Cray `ftn` or NVIDIA `nvfortran` for GPU offloading; also build CPU-only with
   GNU `gfortran` and Intel `ifx`/`ifort`.
+
+- Example GPU macros include the below, among others:
+  - `$:GPU_ROUTINE(parallelism='[seq]')` - Marks GPU-callable routines
+  - `$:GPU_PARALLEL_LOOP(collapse=N)` - Parallelizes loops
+  - `$:GPU_LOOP(parallelism='[seq]')` - Marks sequential loops
+  - `$:GPU_UPDATE(device='[var1,var2]')` - Updates device data
+  - `$:GPU_ENTER_DATA(copyin='[var]')` - Copies data to device
+  - `$:GPU_EXIT_DATA(delete='[var]')` - Removes data from device
+
+---
+
+# 6  Documentation Style
+
+- **Subroutine/Function Documentation**:
+  ```fortran
+  !> This procedure <description>
+  !! @param param_name Description of the parameter
+  !! @return Description of the return value (for functions)
+  ```
+which conforms to the Doxygen Fortran format.
+
+# 7  Error Handling
+
+- **Assertions**:
+  - Use the fypp `ASSERT` macro for validating conditions
+  - Example: `@:ASSERT(predicate, message)`
+
+- **Error Reporting**:
+  - Use `s_mpi_abort(error_message)` for error termination, not `stop`
+  - No `stop` / `error stop` inside device code
+
+# 8  Memory Management
+
+- **Allocation/Deallocation**:
+  - Use fypp macro `@:ALLOCATE(var1, var2)` macro for device-aware allocation
+  - Use fypp macro `@:DEALLOCATE(var1, var2)` macro for device-aware deallocation
+
+# 9. Additional Observed Patterns
+
+- **Derived Types**:
+  - Extensive use of derived types for encapsulation
+  - Use pointers within derived types (e.g., `pointer, dimension(:,:,:) => null()`)
+  - Clear documentation of derived type components
+
+- **Pure & Elemental Functions**:
+  - Use `pure` and `elemental` attributes for side-effect-free functions
+  - Combine them for operations on arrays (`pure elemental function`)
+
+- **Precision Handling**:
+  - Use `wp` (working precision) parameter from `m_precision_select`
+  - Never hardcode precision with `real*8` or similar
+
+- **Loop Optimization**:
+  - Favor array operations over explicit loops when possible
+  - Use `collapse=N` directive to optimize nested loops
+
+# 10. Fortran Practices to Avoid
+
+- **Fixed Format**: Only free-form Fortran is used
+  - No column-position dependent code
+
+- **Older Intrinsics**: Avoid outdated Fortran features like:
+  - `equivalence` statements
+  - `data` statements (use initialization expressions)
+  - Character*N (use `character(len=N)` instead)
+
+- **Using same variable for multiple purposes**: Maintain single responsibility
+  - Each variable should have one clear purpose

.fortlsrc

@@ -0,0 +1,83 @@
+{
+    "source_dirs": [
+        "src/",
+        "src/common/",
+        "src/simulation/",
+        "src/pre_process/",
+        "src/post_process/"
+    ],
+    "excl_paths": [
+        "benchmarks/",
+        "examples/",
+        "tests/",
+        "misc/",
+        "src/pre_process/include/2dHardcodedIC.fpp",
+        "src/pre_process/include/3dHardcodedIC.fpp",
+        "src/pre_process/include/ExtrusionHardcodedIC.fpp",
+        "**/m_nvtx*",
+        "**/syscheck.fpp"
+    ],
+    "include_dirs": [
+        "src/common/include/",
+        "src/simulation/include/",
+        "src/pre_process/include/",
+        "src/post_process/include/"
+    ],
+    "pp_suffixes": [".fpp"],
+    "pp_defs": {
+        "MFC": 1,
+        "MFC_SINGLE_PRECISION": 1,
+        "MFC_OPENACC": 1,
+        "MFC_MPI": 1
+    },
+    "lowercase_intrinsics": true,
+    "debug_log": false,
+    "disable_diagnostics": false,
+    "use_signature_help": true,
+    "variable_hover": true,
+    "hover_signature": true,
+    "enable_code_actions": true,
+    "mod_dirs": [
+        "build/pre_process/",
+        "build/simulation/",
+        "build/post_process/",
+        "build/common/"
+    ],
+    "ext_mod_dirs": [
+        "/usr/include/",
+        "/usr/local/include/",
+        "/opt/homebrew/include/"
+    ],
+    "implicit_external_mods": [
+        "mpi",
+        "m_thermochem",
+        "m_variables_conversion",
+        "hipfort",
+        "hipfort_check",
+        "hipfort_hipfft",
+        "cutensorex",
+        "silo_f9x",
+        "m_model"
+    ],
+    "disable_diagnostics_for_external_modules": true,
+    "max_line_length": -1,
+    "max_comment_line_length": -1,
+    "disable_var_diagnostics": false,
+    "disable_fypp": false,
+    "fypp_strict": false,
+    "incremental_sync": false,
+    "debug_parser": true,
+    "skip_parse_errors": true,
+    "disable_parser": [
+        "src/post_process/m_data_output.fpp",
+        "src/pre_process/include/ExtrusionHardcodedIC.fpp",
+        "src/pre_process/m_checker.fpp",
+        "src/pre_process/include/2dHardcodedIC.fpp",
+        "src/pre_process/include/3dHardcodedIC.fpp",
+        "src/simulation/m_qbmm.fpp",
+        "src/common/m_variables_conversion.fpp",
+        "src/simulation/m_global_parameters.fpp",
+        "**/m_nvtx*",
+        "**/syscheck.fpp"
+    ]
+} 

.github/workflows/bench.yml

@@ -97,10 +97,11 @@ jobs:
         run: |
           cat pr/bench-${{ matrix.device }}.* 2>/dev/null || true
           cat master/bench-${{ matrix.device }}.* 2>/dev/null || true
-      
-      - name: Archive Logs
+
+      # All other runners (non-Phoenix) just run without special env
+      - name: Archive Logs (Frontier)
+        if: always() && matrix.cluster != 'phoenix'
         uses: actions/upload-artifact@v4
-        if: always()
         with:
           name: ${{ matrix.cluster }}-${{ matrix.device }}
           path: |

.github/workflows/frontier/submit-bench.sh

@@ -32,7 +32,7 @@ sbatch <<EOT
 #SBATCH -A CFD154                  # charge account
 #SBATCH -N 1                       # Number of nodes required
 $sbatch_device_opts
-#SBATCH -t 03:59:00                # Duration of the job (Ex: 15 mins)
+#SBATCH -t 02:59:00                # Duration of the job (Ex: 15 mins)
 #SBATCH -o$job_slug.out            # Combined output and error messages file
 #SBATCH -p extended                # Extended partition for shorter queues
 #SBATCH -W                         # Do not exit until the submitted job terminates.

.github/workflows/lint-source.yml

@@ -34,6 +34,10 @@ jobs:
         find ./src -type f -not -name '*nvtx*' -exec sh -c 'fortitude check "$1" | grep -v E001' _ {} \;
         find ./src -type f -not -name '*nvtx*' -exec sh -c 'fortitude check "$1" | grep -v E001' _ {} \; | wc -l | xargs -I{} sh -c '[ {} -gt 0 ] && exit 1 || exit 0'
 
+    - name: Looking for raw directives
+      run: |
+        ! grep -iR '!\$acc\|!\$omp' --exclude="parallel_macros.fpp" --exclude="syscheck.fpp" ./src/*
+        
     - name: No double precision intrinsics
       run: |
         ! grep -iR 'double_precision\|dsqrt\|dexp\|dlog\|dble\|dabs\|double\ precision\|real(8)\|real(4)\|dprod\|dmin\|dmax\|dfloat\|dreal\|dcos\|dsin\|dtan\|dsign\|dtanh\|dsinh\|dcosh\|d0' --exclude-dir=syscheck --exclude="*nvtx*" --exclude="*precision_select*" ./src/*

.github/workflows/phoenix/bench.sh

@@ -2,7 +2,8 @@
 
 n_ranks=12
 
-if [ "$job_device" = "gpu" ]; then
+echo "My benchmarking device is:" $device
+if [ "$device" = "gpu" ]; then
     n_ranks=$(nvidia-smi -L | wc -l)        # number of GPUs on node
     gpu_ids=$(seq -s ' ' 0 $(($n_ranks-1))) # 0,1,2,...,gpu_count-1
     device_opts="--gpu -g $gpu_ids"
@@ -15,7 +16,7 @@ mkdir -p $currentdir
 
 export TMPDIR=$currentdir
 
-if [ "$job_device" = "gpu" ]; then
+if [ "$device" = "gpu" ]; then
     ./mfc.sh bench --mem 12 -j $(nproc) -o "$job_slug.yaml" -- -c phoenix-bench $device_opts -n $n_ranks
 else
     ./mfc.sh bench --mem 1 -j $(nproc) -o "$job_slug.yaml" -- -c phoenix-bench $device_opts -n $n_ranks