Skip to content

Commit cc8541c

Browse files
wilfonbawilfonba
committed
Merge remote-tracking branch 'upstream/master' into comprehensiveDebug
2 parents f885194 + ecdf5b1 commit cc8541c

File tree

11 files changed

+57
-38
lines changed

11 files changed

+57
-38
lines changed

‎docs/documentation/expectedPerformance.md‎

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,8 @@ The following table outlines observed performance as nanoseconds per grid point
99
We solve an example 3D, inviscid, 5-equation model problem with two advected species (8 PDEs) and 8M grid points (158-cubed uniform grid).
1010
The numerics are WENO5 finite volume reconstruction and HLLC approximate Riemann solver.
1111
This case is located in `examples/3D_performance_test`.
12-
You can run it via `./mfc.sh run -n <num_processors> -j $(nproc) ./examples/3D_performance_test/case.py -t pre_process simulation --case-optimization`, which will build an optimized version of the code for this case then execute it.
12+
You can run it via `./mfc.sh run -n <num_processors> -j $(nproc) ./examples/3D_performance_test/case.py -t pre_process simulation --case-optimization` for CPU cases right after building MFC, which will build an optimized version of the code for this case then execute it.
13+
For benchmarking GPU devices, you will likely want to use `-n <num_gpus>` where `<num_gpus>` should likely be `1`.
1314
If the above does not work on your machine, see the rest of this documentation for other ways to use the `./mfc.sh run` command.
1415

1516
Results are for MFC v4.9.3 (July 2024 release), though numbers have not changed meaningfully since then.
@@ -18,27 +19,33 @@ All results are for the compiler that gave the best performance.
1819
Note:
1920
* CPU results may be performed on CPUs with more cores than reported in the table; we report results for the best performance given the full processor die by checking the performance for different core counts on that device. CPU results are the best performance we achieved using a single socket (or die).
2021
These are reported as (X/Y cores), where X is the used cores, and Y is the total on the die.
21-
* GPU results are for a single GPU device. For single-precision (SP) GPUs, we performed computation in double-precision via conversion in compiler/software; these numbers are _not_ for single-precision computation. AMD MI250X and MI300A GPUs have multiple graphics compute dies (GCDs) per device; we report results for one _GCD_*, though one can quickly estimate full device runtime by dividing the grind time number by the number of GCDs on the device (the MI250X has 2 GCDs). We gratefully acknowledge the permission of LLNL, HPE/Cray, and AMD for permission to release MI300A performance numbers.
22+
* GPU results are for a single GPU device. For single-precision (SP) GPUs, we performed computation in double-precision via conversion in compiler/software; these numbers are _not_ for single-precision computation. AMD MI250X and MI300A GPUs have multiple compute dies per socket; we report results for one _GCD_* for the MI250X and the entire APU (6 XCDs) for MI300A, though one can quickly estimate full device runtime by dividing the grind time number by the number of GCDs on the device (the MI250X has 2 GCDs). We gratefully acknowledge the permission of LLNL, HPE/Cray, and AMD for permission to release MI300A performance numbers.
2223

2324
| Hardware | Details | Type | Usage | Grind Time [ns] | Compiler | Computer |
2425
| ---: | ----: | ----: | ----: | ----: | :--- | :--- |
2526
| NVIDIA GH200 | GPU only | APU | 1 GPU | 0.32 | NVHPC 24.1 | GT Rogues Gallery |
26-
| NVIDIA H100 | | GPU | 1 GPU | 0.45 | NVHPC 24.5 | GT Rogues Gallery |
27-
| AMD MI300A | | APU | 1 _GCD_* | 0.60 | CCE 18.0.0 | LLNL Tioga |
27+
| NVIDIA H100 SXM5 | | GPU | 1 GPU | 0.38 | NVHPC 24.5 | GT ICE |
28+
| NVIDIA H100 PCIe | | GPU | 1 GPU | 0.45 | NVHPC 24.5 | GT Rogues Gallery |
29+
| AMD MI300A | | APU | 1 APU | 0.60 | CCE 18.0.0 | LLNL Tioga |
2830
| NVIDIA A100 | | GPU | 1 GPU | 0.62 | NVHPC 22.11 | GT Phoenix |
2931
| NVIDIA V100 | | GPU | 1 GPU | 0.99 | NVHPC 22.11 | GT Phoenix |
3032
| NVIDIA A30 | | GPU | 1 GPU | 1.1 | NVHPC 24.1 | GT Rogues Gallery |
3133
| AMD MI250X | | GPU | 1 _GCD_* | 1.1 | CCE 16.0.1 | OLCF Frontier |
3234
| AMD MI100 | | GPU | 1 GPU | 1.4 | CCE 16.0.1 | Cray internal system |
3335
| NVIDIA L40S | FP32-only GPU | GPU | 1 GPU | 1.7 | NVHPC 24.5 | GT ICE |
34-
| AMD EPYC 9654 | Genoa | CPU | 96 cores | 1.7 | Intel 2021.9 | DOD Carpenter |
36+
| AMD EPYC 9654 | Genoa | CPU | 96 cores | 1.7 | Intel 2021.9 | DOD Carpenter |
37+
| Intel Xeon 6960P | Granite Rapids | CPU | 72 cores | 1.7 | Intel 2024.2 | Intel AI Cloud |
3538
| NVIDIA P100 | | GPU | 1 GPU | 2.4 | NVHPC 23.5 | GT CSE Internal |
39+
| Intel Xeon 8592+ | Emerald Rapids | CPU | 64 cores | 2.6 | Intel 2024.2 | Intel AI Cloud |
40+
| Intel Xeon SF-AP | Sierra Forest Advanced, 2.8GHz Boost, 384 MiB L3 | CPU | 192 cores | 2.6 | Intel 2024.2 | Intel AI Cloud |
3641
| AMD EPYC 9534 | Genoa | CPU | 64 cores | 2.7 | GNU 12.3.0 | GT Phoenix |
3742
| NVIDIA A40 | FP32-only GPU | GPU | 1 GPU | 3.3 | NVHPC 22.11 | NCSA Delta |
3843
| Intel Xeon Max 9468 | Sapphire Rapids HBM | CPU | 48 cores | 3.5 | NVHPC 24.5 | GT Rogues Gallery |
3944
| NVIDIA Grace CPU | Arm, Neoverse V2 | CPU | 72 cores | 3.7 | NVHPC 24.1 | GT Rogues Gallery |
4045
| NVIDIA RTX6000 | FP32-only GPU | GPU | 1 GPU | 3.9 | NVHPC 22.11 | GT Phoenix |
4146
| AMD EPYC 7763 | Milan | CPU | 64 cores | 4.1 | GNU 11.4.0 | NCSA Delta |
47+
| Intel Xeon 6740E | Sierra Forest | CPU | 92 cores | 4.2 | Intel 2024.2 | Intel AI Cloud |
48+
| NVIDIA A10 | FP32-only GPU | GPU | 1 GPU | 4.3 | NVHPC 24.1 | TAMU Faster |
4249
| AMD EPYC 7713 | Milan | CPU | 64 cores | 5.0 | GNU 12.3.0 | GT Phoenix |
4350
| Intel Xeon 8480CL | Sapphire Rapids | CPU | 56 cores | 5.0 | NVHPC 24.5 | GT Phoenix |
4451
| Intel Xeon 6454S | Sapphire Rapids | CPU | 32 cores | 5.6 | NVHPC 24.5 | GT Rogues Gallery |
@@ -48,15 +55,16 @@ These are reported as (X/Y cores), where X is the used cores, and Y is the total
4855
| Ampere Altra Q80-28 | Arm, Neoverse-N1 | CPU | 80 cores | 6.8 | GNU 12.2.0 | OLCF Wombat |
4956
| AMD EPYC 7513 | Milan | CPU | 32 cores | 7.4 | GNU 12.3.0 | GT ICE |
5057
| AMD EPYC 7452 | Rome | CPU | 32 cores | 8.4 | GNU 12.3.0 | GT ICE |
58+
| NVIDIA T4 | FP32-only GPU | GPU | 1 GPU | 8.8 | NVHPC 24.1 | TAMU Faster |
5159
| IBM Power10 | | CPU | 24 cores | 10 | GNU 13.3.1 | GT Rogues Gallery |
5260
| AMD EPYC 7401 | Naples | CPU | 24 cores | 10 | GNU 10.3.1 | LLNL Corona |
53-
| Apple M1 Pro | | CPU | 8 cores | 14 | GNU 13.2.0 | N/A |
5461
| Intel Xeon 6226 | Cascade Lake | CPU | 12 cores | 17 | GNU 12.3.0 | GT ICE |
55-
| Apple M1 Max | | CPU | 8 cores | 18 | GNU 14.1.0 | N/A |
62+
| Apple M1 Max | | CPU | 10 cores | 20 | GNU 14.1.0 | N/A |
5663
| IBM Power9 | | CPU | 20 cores | 21 | GNU 9.1.0 | OLCF Summit |
5764
| Cavium ThunderX2 | Arm | CPU | 32 cores | 21 | GNU 13.2.0 | SBU Ookami |
5865
| Arm Cortex-A78AE | Arm, BlueField3 | CPU | 16 cores | 25 | NVHPC 24.5 | GT Rogues Gallery |
5966
| Intel Xeon E5-2650V4 | Broadwell | CPU | 12 cores | 27 | NVHPC 23.5 | GT CSE Internal |
67+
| Apple M2 | | CPU | 8 cores | 32 | GNU 14.1.0 | N/A |
6068
| Intel Xeon E7-4850V3 | Haswell | CPU | 14 cores | 34 | GNU 9.4.0 | GT CSE Internal |
6169
| Fujitsu A64FX | Arm | CPU | 48 cores | 63 | GNU 13.2.0 | SBU Ookami |
6270

‎docs/header.html‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
<meta name="viewport" content="width=device-width, initial-scale=1"/>
1919
<!--BEGIN PROJECT_NAME--><title>$projectname: $title</title><!--END PROJECT_NAME-->
2020
<!--BEGIN !PROJECT_NAME--><title>$title</title><!--END !PROJECT_NAME-->
21+
<meta name=”keywords” content="exascale, fluid dynamics, cfd, computational fluid dynamics, compressible, hpc, bryngelson, colonius, subgrid, multiphase, frontier, summit, el capitan, aurora, amd gpu, gpu, nvidia"/>
2122
<link href="$relpath^tabs.css" rel="stylesheet" type="text/css"/>
2223
<script type="text/javascript" src="$relpath^jquery.js"></script>
2324
<script type="text/javascript" src="$relpath^dynsections.js"></script>

‎docs/index.html‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@
1515
<meta http-equiv="X-UA-Compatible" content="IE=edge">
1616
<meta name="viewport" content="width=device-width, initial-scale=1.0">
1717
<title>MFC | Home</title>
18+
<meta name="description" content="MFC is an exascale multiphysics CFD solver maintained by the research groups of Profs. Bryngelson and Colonius." />
19+
<meta name=”keywords” content="exascale, fluid dynamics, cfd, computational fluid dynamics, compressible, hpc, bryngelson, colonius, subgrid, multiphase, frontier, summit, el capitan, aurora, amd gpu, gpu, nvidia"/>
1820
<script src="https://cdn.tailwindcss.com"></script>
1921
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.0/css/all.min.css">
2022
<link rel="icon" type="image/x-icon" href="res/icon.ico">

‎src/common/m_thermochem.fpp‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,16 @@
33
module m_thermochem
44

55
#:if chemistry
6-
use m_pyrometheus
7-
#:else
6+
use m_pyrometheus, only: &
7+
num_species, species_names, gas_constant, mol_weights, &
8+
get_temperature, get_net_production_rates, get_pressure, &
9+
get_mixture_molecular_weight, get_mixture_energy_mass, &
10+
get_temperature
11+
#:endif
12+
13+
implicit none
14+
15+
#:if not chemistry
816
integer, parameter :: num_species = 0
917
character(len=:), allocatable, dimension(:) :: species_names
1018
#:endif

‎src/post_process/m_global_parameters.fpp‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -127,7 +127,7 @@ module m_global_parameters
127127
integer :: pi_inf_idx !< Index of liquid stiffness func. eqn.
128128
type(int_bounds_info) :: stress_idx !< Indices of elastic stresses
129129
integer :: c_idx !< Index of color function
130-
type(int_bounds_info) :: chemistry_idx !< Indexes of first & last concentration eqns.
130+
type(int_bounds_info) :: species_idx !< Indexes of first & last concentration eqns.
131131
type(int_bounds_info) :: temperature_idx !< Indexes of first & last temperature eqns.
132132
!> @}
133133

@@ -613,13 +613,18 @@ contains
613613
end if
614614

615615
if (chemistry) then
616-
chemistry_idx%beg = sys_size + 1
617-
chemistry_idx%end = sys_size + num_species
618-
sys_size = chemistry_idx%end
616+
species_idx%beg = sys_size + 1
617+
species_idx%end = sys_size + num_species
618+
sys_size = species_idx%end
619619

620620
temperature_idx%beg = sys_size + 1
621621
temperature_idx%end = sys_size + 1
622622
sys_size = temperature_idx%end
623+
else
624+
species_idx%beg = 1
625+
species_idx%end = 1
626+
temperature_idx%beg = 1
627+
temperature_idx%end = 1
623628
end if
624629

625630
momxb = mom_idx%beg
@@ -634,8 +639,8 @@ contains
634639
strxe = stress_idx%end
635640
intxb = internalEnergies_idx%beg
636641
intxe = internalEnergies_idx%end
637-
chemxb = chemistry_idx%beg
638-
chemxe = chemistry_idx%end
642+
chemxb = species_idx%beg
643+
chemxe = species_idx%end
639644
tempxb = temperature_idx%beg
640645
tempxe = temperature_idx%end
641646

‎src/pre_process/m_global_parameters.fpp‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,7 @@ module m_global_parameters
105105
integer :: pi_inf_idx !< Index of liquid stiffness func. eqn.
106106
type(int_bounds_info) :: stress_idx !< Indexes of elastic shear stress eqns.
107107
integer :: c_idx !< Index of the color function
108-
type(int_bounds_info) :: chemistry_idx !< Indexes of first & last concentration eqns.
108+
type(int_bounds_info) :: species_idx !< Indexes of first & last concentration eqns.
109109
type(int_bounds_info) :: temperature_idx !< Indexes of first & last temperature eqns.
110110

111111
type(int_bounds_info) :: bc_x, bc_y, bc_z !<
@@ -696,9 +696,9 @@ contains
696696
end if
697697

698698
if (chemistry) then
699-
chemistry_idx%beg = sys_size + 1
700-
chemistry_idx%end = sys_size + num_species
701-
sys_size = chemistry_idx%end
699+
species_idx%beg = sys_size + 1
700+
species_idx%end = sys_size + num_species
701+
sys_size = species_idx%end
702702

703703
temperature_idx%beg = sys_size + 1
704704
temperature_idx%end = sys_size + 1
@@ -717,8 +717,8 @@ contains
717717
strxe = stress_idx%end
718718
intxb = internalEnergies_idx%beg
719719
intxe = internalEnergies_idx%end
720-
chemxb = chemistry_idx%beg
721-
chemxe = chemistry_idx%end
720+
chemxb = species_idx%beg
721+
chemxe = species_idx%end
722722
tempxb = temperature_idx%beg
723723
tempxe = temperature_idx%end
724724

‎src/simulation/m_global_parameters.fpp‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -233,7 +233,7 @@ module m_global_parameters
233233
integer :: pi_inf_idx !< Index of liquid stiffness func. eqn.
234234
type(int_bounds_info) :: stress_idx !< Indexes of first and last shear stress eqns.
235235
integer :: c_idx ! Index of the color function
236-
type(int_bounds_info) :: chemistry_idx !< Indexes of first & last concentration eqns.
236+
type(int_bounds_info) :: species_idx !< Indexes of first & last concentration eqns.
237237
type(int_bounds_info) :: temperature_idx !< Indexes of first & last temperature eqns.
238238
!> @}
239239

@@ -287,7 +287,7 @@ module m_global_parameters
287287

288288
integer :: startx, starty, startz
289289

290-
!$acc declare create(sys_size, buff_size, startx, starty, startz, E_idx, gamma_idx, pi_inf_idx, alf_idx, n_idx, stress_idx, chemistry_idx)
290+
!$acc declare create(sys_size, buff_size, startx, starty, startz, E_idx, gamma_idx, pi_inf_idx, alf_idx, n_idx, stress_idx, species_idx)
291291

292292
! END: Simulation Algorithm Parameters =====================================
293293

@@ -1072,9 +1072,9 @@ contains
10721072
end if
10731073

10741074
if (chemistry) then
1075-
chemistry_idx%beg = sys_size + 1
1076-
chemistry_idx%end = sys_size + num_species
1077-
sys_size = chemistry_idx%end
1075+
species_idx%beg = sys_size + 1
1076+
species_idx%end = sys_size + num_species
1077+
sys_size = species_idx%end
10781078

10791079
temperature_idx%beg = sys_size + 1
10801080
temperature_idx%end = sys_size + 1
@@ -1093,13 +1093,13 @@ contains
10931093
strxe = stress_idx%end
10941094
intxb = internalEnergies_idx%beg
10951095
intxe = internalEnergies_idx%end
1096-
chemxb = chemistry_idx%beg
1097-
chemxe = chemistry_idx%end
1096+
chemxb = species_idx%beg
1097+
chemxe = species_idx%end
10981098
tempxb = temperature_idx%beg
10991099
tempxe = temperature_idx%end
11001100

11011101
!$acc update device(momxb, momxe, advxb, advxe, contxb, contxe, bubxb, bubxe, intxb, intxe, sys_size, buff_size, E_idx, alf_idx, n_idx, adv_n, adap_dt, pi_fac, strxb, strxe, chemxb, chemxe, tempxb, tempxe)
1102-
!$acc update device(chemistry_idx)
1102+
!$acc update device(species_idx)
11031103
!$acc update device(cfl_target, m, n, p)
11041104

11051105
!$acc update device(alt_soundspeed, acoustic_source, num_source)

‎toolchain/mfc/build.py‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,6 @@ def configure(self, case: input.MFCInputFile):
102102
install_prefixes = ';'.join([
103103
t.get_install_dirpath(case) for t in self.requires.compute()
104104
])
105-
mod_dirs = f"{HIPFORT.get_install_dirpath(case)}/include/hipfort/amdgcn"
106105

107106
flags: list = self.flags.copy() + [
108107
# Disable CMake warnings intended for developers (us).
@@ -133,9 +132,6 @@ def configure(self, case: input.MFCInputFile):
133132
# Location prefix to install bin/, lib/, include/, etc.
134133
# See: https://cmake.org/cmake/help/latest/command/install.html.
135134
f"-DCMAKE_INSTALL_PREFIX={install_dirpath}",
136-
# Fortran .mod include directories. Currently used for the HIPFORT
137-
# dependency that has this missing from its config files.
138-
f"-DCMAKE_Fortran_MODULE_DIRECTORY={mod_dirs}",
139135
]
140136

141137
if ARG("verbose"):

‎toolchain/modules‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ f-gpu craype-accel-amd-gfx90a
5454
d NCSA Delta
5555
d-all python/3.11.6
5656
d-cpu gcc/11.4.0 openmpi
57-
d-gpu nvhpc/22.11 openmpi+cuda/4.1.5+cuda cmake
57+
d-gpu nvhpc/24.1 cuda/12.3.0 openmpi/4.1.5+cuda cmake
5858
d-gpu CC=nvc CXX=nvc++ FC=nvfortran
5959

6060
c DoD Carpenter

‎toolchain/requirements.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,4 +25,4 @@ pandas
2525

2626
# Chemistry
2727
cantera
28-
pyrometheus @ git+https://github.com/henryleberre/pyrometheus.git@multi-lang-part-2
28+
pyrometheus @ git+https://github.com/pyrometheus/pyrometheus.git@mfc

0 commit comments

Comments
 (0)