Skip to content

Commit 64dbf1f

Browse files
sbryngelsonsbryngelson
authored
Merge pull request #117 from henryleberre/merge
2 parents d929c4b + be76f43 commit 64dbf1f

File tree

26 files changed

+1019
-633
lines changed

26 files changed

+1019
-633
lines changed

‎.gitignore‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,10 @@ __pycache__
2121
/tests/*/**
2222
!/tests/golden.txt
2323

24+
# NVIDIA Nsight Compute
25+
*.nsys-rep
26+
*.sqlite
27+
2428
examples/*batch/*/
2529
examples/*/D/*
2630
examples/*/p*
@@ -34,8 +38,5 @@ examples/*/restart*
3438
examples/*/*.out
3539
examples/*/binary
3640
examples/*/fort.1
37-
examples/*/pre_process.sh
38-
examples/*/simulation.sh
39-
examples/*/post_process.sh
40-
examples/*/fort.1
4141
examples/*/*.sh
42+
examples/*/*.err

‎docs/documentation/case.md‎

Lines changed: 554 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/documentation/readme.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,9 @@
33
## User Documentation
44

55
- [Getting Started](getting-started.md)
6-
- [Running MFC](running.md)
76
- [Testing MFC](testing.md)
7+
- [Case Files](case.md)
8+
- [Running MFC](running.md)
89
- [Flow Visualisation](visualisation.md)
910
- [MFC's Authors](authors.md)
1011
- [References](references.md)
@@ -13,7 +14,7 @@
1314

1415
MFC's three codes have their own documentation:
1516

16-
- [Pre Process](../pre_process/)
17+
- [Pre-Process](../pre_process/)
1718
- [Simulation](../simulation/)
18-
- [Post Process](../post_process/)
19+
- [Post-Process](../post_process/)
1920

‎docs/documentation/running.md‎

Lines changed: 4 additions & 533 deletions
Large diffs are not rendered by default.

‎docs/documentation/testing.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@ To (re)generate *golden files*, append the `-g` (i.e `--generate`) option:
2222
$ ./mfc.sh test -g -j 8
2323
```
2424

25+
It is recommended that a range be specified when generating golden files for new test cases, as described in the previous section, in an effort not to regenerate the golden files of existing test cases.
26+
2527
Adding a new test case can be done by modifying [cases.py](toolchain/mfc/tests/cases.py). The function `generate_cases` is responsible for generating the list of test cases. Loops and conditionals are used to vary parameters, whose defaults can be found in the `BASE_CFG` case object within [case.py](toolchain/mfc/tests/case.py). The function operates on two variables:
2628

2729
- `stack`: A stack that holds the variations to the default case parameters. By pushing and popping the stack inside loops and conditionals, it is easier to nest test case descriptions, as it holds the variations that are common to all future test cases within the same indentation level (in most scenarios).

‎examples/3D_weak_scaling/README.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# 3D Weak Scaling
2+
3+
The [**3D_weak_scaling**](case.py) case depends on two parameters:
4+
5+
- **The number of MPI ranks** (_procs_): As _procs_ increases, the problem
6+
size per rank remains constant. _procs_ is determined using information provided
7+
to the case file by `mfc.sh run`.
8+
9+
- **GPU memory usage per rank** (_gbpp_): As _gbpp_ increases, the problem
10+
size per rank increases and the number of timesteps decreases so that wall times
11+
consistent. _gbpp_ is a user-defined optional argument to the [case.py](case.py)
12+
file. It can be specified right after the case filepath when invoking `mfc.sh run`.
13+
14+
Weak scaling benchmarks can be produced by keeping _gbpp_ constant and varying _procs_.
15+
16+
For example, to run a weak scaling test that uses ~4GB of GPU memory per rank
17+
on 8 2-rank nodes with case optimization, one could:
18+
19+
```console
20+
./mfc.sh run examples/3D_weak_scaling/case.py 4 -t pre_process simulation \
21+
-e batch -p mypartition -N 8 -n 2 -w "01:00:00" -# "MFC Weak Scaling" \
22+
--case-optimization -j 32
23+
```
24+

‎examples/3D_weak_scaling/case.py‎

Lines changed: 287 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,287 @@
1+
#!/usr/bin/env python3
2+
3+
# Case file contributed by Anand Radhakrishnan and modified by Henry Le Berre
4+
# for integration as a weak scaling benchmark for MFC.
5+
6+
import json, math, argparse
7+
8+
parser = argparse.ArgumentParser(
9+
prog="3D_weak_scaling",
10+
description="This MFC case was created for the purposes of weak scaling.",
11+
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
12+
13+
parser.add_argument("dict", type=str, metavar="DICT", help=argparse.SUPPRESS)
14+
parser.add_argument("gbpp", type=int, metavar="MEM", default=16, help="Adjusts the problem size per rank to fit into [MEM] GB of GPU memory per GPU.")
15+
16+
ARGS = vars(parser.parse_args())
17+
DICT = json.loads(ARGS["dict"])
18+
19+
ppg = 8000000 / 16.0
20+
procs = DICT["nodes"] * DICT["tasks_per_node"]
21+
ncells = math.floor(ppg * procs * ARGS["gbpp"])
22+
s = math.floor((ncells / 2.0) ** (1/3))
23+
Nx, Ny, Nz = 2*s, s, s
24+
25+
# athmospheric pressure - Pa (used as reference value)
26+
patm = 101325
27+
28+
# Initial Droplet Diameter / Reference length - m
29+
D0 = 1.0E-3
30+
31+
# cavity to droplet ratio
32+
CtD = 0.06
33+
34+
# cavity relative eccentricity (distance between radii)
35+
ecc = 0.564
36+
37+
# initial shock distance from the y axis. Note that the droplet center is located at y = 0. Thus, the distance from the shock to
38+
# the droplet is about D0/8
39+
ISD = 5.0/8 * D0
40+
41+
## pre-shock properties - AIR
42+
43+
# pressure - Pa
44+
p0a = patm
45+
46+
# density - kg/m3
47+
rho0a = 1.204
48+
49+
# gamma
50+
gama = 1.40
51+
52+
# pi infinity - Pa
53+
pia = 0
54+
55+
# speed of sound - M/s
56+
c_a = math.sqrt( gama * ( p0a + pia ) / rho0a )
57+
58+
## Droplet - WATER
59+
60+
# surface tension - N / m
61+
st = 0.00E+0
62+
63+
# Delta Pressure - Pa
64+
DP = -st * 4 / D0
65+
66+
# initial pressure inside the droplet - Pa
67+
p0w = p0a - DP
68+
69+
# density - kg/m3
70+
rho0w = 1000
71+
72+
# gama
73+
gamw = 6.12
74+
75+
# pi infty - Pa
76+
piw = 3.43E+08
77+
78+
# speed of sound - m/s
79+
c_w = math.sqrt( gamw * ( p0w + piw ) / rho0w )
80+
81+
# Shock Mach number of interest. Note that the post-shock properties can be defined in terms of either
82+
# Min or psOp0a. Just comment/uncomment appropriatelly
83+
Min = 2.4
84+
85+
## Pos to pre shock ratios - AIR
86+
87+
# pressure
88+
psOp0a = ( Min ** 2 -1 ) * 2 * gama / ( gama + 1 ) + 1
89+
# psOp0a = 4.5
90+
91+
# density
92+
rhosOrho0a = ( 1 + ( gama + 1 ) / ( gama - 1) * psOp0a ) / ( ( gama + 1 ) / ( gama - 1) + psOp0a )
93+
94+
# Mach number of the shocked region - just a checker, as it must return "Min"
95+
Ms = math.sqrt( ( gama + 1. ) / ( 2. * gama ) * ( psOp0a - 1. ) * ( p0a / ( p0a + pia ) ) + 1. )
96+
97+
# shock speed of sound - m/s
98+
ss = Ms * c_a
99+
100+
## post-shock - AIR
101+
102+
# pressure - Pa
103+
ps = psOp0a * p0a
104+
105+
# density - kg / m3
106+
rhos = rhosOrho0a * rho0a
107+
108+
# post shock speed of sound - m/s
109+
c_s = math.sqrt( gama * (ps + pia) / rhos )
110+
111+
# velocity at the post shock - m/s
112+
vel = c_a/gama * (psOp0a - 1.) * p0a / ( p0a + pia ) / Ms
113+
114+
## Domain boundaries - m
115+
116+
# x direction
117+
xb = -8.4707 * D0
118+
xe = 9.6226 * D0
119+
120+
#xb = -10 * D0
121+
#xe = 10 * D0
122+
123+
# y direction
124+
yb = 0 * D0
125+
ye = 10 * D0
126+
127+
# y direction
128+
zb = 0 * D0
129+
ze = 10 * D0
130+
131+
# Stretching factor, to make sure the domaing is sufficiently large after the mesh strecth
132+
StF = 4.0
133+
134+
# grid delta x if mesh were uniform in x direction - m. Note that I do not need a measure for dy
135+
dx = ( xe - xb ) / Nx
136+
137+
# I calculate tend twice; first is an estimate, second is
138+
# the actual value used. This is because I am getting errors in the
139+
# post process part every time I approximate the actual Nt by an integer
140+
# number (think of a smarter way).
141+
142+
# dimensionless time
143+
ttilde = 1.92
144+
145+
# auxiliary simulation physical time - s. This is not YET the total simulation time, as it will be corrected so as to avoid
146+
# mismatches in simulation and post_process parts. Note that I wrote it this way so I have better control over the # of autosaves
147+
tendA = ttilde * D0 / vel
148+
149+
cfl = 0.1
150+
151+
# time-step - s
152+
dt = dx * cfl/ ss
153+
154+
# Save Frequency. Note that the number of autosaves will be SF + 1, as th IC (0.dat) is also saved
155+
SF = 400
156+
157+
## making Nt divisible by SF
158+
# 1 - ensure NtA goes slightly beyond tendA
159+
NtA = int( tendA//dt + 1 )
160+
161+
# Array of saves. It is the same as Nt/Sf = t_step_save
162+
AS = int( NtA // SF + 1 )
163+
164+
# Nt = total number of steps. Note that Nt >= NtA (so at least tendA is completely simulated)
165+
Nt = AS * SF
166+
167+
# total simulation time - s. Note that tend >= tendA
168+
tend = Nt * dt
169+
170+
# Configuring case dictionary
171+
print(json.dumps({
172+
# Logistics ================================================
173+
'run_time_info' : 'T',
174+
# ==========================================================
175+
176+
# Computational Domain Parameters ==========================
177+
'x_domain%beg' : xb,
178+
'x_domain%end' : xe,
179+
'y_domain%beg' : yb,
180+
'y_domain%end' : ye,
181+
'z_domain%beg' : zb,
182+
'z_domain%end' : ze,
183+
'm' : Nx,
184+
'n' : Ny,
185+
'p' : Nz,
186+
'cyl_coord' : 'F',
187+
'dt' : dt,
188+
't_step_start' : 0,
189+
't_step_stop' : int(5000*16.0/ARGS["gbpp"]),
190+
't_step_save' : int(1000*16.0/ARGS["gbpp"]),
191+
# ==========================================================
192+
193+
# Simulation Algorithm Parameters ==========================
194+
'num_patches' : 3,
195+
'model_eqns' : 2,
196+
'alt_soundspeed' : 'F',
197+
'num_fluids' : 2,
198+
'adv_alphan' : 'T',
199+
'mpp_lim' : 'T',
200+
'mixture_err' : 'T',
201+
'time_stepper' : 3,
202+
'weno_vars' : 2,
203+
'weno_order' : 3,
204+
'weno_eps' : 1.0E-16,
205+
'mapped_weno' : 'T',
206+
'riemann_solver' : 2,
207+
'wave_speeds' : 1,
208+
'avg_state' : 2,
209+
'bc_x%beg' : -6,
210+
'bc_x%end' : -6,
211+
'bc_y%beg' : -2,
212+
'bc_y%end' : -3,
213+
'bc_z%beg' : -2,
214+
'bc_z%end' : -3,
215+
# ==========================================================
216+
217+
# Formatted Database Files Structure Parameters ============
218+
'format' : 1,
219+
'precision' : 2,
220+
'prim_vars_wrt' :'T',
221+
'parallel_io' :'T',
222+
# ==========================================================
223+
# I will use 1 for WATER properties, and 2 for AIR properties
224+
# Patch 1: Background (AIR - 2) =============================
225+
'patch_icpp(1)%geometry' : 9,
226+
'patch_icpp(1)%x_centroid' : (xb+xe) / 2 * StF,
227+
'patch_icpp(1)%y_centroid' : (yb+ye) / 2 * StF,
228+
'patch_icpp(1)%z_centroid' : (yb+ye) / 2 * StF,
229+
'patch_icpp(1)%length_x' : (xe-xb) * StF,
230+
'patch_icpp(1)%length_y' : (ye-yb) * StF,
231+
'patch_icpp(1)%length_z' : (ze-zb) * StF,
232+
'patch_icpp(1)%vel(1)' : 0.0E+00,
233+
'patch_icpp(1)%vel(2)' : 0.0E+00,
234+
'patch_icpp(1)%vel(3)' : 0.0E+00,
235+
'patch_icpp(1)%pres' : p0a,
236+
'patch_icpp(1)%alpha_rho(1)' : 0.0E+00,
237+
'patch_icpp(1)%alpha_rho(2)' : rho0a,
238+
'patch_icpp(1)%alpha(1)' : 0.0E+00,
239+
'patch_icpp(1)%alpha(2)' : 1.0E+00,
240+
# ==========================================================
241+
242+
# Patch 2: Shocked state (AIR - 2) =========================
243+
'patch_icpp(2)%geometry' : 9,
244+
'patch_icpp(2)%alter_patch(1)' : 'T',
245+
'patch_icpp(2)%x_centroid' : -ISD - ( xe - xb ) / 2 * StF,
246+
'patch_icpp(2)%y_centroid' : ( yb + ye ) / 2 * StF,
247+
'patch_icpp(2)%z_centroid' : ( zb + ze ) / 2 * StF,
248+
'patch_icpp(2)%length_x' : ( xe - xb ) * StF,
249+
'patch_icpp(2)%length_y' : ( ye - yb ) * StF,
250+
'patch_icpp(2)%length_z' : ( ze - zb ) * StF,
251+
'patch_icpp(2)%vel(1)' : vel,
252+
'patch_icpp(2)%vel(2)' : 0.0E+00,
253+
'patch_icpp(2)%vel(3)' : 0.0E+00,
254+
'patch_icpp(2)%pres' : ps,
255+
'patch_icpp(2)%alpha_rho(1)' : 0.0E+00,
256+
'patch_icpp(2)%alpha_rho(2)' : rhos,
257+
'patch_icpp(2)%alpha(1)' : 0.0E+00,
258+
'patch_icpp(2)%alpha(2)' : 1.0E+00,
259+
# ==========================================================
260+
261+
# Patch 3: Droplet (WATER - 1) =============================
262+
'patch_icpp(3)%geometry' : 8,
263+
'patch_icpp(3)%x_centroid' : 0.0E+00,
264+
'patch_icpp(3)%y_centroid' : 0.0E+00,
265+
'patch_icpp(3)%z_centroid' : 0.0E+00,
266+
'patch_icpp(3)%radius' : D0/2,
267+
'patch_icpp(3)%alter_patch(1)' : 'T',
268+
'patch_icpp(3)%vel(1)' : 0.0E+00,
269+
'patch_icpp(3)%vel(2)' : 0.0E+00,
270+
'patch_icpp(3)%vel(3)' : 0.0E+00,
271+
'patch_icpp(3)%pres' : p0w,
272+
'patch_icpp(3)%alpha_rho(1)' : rho0w,
273+
'patch_icpp(3)%alpha_rho(2)' : 0.0E+00,
274+
'patch_icpp(3)%alpha(1)' : 1.0E+00,
275+
'patch_icpp(3)%alpha(2)' : 0.0E+00,
276+
# ==========================================================
277+
278+
# Fluids Physical Parameters ===============================
279+
'fluid_pp(1)%gamma' : 1.0E+00/(gamw-1),
280+
'fluid_pp(1)%pi_inf' : gamw*piw/(gamw-1),
281+
'fluid_pp(2)%gamma' : 1.0E+00/(gama-1),
282+
'fluid_pp(2)%pi_inf' : gama*pia/(gama-1),
283+
# ==========================================================
284+
}))
285+
286+
# ==============================================================================
287+

‎mfc.sh‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -77,12 +77,12 @@ if [ "$1" == "load" ]; then
7777
if [ "$u_cg" == "c" ]; then
7878
MODULES=("gcc/12.1.0")
7979
elif [ "$u_cg" == "g" ]; then
80-
MODULES=("nvhpc/22.5" "cuda/nvhpc")
80+
MODULES=("nvhpc/22.11" "cuda/nvhpc")
8181
fi
8282

8383
MODULES=("${MODULES[@]}" "python/3.8.10" "darshan-runtime/3.3.1-lite"
8484
"hsi/5.0.2.p5" "xalt/1.2.1" "lsf-tools/2.0"
85-
"cmake/3.23.1" "ninja/1.10.2" "spectrum-mpi/10.4.0.3-20210112")
85+
"cmake/3.23.2" "ninja/1.10.2" "spectrum-mpi/10.4.0.3-20210112")
8686
elif [ "$u_computer" == "b" ]; then # Bridges2
8787
if [ "$u_cg" == "c" ]; then
8888
MODULES=("allocations/1.0" "gcc/10.2.0" "python/3.8.6"
@@ -115,10 +115,10 @@ if [ "$1" == "load" ]; then
115115
if [ "$u_cg" == "c" ]; then
116116
MODULES=("gcc/11.1.0" "openmpi/4.0.5_gcc")
117117
elif [ "$u_cg" == "g" ]; then
118-
MODULES=("cuda/11.5.1" "/sw/wombat/Nvidia_HPC_SDK/modulefiles/nvhpc/22.1")
118+
MODULES=("nvhpc/22.11")
119119
fi
120120

121-
MODULES=("${MODULES[@]}" "cmake/3.22.1" "python/3.9.9")
121+
MODULES=("${MODULES[@]}" "cmake/3.25.1" "python/3.10.8")
122122
elif [ "$u_computer" == "e" ]; then # Expanse
123123
if [ "$u_cg" == "c" ]; then
124124
warn "Please set CC=icc, CXX=icx, and FC=ifort."

0 commit comments

Comments
 (0)