You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+7-9Lines changed: 7 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -20,8 +20,8 @@
20
20
</p>
21
21
22
22
Welcome to the home of MFC!
23
-
MFC simulates compressible multi-component and multi-phase flows, amongst other things.
24
-
It scales <b>ideally to exascale</b>; [tens of thousands of GPUs on NVIDIA- and AMD-GPU Machines](#is-this-really-exascale), like Oak Ridge Summit and Frontier.
23
+
MFC simulates compressible multi-component and multi-phase flows, [amongst other things](#what-else-can-this-thing-do).
24
+
It scales <b>ideally to exascale</b>; [tens of thousands of GPUs on NVIDIA- and AMD-GPU machines](#is-this-really-exascale), like Oak Ridge Summit and Frontier.
25
25
MFC is written in Fortran and makes use of metaprogramming to keep the code short (about 20K lines).
26
26
27
27
Get in touch with the maintainers, like <ahref="mailto:[email protected]">Spencer</a>, if you have questions!
@@ -96,7 +96,7 @@ They are organized below, just click the drop-downs!
96
96
* Multi- and single-phase
97
97
* Phase change via p, pT, and pTg schemes
98
98
* Grids
99
-
* 1-3D Cartesian, Cylindrical, Axi-symmetric.
99
+
* 1-3D Cartesian, cylindrical, axi-symmetric.
100
100
* Arbitrary grid stretching for multiple domain regions available.
101
101
* Complex/arbitrary geometries via immersed boundary methods
102
102
* STL geometry files supported
@@ -162,12 +162,10 @@ If you use MFC, consider citing it:
162
162
163
163
## License
164
164
165
-
Copyright 2021-2024.
165
+
Copyright 2021-2024 Spencer Bryngelson and Tim Colonius.
166
166
MFC is under the MIT license (see [LICENSE](LICENSE) file for full text).
167
167
168
168
## Acknowledgements
169
-
170
-
<palign="justify">
171
-
MFC development was supported by multiple current and past grants from the US Department of Defense, National Institute of Health (NIH), Department of Energy (DOE), and the National Science Foundation (NSF).
172
-
MFC computations use OLCF Frontier, Summit, and Wombat under allocation CFD154 (PI Bryngelson) and ACCESS-CI under allocations TG-CTS120005 (PI Colonius) and TG-PHY210084 (PI Bryngelson).
173
-
</p>
169
+
170
+
Multiple federal sponsors have supported MFC development, including the US Department of Defense (DOD), National Institutes of Health (NIH), Department of Energy (DOE), and National Science Foundation (NSF).
171
+
MFC computations use OLCF Frontier, Summit, and Wombat under allocation CFD154 (PI Bryngelson) and ACCESS-CI under allocations TG-CTS120005 (PI Colonius) and TG-PHY210084 (PI Bryngelson).
@@ -88,35 +87,32 @@ You can now follow the appropriate instructions for your distribution.
88
87
</details>
89
88
90
89
<details>
91
-
<summary><h3>MacOS (x86 and Apple Silicon)</h3></summary>
90
+
<summary><h3>MacOS</h3></summary>
92
91
93
-
**Note:** macOS remains the most difficult platform to consistently compile MFC on.
94
-
If you run into issues, we suggest you try using Docker (instructions above).
95
-
96
-
-**MacOS v10.15 (Catalina) or newer [ZSH]** (Verify with `echo $SHELL`)
92
+
-**If you use [ZSH]** (Verify with `echo $SHELL`)
97
93
98
94
```console
99
-
$ touch ~/.zshrc
100
-
$ open ~/.zshrc
95
+
touch ~/.zshrc
96
+
open ~/.zshrc
101
97
```
102
98
103
-
-**Older than MacOS v10.15 (Catalina)[BASH]** (Verify with `echo $SHELL`)
99
+
-**If you use[BASH]** (Verify with `echo $SHELL`)
104
100
105
101
```console
106
-
$ touch ~/.bash_profile
107
-
$ open ~/.bash_profile
102
+
touch ~/.bash_profile
103
+
open ~/.bash_profile
108
104
```
109
105
110
106
An editor should open.
111
107
Please paste the following lines into it before saving the file.
112
-
If you wish to use a version of GNU's GCC other than 11, modify the first assignment.
108
+
If you wish to use a version of GNU's GCC other than 13, modify the first assignment.
113
109
These lines ensure that LLVM's Clang, and Apple's modified version of GCC, won't be used to compile MFC.
114
110
Further reading on `open-mpi` incompatibility with `clang`-based `gcc` on macOS: [here](https://stackoverflow.com/questions/27930481/how-to-build-openmpi-with-homebrew-and-gcc-4-9).
115
-
We do *not* support `clang` due to conflicts with our Silo dependency.
111
+
We do *not* support `clang` due to conflicts with the Silo dependency.
They will download the dependencies MFC requires to build itself. `open-mpi` will be compiled from source, using the version of GCC we specified above with the environment variables `HOMEBREW_CC` and `HOMEBREW_CXX`.
@@ -149,22 +145,22 @@ First install Docker and Git:
Copy file name to clipboardExpand all lines: docs/documentation/running.md
+40-40Lines changed: 40 additions & 40 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -33,7 +33,7 @@ Please refer to `./mfc.sh run -h` for a complete list of arguments and options,
33
33
To run all stages of MFC, that is [pre_process](https://github.com/MFlowCode/MFC/tree/master/src/pre_process/), [simulation](https://github.com/MFlowCode/MFC/tree/master/src/simulation/), and [post_process](https://github.com/MFlowCode/MFC/tree/master/src/post_process/) on the sample case [2D_shockbubble](https://github.com/MFlowCode/MFC/tree/master/examples/2D_shockbubble/),
34
34
35
35
```console
36
-
$ ./mfc.sh run examples/2D_shockbubble/case.py
36
+
./mfc.sh run examples/2D_shockbubble/case.py
37
37
```
38
38
39
39
If you want to run a subset of the available stages, you can use the `-t` argument.
@@ -46,59 +46,59 @@ For example,
46
46
- Running [pre_process](https://github.com/MFlowCode/MFC/tree/master/src/pre_process/) with 2 cores:
47
47
48
48
```console
49
-
$ ./mfc.sh run examples/2D_shockbubble/case.py -t pre_process -n 2
49
+
./mfc.sh run examples/2D_shockbubble/case.py -t pre_process -n 2
50
50
```
51
51
52
52
- Running [simulation](https://github.com/MFlowCode/MFC/tree/master/src/simulation/) and [post_process](https://github.com/MFlowCode/MFC/tree/master/src/post_process/)
53
53
using 4 cores:
54
54
55
55
```console
56
-
$ ./mfc.sh run examples/2D_shockbubble/case.py -t simulation post_process -n 4
56
+
./mfc.sh run examples/2D_shockbubble/case.py -t simulation post_process -n 4
57
57
```
58
58
59
59
## Batch Execution
60
60
61
61
The MFC detects which scheduler your system is using and handles the creation and execution of batch scripts.
62
62
The batch engine is requested via the `-e batch` option.
63
-
The number of nodes can be specified with the `-N` (i.e `--nodes`) option.
63
+
The number of nodes can be specified with the `-N` (i.e.,`--nodes`) option.
64
64
65
65
We provide a list of (baked-in) submission batch scripts in the `toolchain/templates` folder.
-`-@ [email protected]` to receive emails from the scheduler. (i.e `--email`)
75
-
-`-w hh:mm:ss` to specify the job's maximum allowed walltime. (i.e `--walltime`)
76
-
-`-a <account name>` to identify the account to be charged for the job. (i.e `--account`)
77
-
-`-p <partition name>` to select the job's partition. (i.e `--partition`)
73
+
-`-# <job name>` to name your job. (i.e.,`--name`)
74
+
-`-@ [email protected]` to receive emails from the scheduler. (i.e.,`--email`)
75
+
-`-w hh:mm:ss` to specify the job's maximum allowed walltime. (i.e.,`--walltime`)
76
+
-`-a <account name>` to identify the account to be charged for the job. (i.e.,`--account`)
77
+
-`-p <partition name>` to select the job's partition. (i.e.,`--partition`)
78
78
79
79
As an example, one might request GPUs on a SLURM system using the following:
80
80
81
81
**Disclaimer**: IBM's JSRUN on LSF-managed computers does not use the traditional node-based approach to
82
-
allocate resources. Therefore, the MFC constructs equivalent resource-sets in task and GPU count.
82
+
allocate resources. Therefore, the MFC constructs equivalent resourcesets in the task and GPU count.
83
83
84
84
### Profiling with NVIDIA Nsight
85
85
86
-
MFC provides two different argument to facilitate profiling with NVIDIA Nsight.
87
-
**Please ensure that the used argument is placed at the end so that their respective flags can be appended.**
86
+
MFC provides two different arguments to facilitate profiling with NVIDIA Nsight.
87
+
**Please ensure the used argument is placed at the end so their respective flags can be appended.**
88
88
- Nsight Systems (Nsys): `./mfc.sh run ... --nsys [nsys flags]` allows one to visualize MFC's system-wide performance with [NVIDIA Nsight Systems](https://developer.nvidia.com/nsight-systems).
89
-
NSys is best for getting a general understanding of the order and execution times of major subroutines (WENO, Riemann, etc.) in MFC.
89
+
NSys is best for understanding the order and execution times of major subroutines (WENO, Riemann, etc.) in MFC.
90
90
When used, `--nsys` will run the simulation and generate `.nsys-rep` files in the case directory for all targets.
91
-
These files can then be imported into Nsight System's GUI, which can be downloaded [here](https://developer.nvidia.com/nsight-systems/get-started#latest-Platforms). It is best to run case files with a few timesteps so that the report files remain small. Learn more about NVIDIA Nsight Systems [here](https://docs.nvidia.com/nsight-systems/UserGuide/index.html).
91
+
These files can then be imported into Nsight System's GUI, which can be downloaded [here](https://developer.nvidia.com/nsight-systems/get-started#latest-Platforms). It is best to run case files with a few timesteps to keep the report files small. Learn more about NVIDIA Nsight Systems [here](https://docs.nvidia.com/nsight-systems/UserGuide/index.html).
92
92
- Nsight Compute (NCU): `./mfc.sh run ... --ncu [ncu flags]` allows one to conduct kernel-level profiling with [NVIDIA Nsight Compute](https://developer.nvidia.com/nsight-compute).
93
93
NCU provides profiling information for every subroutine called and is more detailed than NSys.
94
94
When used, `--ncu` will output profiling information for all subroutines, including elapsed clock cycles, memory used, and more after the simulation is run.
95
-
Please note that adding this argument will significantly slow down the simulation and should only be used on case files with a few timesteps.
95
+
Adding this argument will significantly slow the simulation and should only be used on case files with a few timesteps.
96
96
Learn more about NVIDIA Nsight Compute [here](https://docs.nvidia.com/nsight-compute/NsightCompute/index.html).
97
97
98
98
### Restarting Cases
99
99
100
100
When running a simulation, MFC generates a `./restart_data` folder in the case directory that contains `lustre_*.dat` files that can be used to restart a simulation from saved timesteps.
101
-
This allows a user to run a simulation to some timestep $X$, then later continue it to run to another timestep $Y$, where $Y > X$.
101
+
This allows a user to simulate some timestep $X$, then continue it to run to another timestep $Y$, where $Y > X$.
102
102
The user can also choose to add new patches at the intermediate timestep.
103
103
104
104
If you want to restart a simulation,
@@ -108,29 +108,29 @@ If you want to restart a simulation,
108
108
-`t_step_stop` : $t_f$
109
109
-`t_step_save` : $SF$
110
110
in which $t_i$ is the starting time, $t_f$ is the final time, and $SF$ is the saving frequency time.
111
-
- Run pre-process and simulation on the case.
111
+
- Run `pre_process` and `simulation` on the case.
112
112
-`./mfc.sh run case.py -t pre_process simulation `
113
-
- As the simulation runs, it will create LUSTRE files for each saved timestep in `./restart_data`.
114
-
- When the simulation stops, choose any LUSTRE file as the restarting point (lustre_ $t_s$.dat)
115
-
- Create a new duplicate input file, (ex.`restart_case.py`), on which it should:
113
+
- As the simulation runs, it will create Lustre files for each saved timestep in `./restart_data`.
114
+
- When the simulation stops, choose any Lustre file as the restarting point (lustre_ $t_s$.dat)
115
+
- Create a new duplicate input file (e.g.,`restart_case.py`), which should have:
116
116
117
117
1. For the Computational Domain Parameters
118
-
- Have the following removed BUT`m`, `n`, and `p`:
119
-
- All domaing/mesh information
118
+
- Have the following removed __except__`m`, `n`, and `p`:
119
+
- All domain/mesh information
120
120
- `(xyz)_domain%beg`
121
121
- `(xyz)_domain%end`
122
122
- `stretch_(xyz)`
123
123
- `a_(xyz)`
124
124
- `(xyz)_a`
125
125
- `(xyz)_b`
126
-
- Have the following altered:
127
-
- `t_step_start` : $t_s$ # this is the point at which the simulation will restart
128
-
- `t_step_stop` : $t_{f2}$ # your new final simulation time, which can be the same as $t_f$
129
-
- `t_step_save` : ${SF}_2$ # if interested in changing the saving frequency
130
-
- Have the following ADDED:
131
-
- `old_ic` : 'T' # to specify that we have initial conditions from previous simulations
132
-
- `old_grid` : 'T' # to specify that we have a grid from previous simulations (maybe I do not need m, n, and p, then?)
133
-
- `t_step_old` : $t_i$ # this is the time step used as the `t_step_start` of the original `case.py` file
126
+
- Alter the following:
127
+
- `t_step_start` : $t_s$ (the point at which the simulation will restart)
128
+
- `t_step_stop` : $t_{f2}$ (new final simulation time, which can be the same as $t_f$)
129
+
- `t_step_save` : ${SF}_2$ (if interested in changing the saving frequency)
130
+
- Add the following:
131
+
- `old_ic` : 'T' (to specify that we have initial conditions from previous simulations)
132
+
- `old_grid` : 'T' (to specify that we have a grid from previous simulations)
133
+
- `t_step_old` : $t_i$ (the time step used as the `t_step_start` of the original `case.py` file)
134
134
2. For the Simulation Algorithm Parameters
135
135
- Substitute `num_patches` to reflect the number of ADDED patches in the `restart_case.py` file. If no patches are added, set `num_patches: 0`
136
136
@@ -145,32 +145,32 @@ in which $t_i$ is the starting time, $t_f$ is the final time, and $SF$ is the sa
145
145
4. For Fluid properties
146
146
- Keep information about the fluid properties
147
147
148
-
- Run pre-process and simulation on restart_case.py
148
+
- Run pre-process and simulation on `restart_case.py`
149
149
-`./mfc.sh run restart_case.py -t pre_process simulation `
150
150
151
151
- Run the post_process
152
152
- There are several ways to do this. Keep in mind that, regardless of the .py file used, the post_process command will generate output files in the [`t_step_start`, `t_step_stop`] range, with `t_step_save` as the spacing between files.
153
153
- One way is to set `t_step_stop` to the restarting point $t_s$ in `case.py`. Then, run the commands below. The first command will run on timesteps $[t_i, t_s]$. The second command will run on $[t_s, t_{f2}]$. Therefore, the whole range $[t_i, t_{f2}]$ will be post processed.
154
154
155
155
```console
156
-
$ ./mfc.sh run case.py -t post_process
157
-
$ ./mfc.sh run restart_case.py -t post_process
156
+
./mfc.sh run case.py -t post_process
157
+
./mfc.sh run restart_case.py -t post_process
158
158
```
159
159
160
-
We have provided an example `case.py` and `restart_case.py` in `/examples/1D_vacuum_restart/`. This simulation is a duplicate of the `1D_vacuum` case. It demonstrates stopping at timestep 7000, adding a new patch, and restarting the simulation. To test this code, run:
160
+
We have provided an example,`case.py` and `restart_case.py` in `/examples/1D_vacuum_restart/`. This simulation is a duplicate of the `1D_vacuum` case. It demonstrates stopping at timestep 7000, adding a new patch, and restarting the simulation. To test this code, run:
161
161
162
162
```console
163
-
$ ./mfc.sh run examples/1D_vacuum_restart/case.py -t pre_process simulation
164
-
$ ./mfc.sh run examples/1D_vacuum_restart/restart_case.py -t pre_process simulation
165
-
$ ./mfc.sh run examples/1D_vacuum_restart/case.py -t post_process
166
-
$ ./mfc.sh run examples/1D_vacuum_restart/restart_case.py -t post_process
163
+
./mfc.sh run examples/1D_vacuum_restart/case.py -t pre_process simulation
164
+
./mfc.sh run examples/1D_vacuum_restart/restart_case.py -t pre_process simulation
165
+
./mfc.sh run examples/1D_vacuum_restart/case.py -t post_process
166
+
./mfc.sh run examples/1D_vacuum_restart/restart_case.py -t post_process
167
167
```
168
168
169
169
### Example Runs
170
170
171
171
- Oak Ridge National Laboratory's [Summit](https://www.olcf.ornl.gov/summit/):
172
172
173
173
```console
174
-
$ ./mfc.sh run examples/2D_shockbubble/case.py -e batch \
174
+
./mfc.sh run examples/2D_shockbubble/case.py -e batch \
0 commit comments