Skip to content

Commit e316d3a

Browse files
bruno-garbacciobruno-garbaccio
authored
Merge pull request #1741 from oracle-devrel/opm-flow-gpu
Opm flow gpu
2 parents d599c52 + 6b2056a commit e316d3a

File tree

6 files changed

+270
-0
lines changed

6 files changed

+270
-0
lines changed

‎cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/README.md‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
# Building and Running OPM Flow on OCI GPU infrastructure
2+
3+
## The OPM initiative
4+
5+
The [Open Porous Media (OPM)](https://opm-project.org/) initiative is an innovative open source project for modeling and simulating porous media processes.
6+
7+
![OPM Flow official illustration](assets/images/OPM-Flow.png "Norne model")
8+
9+
OPM Flow is a fully-implicit, black-oil simulator capable of running industry-standard simulation models. The simulator is implemented using automatic differentiation to enable rapid development of new fluid models.
10+
11+
## Configuring a GPU instance on OCI
12+
13+
Oracle Cloud Infrastructure offers GPU shapes for all kinds of workloads, including:
14+
- NVIDIA A10 24GB (available on-demand)
15+
- VM with 1 or 2 GPUs
16+
- BM with 4 GPUs
17+
- NVIDIA A100 40GB (available on-demand)
18+
- VM with 1 GPU
19+
- BM with 8 GPUs, NVLink and RDMA cluster networking
20+
- NVIDIA L40S 48GB
21+
- BM with 4 GPUs and RDMA cluster networking
22+
- NVIDIA A100 80GB
23+
- VM with 1 GPU
24+
- BM with 8 GPUs, NVLink and RDMA cluster networking
25+
- NVIDIA H100 80GB
26+
- BM with 8 GPUs, NVLink and RDMA cluster networking
27+
- AMD MI300X 192GB
28+
- BM with 8 GPUs, HIP and RDMA cluster networking
29+
30+
This tutorial covers both NVIDIA and AMD GPU shapes. More specifically, we use here:
31+
- A 1 x A100 80 GB virtual machine (VM.GPU.A100.80GB.1)
32+
- A 8 x MI300X 192 GB bare metal (BM.GPU.MI300X.8)
33+
34+
Both shapes are configured with a Cannonical Ubuntu 22.04 image with respectively CUDA and ROCm preinstalled.
35+
36+
On the NVIDIA GPU shapes, it is necessary to install the NVIDIA CUDA Compiler (NVCC) to build Flow:
37+
```
38+
sudo apt-get install -y nvidia-cuda-toolkit
39+
```
40+
41+
## Installing OPM Flow on OCI GPU infrastructure
42+
43+
The project can be installed from packages or built from sources. For GPU support, building from sources is mandatory.
44+
45+
### Installing binaries
46+
47+
When GPU are not required or not available, OPM Flow can run on CPU only. In this case, installing the toolset from packages is the easiest way (see tutorial [here](https://opm-project.org/?page_id=245)).
48+
49+
### Building from sources
50+
51+
Running OPM Flow on GPU requires to build the toolset from sources in order to setup the GPU support. Common Building from Source guidelines are available [here](https://opm-project.org/?page_id=231). In the sequel, one can get more hands-on commands and advices for a complete configuration.
52+
53+
#### Installing prerequisites
54+
55+
A collection of libraries, framework and tools are necessary to build Flow. A complete description is available [here](https://opm-project.org/?page_id=239). They can be installed using the `install_prerequisites.sh` script.
56+
```
57+
sudo ./assets/scripts/install_prerequisites.sh
58+
```
59+
60+
#### Building the modules
61+
62+
4 modules must be built in a specific order:
63+
1. `opm-common`
64+
2. `opm-grid`
65+
3. `opm-simulators`
66+
4. `opm-upscaling`
67+
68+
For `opm-common`and `opm-grid`, no specific actions is required. Simply follow the build instructions:
69+
```
70+
git clone https://github.com/OPM/[modulename].git
71+
mkdir [modulename]/build && cd [modulename]/build
72+
cmake ..
73+
make
74+
```
75+
However, for `opm-simulators` it is necessary to turn off the `USE_GPU_BRIDGE` option at the `cmake` stage:
76+
```
77+
git clone https://github.com/OPM/opm-simulators.git
78+
mkdir opm-simulators/build && cd opm-simulators/build
79+
cmake .. -DUSE_GPU_BRIDGE=OFF # add the -DCONVERT_CUDA_TO_HIP=ON option for AMD GPUs
80+
make
81+
```
82+
The following error might occur at the `make` stage:
83+
```
84+
/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
85+
435 | function(_Functor&& __f)
86+
| ^
87+
/usr/include/c++/11/bits/std_function.h:435:145: note: ‘_ArgTypes’
88+
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
89+
530 | operator=(_Functor&& __f)
90+
| ^
91+
/usr/include/c++/11/bits/std_function.h:530:146: note: ‘_ArgTypes’
92+
make[2]: *** [CMakeFiles/opmsimulators.dir/build.make:2680: CMakeFiles/opmsimulators.dir/opm/simulators/linalg/gpuistl/detail/vector_operations.cu.o] Error 1
93+
make[2]: *** Waiting for unfinished jobs....
94+
make[1]: *** [CMakeFiles/Makefile2:530: CMakeFiles/opmsimulators.dir/all] Error 2
95+
make: *** [Makefile:146: all] Error 2
96+
```
97+
If it does, two lines (beginning by the `noexcept` keyword) must be commented in the `std_function.h` file, i.e.:
98+
- approximately line 430:
99+
```
100+
template<typename _Functor,
101+
typename _Constraints = _Requires<_Callable<_Functor>>>
102+
function(_Functor&& __f)
103+
//noexcept(_Handler<_Functor>::template _S_nothrow_init<_Functor>()) // CUDA does not support this line
104+
: _Function_base()
105+
106+
```
107+
- approximately line 450:
108+
```
109+
template<typename _Functor,
110+
typename _Constraints = _Requires<_Callable<_Functor>>>
111+
function(_Functor&& __f)
112+
//noexcept(_Handler<_Functor>::template _S_nothrow_init<_Functor>()) // CUDA does not support this line
113+
: _Function_base()
114+
115+
```
116+
Finally, build `opm-upscaling`:
117+
```
118+
git clone https://github.com/OPM/opm-upscaling.git
119+
mkdir opm-upscaling/build && cd opm-upscaling/build
120+
cmake ..
121+
make
122+
```
123+
124+
## Running OPM Flow on GPU
125+
126+
Running the Flow simulator on one or more GPUs requires to use GPU-specific solver and preconditionners. They can be defined in a linear solver [file](/assets/scripts/gpu-solver.json) in the JSON format, for example:
127+
```
128+
{
129+
"tol": "0.01",
130+
"maxiter": "200",
131+
"verbosity": "0",
132+
"solver": "gpubicgstab",
133+
"preconditioner": {
134+
"type": "GPUDILU",
135+
"verbosity" : 0,
136+
"split_matrix": "true",
137+
"tune_gpu_kernels": "true",
138+
"mixed_precision_scheme": 1
139+
}
140+
}
141+
```
142+
To run the simulation, open datasets are available [here](https://github.com/OPM/opm-data.git). For example, to run the Norne use case, simply execute:
143+
```
144+
/home/ubuntu/opm-simulators/build/bin/flow NORNE_ATW2013.DATA --output-dir=out_gpu --matrix-add-well-contributions=true --linear-solver=/home/ubuntu/gpu-solver.json
145+
```
146+
for 1 GPU and:
147+
```
148+
mpirun -np N /home/ubuntu/opm-simulators/build/bin/flow NORNE_ATW2013.DATA --output-dir=out_gpu --matrix-add-well-contributions=true --threads-per-process=1 --linear-solver=/home/ubuntu/gpu-solver.json
149+
```
150+
for N GPUs.
151+
152+
## Notes:
153+
154+
Here are a few comments to take into account when considering running Flow on GPUs:
155+
* The options `--matrix-add-well-contributions=true` and `--threads-per-process=1` are recommended by OPM.
156+
* Because of the import CPU/GPU traffic, running Flow on GPU is only relevant for models above a certain size (few hundreds of thousands of cells).
157+
* Using a CUDA-aware or ROCm-aware version of Open MPI may improve the overall performance of the simulation.
158+
159+
## External Links
160+
161+
* [The Open Porous Media Initiative](https://opm-project.org/)
162+
* [The Best Public Cloud for Oil and Gas Reservoir Simulation](https://blogs.oracle.com/cloud-infrastructure/post/the-best-public-cloud-for-oil-and-gas-reservoir-simulation)
163+
* [Building CUDA-aware Open MPI](https://www.open-mpi.org/faq/?category=buildcuda)
164+
165+
## License
166+
167+
Copyright (c) 2025 Oracle and/or its affiliates.
168+
169+
Licensed under the Universal Permissive License (UPL), Version 1.0.
170+
171+
See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/main/LICENSE) for more details.

‎cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/images/OPM-Flow.png‎

174 KB
Loading

‎cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/scripts/LICENSE‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
Copyright (c) 2024 Oracle and/or its affiliates.
2+
3+
The Universal Permissive License (UPL), Version 1.0
4+
5+
Subject to the condition set forth below, permission is hereby granted to any
6+
person obtaining a copy of this software, associated documentation and/or data
7+
(collectively the "Software"), free of charge and under any and all copyright
8+
rights in the Software, and any and all patent rights owned or freely
9+
licensable by each licensor hereunder covering either (i) the unmodified
10+
Software as contributed to or provided by such licensor, or (ii) the Larger
11+
Works (as defined below), to deal in both
12+
13+
(a) the Software, and
14+
(b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
15+
one is included with the Software (each a "Larger Work" to which the Software
16+
is contributed by such licensors),
17+
18+
without restriction, including without limitation the rights to copy, create
19+
derivative works of, display, perform, and distribute the Software and make,
20+
use, sell, offer for sale, import, export, have made, and have sold the
21+
Software and the Larger Work(s), and to sublicense the foregoing rights on
22+
either these or other terms.
23+
24+
This license is subject to the following condition:
25+
The above copyright notice and either this complete permission notice or at
26+
a minimum a reference to the UPL must be included in all copies or
27+
substantial portions of the Software.
28+
29+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
30+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
31+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
32+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
33+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
34+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
35+
SOFTWARE.

‎cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/scripts/build_flow_gpu.sh‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/bin/bash
2+
3+
for repo in opm-common opm-grid
4+
do
5+
git clone https://github.com/OPM/$repo.git
6+
mkdir $repo/build && cd $repo/build
7+
cmake ..
8+
make -j 8
9+
cd ../..
10+
done
11+
12+
git clone https://github.com/OPM/opm-simulators.git
13+
mkdir opm-simulators/build && cd opm-simulators/build
14+
cmake .. -DUSE_GPU_BRIDGE=OFF # add the -DCONVERT_CUDA_TO_HIP=ON option for AMD GPUs
15+
make -j 8
16+
cd ../..
17+
18+
git clone https://github.com/OPM/opm-upscaling.git
19+
mkdir opm-upscaling/build && cd opm-upscaling/build
20+
cmake ..
21+
make -j 8
22+
cd ../..

‎cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/scripts/gpu-solver.json‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{
2+
"tol": "0.01",
3+
"maxiter": "200",
4+
"verbosity": "0",
5+
"solver": "gpubicgstab",
6+
"preconditioner": {
7+
"type": "GPUDILU",
8+
"verbosity" : 0,
9+
"split_matrix": "true",
10+
"tune_gpu_kernels": "true",
11+
"mixed_precision_scheme": 1
12+
}
13+
}

‎cloud-infrastructure/ai-infra-gpu/ai-infrastructure/opm-flow-gpu/assets/scripts/install_prerequisites.sh‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
#!/bin/bash
2+
3+
# Make sure we have updated URLs to packages etc.
4+
sudo apt-get update -y
5+
6+
# For server edition of Ubuntu add-apt-repository depends on
7+
sudo apt-get install -y software-properties-common
8+
9+
# Add PPA for OPM packages
10+
sudo add-apt-repository -y ppa:opm/ppa
11+
sudo apt-get update -y
12+
13+
# Packages necessary for building
14+
sudo apt-get install -y build-essential gfortran pkg-config cmake
15+
16+
# Packages necessary for documentation
17+
sudo apt-get install -y doxygen ghostscript texlive-latex-recommended gnuplot
18+
19+
# Packages necessary for version control
20+
sudo apt-get install -y git-core
21+
22+
# MPI for parallel programs
23+
sudo apt-get install -y mpi-default-dev
24+
25+
# Prerequisite libraries
26+
sudo apt-get install -y libblas-dev libboost-all-dev libsuitesparse-dev libtrilinos-zoltan-dev libfmt-dev libcjson-dev
27+
28+
# Parts of Dune needed
29+
sudo apt-get install -y libdune-common-dev libdune-geometry-dev libdune-istl-dev libdune-grid-dev

0 commit comments

Comments
 (0)