Non GPU Programs on the GPU?/No GPU Compute Usage with ROCm

[TheFloatingBrain]

Background information

I am trying to run scripts using meep and accelerate them using a GPU without modifying meep. I asked another meep, gpu, ompi related question, but am wondering if my assumptions about how ompi can be used are wrong

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

v5.0.5

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

I built it from source and installed it (and ucx) to a local directory
ompi config:

../configure --prefix=$HOME/.ompi_for_gpu/ompi --with-ucx=$HOME/.ompi_for_gpu/ucx --with-rocm --enable-rocm --without-cuda

Also compiled meep with OpenMP support

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

No output

Please describe the system on which you are running

• Operating system/version: Fedora 40
• Computer hardware: Laptop with AMD Radeon integrated card
• Network type: WiFi

---

When I run the program, I dont see GPU usage significantly increase due to the program (though I see GPU memory being utilized). So i had two questions:
Can I use OpenMPI to run non GPU (programs not written with any GPU code) programs on a GPU?
if so, do I need to modify this command?

mpirun --mca pml ucx -n 8 python ./bent_waveguide.py -d rocm D D

[devreal]

Without more details it is hard to provide an in-depth answer as it's not clear what you are trying to achieve. If you expect to magically run the entire application on the device, here are some datapoints to consider:

• You'd have to compile all of your application for hip/rocm and submit that as a kernel. There has been some work on completely offloading applications to accelerators but it's by far not the norm [1].
• MPI (and by extension Open MPI) does not provide any device bindings so it is not possible today to call MPI functions from the device. Nvshmem is probably the only widely used communication API that enables this today. There are some subtle issues in MPI that make enabling MPI on devices more challenging than a one-sided model like nvshmem (message matching and synchronization among others).

It is the application's task to offload compute kernels to the device. The code controlling that still executes on the host, where MPI will run. I'm not sure Open MPI can help with this.

[1] https://arxiv.org/abs/2306.11686

[edgargabriel]

I am a bit confused also by some of the command lines that you posted. Is the configure line supposed to configure Open MPI? If yes, please note that there is no --enable-rocm flag in Open MPI, just --with-rocm=/opt/rocm (passing the location of the ROCm installation is important).

And the second line, your mpirun command line, it has arguments -d rocm D D which looks a lot like the syntax of the OSU benchmarks. Does the code that you want to run support the same syntax as the OSU benchmarks? Most codes don't have the same syntax, that is OSU benchmark specific.

[TheFloatingBrain]

@devreal Thank you for the answer

It is the application's task to offload compute kernels to the device. The code controlling that still executes on the host, where MPI will run. I'm not sure Open MPI can help with this.

This is more or less what I think I mean, take sections of the application that use MPI to chunk and parallelize and run them on a GPU.

[TheFloatingBrain]

@edgargabriel Thanks for the reply!

I am a bit confused also by some of the command lines that you posted. Is the configure line supposed to configure Open MPI? If yes, please note that there is no --enable-rocm flag in Open MPI

I was initially trying to compile a 4.x (which I think I got from a link on older documentation) version of OpenMPI and saw a --enable-cuda flag or something like that in an online example, it wasn't recognizing --with-rocm (which worked UCX, though I had trouble compiling UCC but it doesent seem nessisary), this was resolved when I cloned v5.0.5. I just threw that for good measure because I had observed that flags that dont exist are skipped, so I figured it couldn't hurt and I didn't want to run it again if it was suppose to be there, a bit lazy

just --with-rocm=/opt/rocm (passing the location of the ROCm installation is important).

👍 Good to know, in my case I do want to note, for some reason there is no /opt/rocm but everything seems to be in the corresponding folders under /usr, not sure why.

And the second line, your mpirun command line, it has arguments -d rocm D D which looks a lot like the syntax of the OSU benchmarks. Does the code that you want to run support the same syntax as the OSU benchmarks? Most codes don't have the same syntax, that is OSU benchmark specific.

It doesn't, I did not realize that was specific to the OSU benchmarks, I before tried with just --mca pml ucx but got the same result as with -d rocm D D, just unsure of the syntax to tell it to offload to the GPU in this case

[devreal]

It is the application's task to offload compute kernels to the device. The code controlling that still executes on the host, where MPI will run. I'm not sure Open MPI can help with this.

This is more or less what I think I mean, take sections of the application that use MPI to chunk and parallelize and run them on a GPU.

MPI will not help with offloading computation to the device. MPI is used to move data in between offloaded kernels. The offloading to the device has to come from the application itself.

[TheFloatingBrain]

It is the application's task to offload compute kernels to the device. The code controlling that still executes on the host, where MPI will run. I'm not sure Open MPI can help with this.

This is more or less what I think I mean, take sections of the application that use MPI to chunk and parallelize and run them on a GPU.

MPI will not help with offloading computation to the device. MPI is used to move data in between offloaded kernels. The offloading to the device has to come from the application itself.

Okay, that is what I have been missing. Thank you

[TheFloatingBrain]

Can I ask what purpose CUDA and ROCm awareness serve?

Push the memory onto the GPU's shared memory?

[edgargabriel]

no, CUDA and ROCm awareness of the MPI library allow the user to pass a device buffer as an argument to MPI_Send or MPI_Recv

[TheFloatingBrain]

no, CUDA and ROCm awareness of the MPI library allow the user to pass a device buffer as an argument to MPI_Send or MPI_Recv

So instead of

Copy from GPU 0 (using API like OpenCL/CUDA) -> MPI_Send (host0) -> MPI_Recv (host1) -> Copy to GPU 1 (using API like OpenCL/CUDA)

It can do

MPI_Send (GPU 0) -> MPI_Recv (GPU 1)

Do I have that about right?

Also I think I was mislead on this point by the mention of another "OpenMP" implementation that can do the offloading, combine that with a message from conda that reads

On Linux, Open MPI is built with CUDA awareness but it is disabled by default.
To enable it, please set the environment variable
OMPI_MCA_opal_cuda_support=true
before launching your MPI processes.
Equivalently, you can set the MCA parameter in the command line:
mpiexec --mca opal_cuda_support 1 ...
Note that you might also need to set UCX_MEMTYPE_CACHE=n for CUDA awareness via
UCX. Please consult UCX documentation for further details.

I think is how I got here

[edgargabriel]

no, CUDA and ROCm awareness of the MPI library allow the user to pass a device buffer as an argument to MPI_Send or MPI_Recv

So instead of

Copy from GPU 0 (using API like OpenCL/CUDA) -> MPI_Send (host0) -> MPI_Recv (host1) -> Copy to GPU 1 (using API like OpenCL/CUDA)

It can do

MPI_Send (GPU 0) -> MPI_Recv (GPU 1)

Do I have that about right?

yes.

Also I think I was mislead on this point by the mention of another "OpenMP" implementation that can do the offloading, combine that with a message from conda that reads

On Linux, Open MPI is built with CUDA awareness but it is disabled by default.
To enable it, please set the environment variable
OMPI_MCA_opal_cuda_support=true
before launching your MPI processes.
Equivalently, you can set the MCA parameter in the command line:
mpiexec --mca opal_cuda_support 1 ...
Note that you might also need to set UCX_MEMTYPE_CACHE=n for CUDA awareness via
UCX. Please consult UCX documentation for further details.

I think is how I got here

That was the case in Open MPI 4.1.x and previously the case, I think that changed in 5.0 and its not required anymore. I know for a fact that its not required for ROCm support, but somebody from Nvidia should probably confirm whether its required for CUDA support.

[TheFloatingBrain]

By required, you mean specify the parameters/env variables?