How to get min/max particle positions across all levels with GPU parallelization?

[Afraid31]

Hi all,

I would like to compute the bounding box of all particles (minimum and maximum coordinates in each dimension) across all levels of my AMReX simulation.

Currently, I have the following code:

// 1. Find particle bounding box
amrex::GpuArray<amrex::Real, BL_SPACEDIM> pbox_lo, pbox_hi;
for (int d = 0; d < BL_SPACEDIM; ++d) {
    pbox_lo[d] = std::numeric_limits<amrex::Real>::max();
    pbox_hi[d] = -std::numeric_limits<amrex::Real>::max();
}

for (int lev = 0; lev <= parent->finestLevel(); ++lev) {
    for (MyParIter pti(*CylinderPoints, lev); pti.isValid(); ++pti) {
        const auto& arr = pti.GetArrayOfStructs();
        for (int n = 0; n < arr.size(); ++n) {
            for (int d = 0; d < BL_SPACEDIM; ++d) {
                pbox_lo[d] = amrex::min(pbox_lo[d], arr[n].pos(d));
                pbox_hi[d] = amrex::max(pbox_hi[d], arr[n].pos(d));
            }
        }
    }
}

This works fine on CPU (with OpenMP parallelization), but it is not GPU-friendly because of the explicit for-loops over particles.

My question is:
How can I modify this to efficiently compute the global min/max particle positions using GPU parallelization (CUDA backend), while still working on CPU if GPUs are not used?

Thanks in advance!

[WeiqunZhang]

You can use the ParticleReduce function.

Here is a link to doxygen documentation (https://amrex-codes.github.io/amrex/doxygen/namespaceamrex.html#a3762aae68593d892d48ede3cb6e317a7). The example code there is not formatted correctly for doxygen. We need to fix it. For now you might want to look at comments in the source code.

amrex/Src/Particle/AMReX_ParticleReduce.H

Line 1207 in 5d1b6c2

* Example usage:

Note that you still need to do MPI reduction yourself. (See the documentation for detail.)