Efficiently handling boundary and interior regions separately in AMReX

[Afraid31]

Hello,

I am currently working with AMReX and need to perform different operations on the boundary layers (ghost cells) and the main computational region. Right now, I am iterating over both the boundary and main regions together, then overwriting the interior data. This approach works, but I am wondering if there's a more efficient way to handle this where I can separate the operations for the boundary and interior regions more cleanly.

Here's the code I’m currently using:
`for (MFIter mfi(S_new, TilingIfNotGPU()); mfi.isValid(); ++mfi)
{
const Box& bx = mfi.tilebox();
const Box& gbx = amrex::grow(bx, num_grow);

Array4<Real const> statein  = Sborder.const_array(mfi);
Array4<Real const> statem = Macro.const_array(mfi);
Array4<Real      > stateout = S_new.array(mfi);

amrex::ParallelFor(gbx,
    [=] AMREX_GPU_DEVICE(int i, int j, int k)
{
    for (int k = 0; k < 9; k++) {
        stateout(i, j, k) = statein(i, j, k);
    }
});

amrex::ParallelFor(bx,
[=] AMREX_GPU_DEVICE (int i, int j, int k)
{
    for (int k = 0; k < 9; k++) {
        Real rho = statem(i, j, 0);
        Real u = statem(i, j, 1);
        Real v = statem(i, j, 2);

        stateout(i, j, k) = (1 - 1.0 / tau[lev]) * statein(i, j, k) +
            1.0 / tau[lev] * get_feq(rho, u, v, k);
    }
});

}
In this code, I first apply an operation to both the boundary and main regions usingamrex::ParallelFor(gbx, ...) and then perform a second operation on the main region only with amrex::ParallelFor(bx, ...)`. However, I want to handle the boundary and interior regions separately to avoid unnecessary overwriting of interior data.

Is there a way to iterate only over the boundary layers for the first operation and only the interior cells for the second operation, without having to perform redundant operations over the entire region? This would help me avoid issues where the interior data is being overwritten unnecessarily.

Any suggestions or best practices would be greatly appreciated!

Thanks in advance for your help!

[WeiqunZhang]

amrex::ParallelFor(gbx,
    [=] AMREX_GPU_DEVICE(int i, int j, int k)
{
    for (int k = 0; k < 9; k++) {
        stateout(i, j, k) = statein(i, j, k);
    }
});

There is variable shadowing in your code. It can be very confusing. So I think you should fix that. It also appears to me that you are assuming the code is always 2d. Even in that case, I would still access the data with (i,j,k,n), not (i,j,n).

That being said, I suggest you do something like

amrex::ParallelFor(gbx, 9, [=] AMREX_GPU_DEVICE (int i, int j, int k, int n)
{
    if (bx.contains(i,j,k)) {
        // work on interior
        stateout(i, j, k,n) = statein(i, j, k,n);
    } else {
        // work on ghost cells
        Real rho = statem(i, j, k, 0);
        Real u = statem(i, j, k, 1);
        Real v = statem(i, j, k, 2);
        stateout(i, j, k, n) = (1 - 1.0 / tau[lev]) * statein(i, j, k, n) +
            1.0 / tau[lev] * get_feq(rho, u, v, n);
    }
});

[Afraid31]

Thanks for the helpful feedback!

[drummerdoc]

If you still think it's better to do the boundary regions separately, you can use completmentIn to get a BoxArray of the grow region and iterate over those boxes.