Are there prospects for allowing a different memory layout in the data storage?

[tpg2114]

Based on a long time optimizing our existing code, we know it performs fastest when the memory is laid out as (nvar, icells, jcells, kcells) because most of the expensive algorithms are doing calculations involving multiple variables at a single cell. We tested/profiled it the other way, (icells, jcells, kcells, nvar) and it was much worse. Re-writing it so the operations are optimized for (icells, jcells, kcells, nvar) is a non-starter -- the code is nearly 500k lines and it works well as is.

We have bolted on AMReX to handle AMR operations and communication and retained our existing data structures so our core code doesn't change and we can keep all of our existing validated capabilities with and without AMReX. However, because the AMReX memory order is (icells, jcells, kcells, nvar), it means we have to allocate duplicate memory and copy the values back and forth.

If AMReX had the ability to put the components first or last in the layout, then it would eliminate virtually all the overhead of bolting on AMReX -- we could provide pointers to existing arrays instead of allocating duplicate arrays, and then we wouldn't need to do any copying back and forth either.

I don't know what the possibility of adding that option would be, however. I would bet it's quite an undertaking and makes the AMR-specific tasks more expensive since they tend to operate on one variable over multiple cells at a time. But based on my profiling studies of the current approach in our code, very little time is spent in the AMR math parts and almost the entire cost of bolting on AMReX is spent on copying between the arrays.

Is there any prospect of allowing either ordering? It's okay if the answer is no or if nobody other than me would be interested in it, but I would like to throw it out there that it would certainly improve our performance if it could be done.

[WeiqunZhang]

We have limited support for the (nvar, icells, jcells, kcells) support. For MultiFab, it's always (icells, jcells, kcells, nvars). However, if nvar is a fixed size known at compile time, we do have FabArray<BaseFab<GpuArray<Real,nvar>>>. Note that MultiFab is derived from FabArray.

[tpg2114]

That's definitely helpful to know, but unfortunately it is not a compile-time constant and there isn't a common-enough use case where I could compile multiple options into a shared library and load the one needed at runtime. Thanks for the insight though, I'll think on it for a bit and see if there's anything I could do to take advantage of what is there.