Specifying manual subdomains with halos #45
Description
Currently, ImplicitGlobalGrid automatically takes care of the grid "chunking" by allocating appropriate arrays on each process and ensuring the arrays have a suitable halo around them.
In some instances, it may be useful to force a part of the domain to be split into its own chunk and proper halo -- even if it's not the only chunk on the process. This is particularly relevant when building heterogenous stencil codes, such that part of the domain may have more complicated materials etc (and thus a more complicated stencil). By ensuring these chunks have halos, one could update these chunks, along with all other chunks asynchronously on a gpu. Then we perform a global halo update and move to the next set of kernels.
For example, suppose a wave-physics problem has perfectly-matched layers (PML). These layers only exist on the edges of the computational domain (but can be several pixels thick in each dimension) and require several auxiliary variables (i.e. arrays) in addition to those of the normal PDE. Naively, one might just allocate these extra arrays for the entire domain, and use a single stencil for everything. However, with memory-bandwidth-bound codes, this is a bad idea! It would be much better to have one kernel for regions with auxiliary fields, and one without!
These kernels can be run in tandem (and asynchronously) so long as their halos were updated before the next iteration, just like a set of arrays living within two different processes.
I realize this removes some of the "implicit" within IGG... but I also think it extends its capabilities to efficient large-scale codes with highly practical applications.