openPMD: Support for mesh refinement

[guj]

Implements the openPMD mesh-refinement extension: openPMD/openPMD-standard#252

• Field naming structure (ADIOS2):

 double  /data/1000/fields/j/x              {2048, 2048, 2048}
 double  /data/1000/fields/j/y              {2048, 2048, 2048}
 double  /data/1000/fields/j/z              {2048, 2048, 2048}
 double  /data/1000/fields/j_lvl1/x         {4096, 4096, 4096}
 double  /data/1000/fields/j_lvl1/y         {4096, 4096, 4096}
 double  /data/1000/fields/j_lvl1/z         {4096, 4096, 4096}
 double  /data/1000/fields/rho              {2048, 2048, 2048}
 double  /data/1000/fields/rho_lvl1         {4096, 4096, 4096}

(future: _<P> patch number for HDF5 as a fallback)

• particles live on the finest level of each patch but get by default initialized on the coarse mesh

[ax3l]

After a little cleanup commit, I did more tests today (ADIOS2, HDF5) and the particles and fields look great, as well as particles transitioning between patches.

I am double-checking with @RevathiJambunathan that the back-transformed diagnostics is still working and after that I think it can go in :)

[RevathiJambunathan]

Thanks for this PR!

I tested this for Back Transformed diagnostics and get an error:

STEP 64 starts ...
STEP 64 ends. TIME = 6.814271277e-13 DT = 1.064729887e-14
Walltime = 19.13449687 s; This step = 0.184028665 s; Avg. per step = 0.2989765135 s
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Series] Detected illegal access to iteration that has been closed previously.
SIGABRT
/usr/bin/addr2line: DWARF error: could not find variable specification at offset 240e1
/usr/bin/addr2line: DWARF error: could not find variable specification at offset 240e1
See Backtrace.0.0 file for details
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI COMMUNICATOR 3 DUP FROM 0
with errorcode 6.

Here is the input file

inputs_3d_slice_reformatted.txt

[guj]

Thanks for this PR!

I tested this for Back Transformed diagnostics and get an error:

STEP 64 starts ...
STEP 64 ends. TIME = 6.814271277e-13 DT = 1.064729887e-14
Walltime = 19.13449687 s; This step = 0.184028665 s; Avg. per step = 0.2989765135 s
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Series] Detected illegal access to iteration that has been closed previously.
SIGABRT
/usr/bin/addr2line: DWARF error: could not find variable specification at offset 240e1
/usr/bin/addr2line: DWARF error: could not find variable specification at offset 240e1
See Backtrace.0.0 file for details
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI COMMUNICATOR 3 DUP FROM 0
with errorcode 6.

Here is the input file

inputs_3d_slice_reformatted.txt

Thanks I will take a look later today

[ax3l]

The crash is likely due to additional patch parts that should remain in #1909 for the transition to group & variable based I/O options. I'll clean them out and re-test.

[guj]

The crash is likely due to additional patch parts that should remain in #1909 for the transition to group & variable based I/O options. I'll clean them out and re-test.

Would be great see why 1909 is not passing the CI.

[ax3l]

I removed the un-related streaming API changes from this PR.
I performed @RevathiJambunathan's test again and BTD works again :)

Let's also add BTD with openPMD to CI in a PR.