Coordinates not set correctly (or at least not output correctly by OpenPMD) in CarpetX

[rhaas80]

The attached parfile, when run with CarpetX git hash f05b7a4, using 1 MPI rank, produces an output file amrex3Lev.it00000000.h5 where some coordinates for lev0 and lev1 are incorrectly stored as 0. Since HDF5 is used as the backend this can be easily verified using h5dump. GitHub forces gz, sorry.

amrex3Lev.par.gz
amrex3Lev.it00000000.h5.gz

[eschnett]

I see that the mesh coordinates_cell_coords_lev00 variable coordinates_ccoordx contains many nonzero values.

I also see that there are too many zero values. Maybe the ghost zones or the outer boundaries are all set to zero for some reason?

[rhaas80]

Let me take a closer look. I had plotted via Steve's openPMD plotting script, gotten zeros and did a h5dump (and saw lots of zeros, but this may have been just inside ghosts or so).

Indeed being more careful and doing a

h5dump -d /data/0/meshes/coordinates_vertex_coords_lev00/coordinates_vcoordx amrex.it00000000.h5

I do see non-zero values as well.

I'll update the description to "some" from "all". There should not be any large sections of 0.0 values even in what may be ghosts, should there?

[eschnett]

The zeros I see are in the outer boundary, and they should not be there.

[rhaas80]

ok. so I guess I will try and check what is present in the simulation grid to make sure that this is not just an output issue.

[eschnett]

The zeros I see in the outer boundary are not present in TSV output. I assume the problem lies with the openPMD output.

[eschnett]

I see that levels 1 and 2 contain only zeros. Only level 0 is "fine", apart from its ghosts. Again this is specific to openPMD output.

[rhaas80]

How strange. Ok, then I need to check what the Python code actually plots since it seems to plot non-zero values near the center only (where lev 1 and lev 2 would exist but not lev 0).

[eschnett]

I understand what is going on.

openPMD specifies that, if the underlying storage format (here: HDF5) supports datasets with missing data (HDF5 does), then chunks should be implemented as if they covered the whole domain, and points not covered by chunks should be represented as missing data.

Thus, in HDF5, every level seems to consisting of a single large dataset covering the whole domain. Points that are not defined are then filled in by HDF5 with its "missing value" placeholder, which is 0 by default. If you look carefully, then then centres of the refinement levels actually do contain data.

I think there is an HDF5 function that lets you extract where data actually exists, but I'm not 100% sure. The easier way would be to parse the openPMD data structure that specifies where chunks lie, and to read only the respective hyperslabs.

It's not a bug, it's a feature.

The outer boundary is indeed not output by CarpetX. I should fix that.

[eschnett]

Here is some Julia code that shows this:

julia> using HDF5
julia> f = h5open("/Users/eschnett/simulations/amrex3Lev/output-0000/amrex3Lev/amrex3Lev.it00000000.h5")
julia> extrema(f["data"]["0"]["meshes"]["coordinates_cell_coords_lev01"]["coordinates_ccoordx"][])
(-0.6171875, 0.6171875)

[eschnett]

The HDF5 mechanism that access parts of dataset is called "chunk", it's part of the dataset API.

[rhaas80]

hmm. ok. and I guess h5dump also cannot distinguish between "no data, use default" and "0 written". Sigh. Sure this is a way to get non-rectangular data into HDF5's arrays. 

well at least one mystery solved.

[eschnett]

H5Pset_fill_value()

[rhaas80]

well yes, but how do I distinguish between the "fill value" being just a filler and it being an actual value? (in openpmd I would have to parse the grid structure to know what to ignore I guess).