Reading from dmrcp index files?

[TomNicholas]

@betolink suggested today the idea of reading from dmrcpp (DMR++) files, which are apparently a sidecar filetype that is often already included with certain NASA datasets, and basically is a chunk references format.

Could we imagine ingesting these into VirtualiZarr as another way to read byte ranges? i.e. create a ChunkManifest/ManifestArray object from the DMR++. Or maybe even imagine writing out to this filetype?

[TomNicholas]

The same thing was suggested on the kerchunk tracker fsspec/kerchunk#221

[TomNicholas]

cc @jgallagher59701

[jgallagher59701]

Short answer: Yes. DMR++ could be used to provide both the Variable data via chunk locations and also attribute name/type/value information. DMR++ supports almost all of HDF5 at this point, almost all of HDF4 and we're close to complete (but with significant optimization needed) HDF-EOS2.

Our work on DMR++ is heavily focused on NASA/ESDIS and Earth Data Cloud needs, hence the focus on HDF5, HDF4 and HDF-EOS2.

If you want to email me directly, jgallagher@opendap.org

[ayushnag]

I think this idea has a lot of potential especially with integration into earthaccess. I can make an adapter for dmr++ to zarr metadata. The concept has already been done here in zarr-eosdis-store but only for single file references (so no concatenation logic). Single reference creation is the most time consuming step so by skipping that, the usability of earthaccess.consolidate_metadata() is much higher

Vision: user can find a NASA collection (or specific files) with earthaccess.search(), then earthaccess.consolidate_metadata() once, and have rapid access to the entire collection. The only issue with this currently is that they need to pass kerchunk_kwargs and know what to pass to MultiZarrToZarr (most scientists probably don't want to do this). Of course the function will also be much faster now since we only need to read DMR++ files and convert them to something virtualizarr understands.

The virtualizarr workflow would be ideal since people can use the familiar xarray interface to concat and then virtualize.

[betolink]

@ayushnag what you described is a fantastic idea! a researcher could create an on-demand virtual datasets with 3 lines of code, and if we use the dmr++ should be pretty fast or at least faster than creating the kerchunk references directly.

In case it's useful, H. Joe Lee from the HDF Group was working on a white paper on kerchunk and created this kerchunk generation from dmr++ reference files utility and other tools, maybe the script requires some updates but it can be done (a pure format to format translation).

If this gets integrated into VirtualiZarr earthaccess could replace what we have in consolidate_metadata() with this and have use cases like "Create a virtual dataset for canopy heights over the Yucatan peninsula"

import earthaccess

# search with spatiotemporal constraints 
results = earthaccess.search_data(
    doi="10.3334/ORNLDAAC/2056",
    bounding_box=(-91.18, 17.43, -86.74, 21.65)
)

# will use Dask to translate the dmr++ references instead of kerchunk 
earthaccess.consolidate_metadata(results,  "gedi_yucatan.parquet", combine='by_coords', format="parquet")

# ... then we open it the same way 

We have bi-weekly hacking hours for earthaccess, maybe we could meet there and discuss this integration with VirtualiZarr and your ideas with the dmr++ files. Thanks for taking the lead on this!

[betolink]

@agoodm I know you're aware of this project, I think all the cool stuff you're prototyping would fit somewhere in VirtualiZarr, @TomNicholas and @ayushnag Alex is working on creating logical cubes for many netcdf in the ITS_LIVE project so that we can load them efficiently in xarray (or other Zarr/parquet compatible tools)

[jgallagher59701]

I've been following along, but I admit, only partly because my knowledge of earthaccess and its capabilities is limited. However, @betolink thanks for calling my/our attention to Joe Lee's work with DMR++ to Zarr.

If you need/want any information of DMR++, including our forthcoming support for HDF4, please let me know.

Also, I'm very interested in learning about this bit of code and what it implies:

earthaccess.consolidate_metadata(results, "gedi_yucatan.parquet", combine='by_coords', format="parquet")

That is, are you using parquet to store the kerchunk json?

Thanks!

[jgallagher59701]

@TomNicholas One thing I should mention is that the DMR++ does not have to be co-located with the data it describes.

While NASA uses the DMR++ documents as sidecar files - they are objects with the same base name as the data files they describe and reside in the same S3 bucket - that is merely how NASA uses them. The DMR++ holds an href to the data it describes and, overriding that 'global' href, can include hrefs to the granularity of each individual chunk of data. Thus, the DMR++ does not have to be a true sidecar file and is not limited to data values from one particular source (or format, for that matter).

Our reader supports both hrefs for HTTP/S and the local file system.

[TomNicholas]

Also, I'm very interested in learning about this bit of code and what it implies:

earthaccess.consolidate_metadata(results,  "gedi_yucatan.parquet", combine='by_coords', format="parquet")

That is, are you using parquet to store the kerchunk json?

I'm assuming this snippet is aspirational, as it seems to mix something kerchunk can do (write references as parquet¹) with the syntax that VirtualiZarr is aiming for (combine='by_coords', like xarray.open_mfdataset).

The DMR++ holds an href to the data it describes and, overriding that 'global' href, can include hrefs to the granularity of each individual chunk of data.

That sounds very close to the chunk manifest (manifest.json) proposal for zarr (see zarr-developers/zarr-specs#287), where you can store a different path (local or remote) for each chunk

{
    "0.0.0": {"path": "s3://bucket/foo.nc", "offset": 100, "length": 100},
    "0.0.1": {"path": "s3://bucket/foo.nc", "offset": 200, "length": 100},  
    "0.1.0": {"path": "s3://bucket/foo.nc", "offset": 300, "length": 100},  
    "0.1.1": {"path": "s3://bucket/foo.nc", "offset": 400, "length": 100}, 
}

So hopefully the translation between DMR++ and zarr chunk manifests should be smooth.

Footnotes

1. VirtualiZarr could also write kerchunk-format references to parquet - the issue to track that feature request is Writing to parquet (following kerchunk format) #72. ↩

[betolink]

Hi @jgallagher59701 Tom is right, the snippet is aspirational. Once these features get integrated into VirtualiZarr we will leverage them in earthaccess to create on-demand virtual datasets, I see this as having a client side Opendap server (in a way).

Supporting HDF EOS/ HDF4 will be fantastic! this is one of the main bottlenecks for a lot of workflows (MODIS, VIRS etc etc)
and yes the serialization of these potentially big cubes will perform better in formats like Parquet but the content will be the same as the Kerchunk jsons or zarr chunk manifests. I think we all should talk to expand this collaboration!

NOTE: we already have this in earthaccess but we use kerchunk, with this dmr++ to chunk manifest translation we will only use kerchunk if there is no dmr++ available for a given dataset.

[jgallagher59701]

@betolink Should we focus on adding the ability to process dmr++ in earthaccess as an initial first step?

I don't have much in the way of cycles this quarter - we're down two developers, but I can answer questions about dmr++.

Also, as an additional thing to work on, we could investigate ways to store/access the dmr++ that would allow faster access to the needed information for a given variable (and possibly elide the boulky attribute information until/if it is needed). Doing that seems like it would provide one way to make your aspirational API.

[betolink]

I think this adapter should live in VirtualiZarr, @ayushnag is planning on starting it soon I believe (thanks Ayush!!). From earthaccess we only need a simple method to verify that there is a dmr++ available for a dataset and then pass the list of the files and what kind of logical concatenation we want from VirtualiZarr. I'm sure we'll have questions about dmr++!

I'm not sure what do you mean by storing the dmr++, are you talking about the resulting logical cubes or the original dmr++ files?